10g Enterprise Manager – HTTP server not starting

Yesterday when I was logging into our OEM, the webpage was down. So I went into the server and check the the services.

oem-01:/opt/oracle/oem10g/oms10g/opmn/bin> opmnctl status

Processes in Instance: EnterpriseManager0.oem-01
——————-+——————–+———+———
ias-component      | process-type       |     pid | status
——————-+——————–+———+———
DSA                | DSA                |     N/A | Down
HTTP_Server        | HTTP_Server        |     N/A | Down
LogLoader          | logloaderd         |     N/A | Down
dcm-daemon         | dcm-daemon         |     N/A | Down
OC4J               | home               |   19765 | Alive
OC4J               | OC4J_EM            |   19764 | Alive
OC4J               | OC4J_EMPROV        |   19766 | Alive
WebCache           | WebCache           |   19767 | Alive
WebCache           | WebCacheAdmin      |   19768 | Alive

The output above shows that the HTTP server is down so I thought bringing it up would fix the issue. So I tried starting it using  opmnctl command (opmnctl startproc ias-component=HTTP_Server) but it did not start. So I thought, maybe bouncing all the components would help (opmnctl stopall then opmnctl startall). Unfortunately, it did not and it says it has issues bringing up the HTTP server but no error code! So I checked oem-01:/opt/oracle/oem10g/oms10g/opmn/logs/HTTP_Server~1 file but found no error and the last message was just:

/opt/oracle/oem10g/oms10g/Apache/Apache/bin/apachectl startssl: execing httpd

Digging through the log files under /opt/oracle/oem10g/oms10g/Apache/Apache/logs I found access_log file. Tried to open this file but I got an error saying “Tmp file too large” so I tailed it found a message something to the effect of “number too large when starting the HTTP server”. Hmmmm…this is when I check the filesystem if it is full, but no, its only 97% used! Then I checked size of the file, and voila, its 2GB! I immediately knew that this must be the issue, although at the back of my mind, 2gb file size limit shouldn’t be the problem, but I copied this file over to /tmp (as backup) and zeroed out the existing file (cp -p /dev/null access_log). I then started the HTTP server and then its went up with no issues! Problem solved!