HTTP Listener Failed at Startup Possible Port Conflict on Port

Few days back I got an alert: Agent is Not Running. Possible port conflict on port(3872): Retrying the operation. Failed to start the agent after 1 attempts. Please check that the port(3872) is available.

Regarding this first check the status of the agent after that we will look into it.

[oracle@orahowracq1] cd /u01/app/oracle/product/12.1.0.2/agent/agent_inst/bin
[oracle@orahowracq1] ./emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Status agent Failure:unable to connect to http server at https://orahowracq1.hiw.com:3872/emd/lifecycle/main/. [peer not authenticated]
Agent is Not Running

Above we can see that agent is not running, so first try to start the agent normally as usually you do.

[oracle@orahowracq1] ./emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting agent ................ failed.
HTTP Listener failed at Startup
Possible port conflict on port(3872): Retrying the operation...
Failed to start the agent after 1 attempts.  Please check that the port(3872) is available.
Consult emctl.log and emagent.nohup in: /u01/app/oracle/product/12.1.0.2/agent/agent_inst/sysman/log

During startup we get an error port conflict. Now you can check which process is listening to this port using simple combination of netstat with grep command.

[oracle@orahowracq1] netstat -anp |grep 3872 

Now find the PID of this process. So that we can mark it for kill.

[oracle@orahowracq1] ps -ef|grep agent

    root  6535     1   0   Jun 17 ?      0:02 /opt/opsware/agent/bin/python /opt/opsware/agent/pylibs/shadowbot/daemonbot.pyc
    root  3804     1   0   Jun 17 ?      206:39 /opt/VRTSobc/pal33/bin/vxpal -a actionagent -x
    root  6549  6535   0   Jun 17 ?      178:38 /opt/opsware/agent/bin/python /opt/opsware/agent/pylibs/shadowbot/daemonbot.pyc
    root  9184     1   0   Jun 17 ?      71:57 /opt/Navisphere/bin/naviagent -f /etc/Navisphere/agent.config
    root  8482  8071   0   Jun 17 ?      114:20 /opt/ecc/exec/mstragent -s
  oracle 15627 15547   0   Mar 13 ?      111:40 /u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.2.0/jdk/bin/sparcv9/java -Xm
  oracle 24178  6501   0 23:22:46 pts/1  0:00 grep agent
  oracle 15547     1   0   Mar 13 ?      0:27 /u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.2.0/perl/bin/perl

Make sure that process must be in: /u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.2.0/jdk/bin/sparcv9/java. Finally kill the process having PID 15627 and start up the agent.

[oracle@orahowracq1] kill -9 15627
[oracle@orahowracq1] cd /u01/app/oracle/product/us/agent12c/bin
[oracle@orahowracq1] ./emctl start agent
[oracle@orahowracq1] ./emctl status agent
[oracle@orahowracq1] ./emctl clearstate agent
[oracle@orahowracq1] ./emctl upload

Example:

./emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Starting agent ....... failed.
HTTP Listener failed at Startup
Possible port conflict on port(3872): Retrying the operation...
Failed to start the agent after 1 attempts. Please check that the port(3872) is available.
Consult emctl.log and emagent.nohup in: /oradba/oemagent/product/12.1.0.2/agent/agent_inst/sysman/log

[oracle@orahowracq1] netstat -anp |grep 3872 
tcp 0 0 :::3872 :::* LISTEN 7507/java

[oracle@orahowracq1 bin]$ ps -ef|grep agent

oracle 7507 7314 1 Apr14 ? 03:10:55 /oradba/oemagent/product/12.1.0.2/agent/core/12.1.0.2.0/jdk/bin/java -Xmx128M -XX:MaxPermSize=96M

[oracle@orahowracq1] kill -9 7507