unexpected error error_class=Errno::EADDRINUSE error="Address already in use - bind(2) for 0.0.0.0:24224"

[jithendra12]

I have used the same td-agent.conf file provided in this repo: fluent.conf.rt to collect application metrics of jfrog artifactory on a centos VM. but when I start td-agent, I keep seeing these below errors that 24224 port is already in use.

2021-02-01 23:58:18 +0000 [error]: #0 unexpected error error_class=Errno::EADDRINUSE error="Address already in use - bind(2) for 0.0.0.0:24224"
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/socket.rb:201:in `bind'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/socket.rb:201:in `listen'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin_helper/server.rb:357:in `server_create_tcp_socket'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin_helper/server.rb:212:in `server_create_for_tcp_connection'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin_helper/server.rb:92:in `server_create_connection'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/in_forward.rb:172:in `start'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/root_agent.rb:200:in `block in start'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/root_agent.rb:189:in `block (2 levels) in lifecycle'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/root_agent.rb:188:in `each'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/root_agent.rb:188:in `block in lifecycle'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/root_agent.rb:175:in `each'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/root_agent.rb:175:in `lifecycle'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/root_agent.rb:199:in `start'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/engine.rb:248:in `start'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/engine.rb:147:in `run'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/supervisor.rb:603:in `block in run_worker'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/supervisor.rb:840:in `main_process'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/supervisor.rb:594:in `run_worker'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/command/fluentd.rb:361:in `<top (required)>'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/bin/fluentd:8:in `<top (required)>'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/bin/fluentd:23:in `load'
  2021-02-01 23:58:18 +0000 [error]: #0 /opt/td-agent/bin/fluentd:23:in `<top (required)>'
  2021-02-01 23:58:18 +0000 [error]: #0 /sbin/td-agent:7:in `load'
  2021-02-01 23:58:18 +0000 [error]: #0 /sbin/td-agent:7:in `<main>'
2021-02-01 23:58:18 +0000 [error]: #0 unexpected error error_class=Errno::EADDRINUSE error="Address already in use - bind(2) for 0.0.0.0:24224"
  2021-02-01 23:58:18 +0000 [error]: #0 suppressed same stacktrace
2021-02-01 23:58:18 +0000 [info]: Worker 0 finished unexpectedly with status 1
2021-02-01 23:58:19 +0000 [info]: adding filter pattern="jfrog.rt.**.service" type="record_transformer"
2021-02-01 23:58:19 +0000 [info]: adding filter pattern="jfrog.rt.**.service" type="parser"
2021-02-01 23:58:19 +0000 [info]: adding filter pattern="jfrog.rt.artifactory.request" type="record_transformer"
2021-02-01 23:58:19 +0000 [info]: adding filter pattern="jfrog.rt.frontend.request" type="record_transformer"
2021-02-01 23:58:19 +0000 [info]: adding filter pattern="jfrog.rt.metadata.request" type="record_transformer"
2021-02-01 23:58:19 +0000 [info]: adding filter pattern="jfrog.**" type="record_transformer"
2021-02-01 23:58:19 +0000 [info]: adding filter pattern="jfrog.callhome" type="record_transformer"
2021-02-01 23:58:19 +0000 [info]: adding match pattern="jfrog.callhome" type="http"
2021-02-01 23:58:19 +0000 [warn]: #0 Status code 503 is going to be removed from default `retryable_response_codes` from fluentd v2. Please add it by yourself if you wish
2021-02-01 23:58:19 +0000 [info]: adding filter pattern="jfrog.rt.artifactory.request" type="prometheus"
2021-02-01 23:58:20 +0000 [info]: adding filter pattern="jfrog.rt.artifactory.service" type="prometheus"
2021-02-01 23:58:20 +0000 [info]: adding filter pattern="jfrog.rt.artifactory.access" type="prometheus"
2021-02-01 23:58:20 +0000 [info]: adding filter pattern="jfrog.rt.access.audit" type="prometheus"
2021-02-01 23:58:20 +0000 [info]: adding source type="prometheus"
2021-02-01 23:58:20 +0000 [info]: adding source type="monitor_agent"
2021-02-01 23:58:20 +0000 [info]: adding source type="forward"
2021-02-01 23:58:20 +0000 [info]: adding source type="prometheus_monitor"
2021-02-01 23:58:20 +0000 [info]: adding source type="prometheus_output_monitor"
2021-02-01 23:58:20 +0000 [info]: adding source type="prometheus_tail_monitor"
2021-02-01 23:58:20 +0000 [info]: adding source type="tail"
2021-02-01 23:58:20 +0000 [info]: adding source type="tail"
2021-02-01 23:58:20 +0000 [info]: adding source type="tail"
2021-02-01 23:58:20 +0000 [info]: adding source type="tail"
2021-02-01 23:58:20 +0000 [info]: adding source type="tail"
2021-02-01 23:58:20 +0000 [info]: adding source type="tail"
2021-02-01 23:58:20 +0000 [info]: adding source type="tail"
2021-02-01 23:58:20 +0000 [info]: adding source type="tail"
2021-02-01 23:58:20 +0000 [info]: adding source type="tail"
2021-02-01 23:58:20 +0000 [info]: adding source type="tail"
2021-02-01 23:58:20 +0000 [info]: adding source type="tail"
2021-02-01 23:58:20 +0000 [info]: adding source type="tail"
2021-02-01 23:58:20 +0000 [info]: adding source type="tail"
2021-02-01 23:58:20 +0000 [info]: adding source type="exec"
2021-02-01 23:58:20 +0000 [info]: adding source type="exec"
2021-02-01 23:58:20 +0000 [info]: #0 starting fluentd worker pid=25680 ppid=25342 worker=0
2021-02-01 23:58:20 +0000 [info]: #0 [access_security_audit_tail] following tail of /app/jfrog/artifactory/var/log/access-security-audit.log
2021-02-01 23:58:20 +0000 [info]: #0 [artifactory_access_tail] following tail of /app/jfrog/artifactory/var/log/artifactory-access.log
2021-02-01 23:58:20 +0000 [info]: #0 [router_request_tail] following tail of /app/jfrog/artifactory/var/log/router-request.log
2021-02-01 23:58:20 +0000 [info]: #0 [metadata_request_tail] following tail of /app/jfrog/artifactory/var/log/metadata-request.log
2021-02-01 23:58:20 +0000 [info]: #0 [frontend_request_tail] following tail of /app/jfrog/artifactory/var/log/frontend-request.log
2021-02-01 23:58:20 +0000 [info]: #0 [artifactory_request_tail] following tail of /app/jfrog/artifactory/var/log/artifactory-request.log
2021-02-01 23:58:20 +0000 [info]: #0 [access_request_tail] following tail of /app/jfrog/artifactory/var/log/access-request.log
2021-02-01 23:58:20 +0000 [info]: #0 [router_traefik_tail] following tail of /app/jfrog/artifactory/var/log/router-traefik.log
2021-02-01 23:58:20 +0000 [info]: #0 [router_service_tail] following tail of /app/jfrog/artifactory/var/log/router-service.log
2021-02-01 23:58:20 +0000 [info]: #0 [metadata_service_tail] following tail of /app/jfrog/artifactory/var/log/metadata-service.log
2021-02-01 23:58:20 +0000 [info]: #0 [frontend_service_tail] following tail of /app/jfrog/artifactory/var/log/frontend-service.log
2021-02-01 23:58:20 +0000 [info]: #0 [artifactory_service_tail] following tail of /app/jfrog/artifactory/var/log/artifactory-service.log
2021-02-01 23:58:20 +0000 [info]: #0 [access_service_tail] following tail of /app/jfrog/artifactory/var/log/access-service.log
2021-02-01 23:58:20 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0"
2021-02-01 23:58:20 +0000 [error]: #0 unexpected error error_class=Errno::EADDRINUSE error="Address already in use - bind(2) for 0.0.0.0:24224"
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/socket.rb:201:in `bind'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/socket.rb:201:in `listen'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin_helper/server.rb:357:in `server_create_tcp_socket'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin_helper/server.rb:212:in `server_create_for_tcp_connection'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin_helper/server.rb:92:in `server_create_connection'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/in_forward.rb:172:in `start'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/root_agent.rb:200:in `block in start'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/root_agent.rb:189:in `block (2 levels) in lifecycle'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/root_agent.rb:188:in `each'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/root_agent.rb:188:in `block in lifecycle'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/root_agent.rb:175:in `each'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/root_agent.rb:175:in `lifecycle'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/root_agent.rb:199:in `start'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/engine.rb:248:in `start'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/engine.rb:147:in `run'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/supervisor.rb:603:in `block in run_worker'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/supervisor.rb:840:in `main_process'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/supervisor.rb:594:in `run_worker'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/command/fluentd.rb:361:in `<top (required)>'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/bin/fluentd:8:in `<top (required)>'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/bin/fluentd:23:in `load'
  2021-02-01 23:58:20 +0000 [error]: #0 /opt/td-agent/bin/fluentd:23:in `<top (required)>'
  2021-02-01 23:58:20 +0000 [error]: #0 /sbin/td-agent:7:in `load'
  2021-02-01 23:58:20 +0000 [error]: #0 /sbin/td-agent:7:in `<main>'

I checked for processes which are using this port to kill them. But couldn't find any. Also, tried opening range of ports 24210 to 24250. Still did not work. Any suggestions on how to resolve or troubleshoot this issue? Thanks!

[peters95]

You have another process already listing on port 24224.

You can change the port that the metrics server is exposed inside the fluent.conf.rt to resolve this issue.

Example:

<source>
  @type prometheus
  port 22222
</source>

[jithendra12]

Yes, I tried it already. On the http interface of the VM, it is working fine when I give a port and metrics_path to the prometheus source. I am able to see logs printing on http interface in the browser or through curl http://<IP of VM>:24231/metrics:

<source>
  @type prometheus
  bind 0.0.0.0
  port 24231
  metrics_path /metrics
</source>

but, when I start td-agent in the VM's shell, the earlier error still shows up:
2021-02-06 00:31:44 +0000 [error]: #0 unexpected error error_class=Errno::EADDRINUSE error="Address already in use - bind(2) for 0.0.0.0:24224"
I was wondering if that effects the logs that are being forwarded to http interface.