tdagent started throwing error in logs even though the systemctl status is active causing application to hang

[savy1212]

my td-agent was up and running fine since 2022-04-06 , suddenly on 2023-03-08 it started throwing error in td-agent.log as below, and then stuck, when checked the systemctl status td-agent , it was active and running, need to know if this is a known bug for fluentd or td-agent or what could have happened suddenly.

td-agent-3.2.0-0.el7.x86_64
fluent-plugins-3.4.1-1.0.2.x86_64

[root@sps22-me-oame-1 td-agent]# cat td-agent.log.1| less
2023-03-08 17:45:01 +0800 [error]: #0 unexpected error error_class=ThreadError error="killed thread"
2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:704:in run' 2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:704:in flush_buffer'
2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:664:in block in install_main_process_signal_handlers' 2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/engine.rb:228:in sleep'
2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/engine.rb:228:in run' 2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:799:in run_engine'
2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:549:in block in run_worker' 2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:724:in main_process'
2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:544:in run_worker' 2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/command/fluentd.rb:316:in <top (required)>'
2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in require' 2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in require'
2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/bin/fluentd:8:in <top (required)>' 2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in load'
2023-03-08 17:45:01 +0800 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in <main>' 2023-03-08 17:45:01 +0800 [error]: #0 unexpected error error_class=ThreadError error="killed thread" 2023-03-08 17:45:01 +0800 [error]: #0 suppressed same stacktrace 2023-03-08 17:45:07 +0800 [error]: #0 unexpected error error_class=Errno::ENOENT error="No such file or directory - connect(2) for /tmp/SERVERENGINE_SOCKETMANAGER_2022-04-06T16:15:06Z_19372" 2023-03-08 17:45:07 +0800 [error]: #0 suppressed same stacktrace 2023-03-08 17:45:07 +0800 [error]: #0 unexpected error error_class=Errno::ENOENT error="No such file or directory - connect(2) for /tmp/SERVERENGINE_SOCKETMANAGER_2022-04-06T16:15:06Z_19372" 2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.0.6/lib/serverengine/socket_manager_unix.rb:27:in initialize'
2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.0.6/lib/serverengine/socket_manager_unix.rb:27:in new' 2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.0.6/lib/serverengine/socket_manager_unix.rb:27:in connect_peer'
2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.0.6/lib/serverengine/socket_manager.rb:40:in listen' 2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.0.6/lib/serverengine/socket_manager.rb:55:in listen_tcp'
2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/server.rb:352:in server_create_tcp_socket' 2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/server.rb:212:in server_create_for_tcp_connection'
2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/server.rb:154:in server_create' 2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_tcp.rb:58:in start'
2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/root_agent.rb:165:in block in start' 2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/root_agent.rb:154:in block (2 levels) in lifecycle'
2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/root_agent.rb:153:in each' 2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/root_agent.rb:153:in block in lifecycle'
2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/root_agent.rb:140:in each' 2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/root_agent.rb:140:in lifecycle'
2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/root_agent.rb:164:in start' 2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/engine.rb:274:in start'
2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/engine.rb:219:in run' 2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:799:in run_engine'
2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:549:in block in run_worker' 2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:724:in main_process'
2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:544:in run_worker' 2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/command/fluentd.rb:316:in <top (required)>'
2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in require' 2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in require'
2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/bin/fluentd:8:in <top (required)>' 2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in load'
2023-03-08 17:45:07 +0800 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in `

'
2023-03-08 17:45:07 +0800 [error]: #0 unexpected error error_class=Errno::ENOENT error="No such file or directory - connect(2) for /tmp/SERVERENGINE_SOCKETMANAGER_2022-04-06T16:15:06Z_19372"
2023-03-08 17:45:07 +0800 [error]: #0 suppressed same stacktrace