fluentd high memory consuption #4126

requiming · 2023-04-03T12:56:35Z

requiming
Apr 3, 2023

Describe the bug

High memory consumption during the work

To Reproduce

I attached sigdump and logs

Expected behavior

Memory consuption around 50-100 MB without spikes

Your Environment

we use fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-graylog-1.1 image

Your Configuration

AKS Cluster on azure 1.25.5
CBLMariner OS

Your Error Log

https://pastebin.com/HF8y9kYk

Additional context

  Thread #<Thread:0x00007f2401d3bc70 run> status=run priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/engine.rb:152:in `sleep'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/engine.rb:152:in `run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/supervisor.rb:760:in `block in run_worker'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/supervisor.rb:1036:in `main_process'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/supervisor.rb:751:in `run_worker'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/command/fluentd.rb:386:in `<top (required)>'
      <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
      <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/bin/fluentd:15:in `<top (required)>'
      /fluentd/vendor/bundle/ruby/3.1.0/bin/fluentd:25:in `load'
      /fluentd/vendor/bundle/ruby/3.1.0/bin/fluentd:25:in `<main>'
  Thread #<Thread:0x00007f23fb73ec88 sleep> status=sleep priority=0
      /usr/local/lib/ruby/3.1.0/openssl/buffering.rb:157:in `sysread'
      /usr/local/lib/ruby/3.1.0/openssl/buffering.rb:157:in `readpartial'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/http-4.4.1/lib/http/timeout/null.rb:45:in `readpartial'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/http-4.4.1/lib/http/connection.rb:212:in `read_more'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/http-4.4.1/lib/http/connection.rb:92:in `readpartial'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/http-4.4.1/lib/http/response/body.rb:30:in `readpartial'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/http-4.4.1/lib/http/response/body.rb:36:in `each'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/kubeclient-4.9.3/lib/kubeclient/watch_stream.rb:25:in `each'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:160:in `process_pod_watcher_notices'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:43:in `block in set_up_pod_thread'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:41:in `loop'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:41:in `set_up_pod_thread'
  Thread #<Thread:0x00007f23fb73e8f0 sleep> status=sleep priority=0
      /usr/local/lib/ruby/3.1.0/openssl/buffering.rb:157:in `sysread'
      /usr/local/lib/ruby/3.1.0/openssl/buffering.rb:157:in `readpartial'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/http-4.4.1/lib/http/timeout/null.rb:45:in `readpartial'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/http-4.4.1/lib/http/connection.rb:212:in `read_more'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/http-4.4.1/lib/http/connection.rb:92:in `readpartial'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/http-4.4.1/lib/http/response/body.rb:30:in `readpartial'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/http-4.4.1/lib/http/response/body.rb:36:in `each'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/kubeclient-4.9.3/lib/kubeclient/watch_stream.rb:25:in `each'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:152:in `process_namespace_watcher_notices'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:42:in `block in set_up_namespace_thread'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:40:in `loop'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:40:in `set_up_namespace_thread'
  Thread #<Thread:0x00007f23fab34500@FFI Callback Dispatcher sleep> status=sleep priority=0
  Thread #<Thread:0x00007f23f9f05b80@flush_thread_0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `sleep'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `wait'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `flush_thread_run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:501:in `block (2 levels) in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f9f057e8@flush_thread_1 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `sleep'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `wait'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `flush_thread_run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:501:in `block (2 levels) in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f9f05540@flush_thread_2 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `sleep'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `wait'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `flush_thread_run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:501:in `block (2 levels) in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f9f05298@flush_thread_3 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `sleep'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `wait'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `flush_thread_run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:501:in `block (2 levels) in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f9f04ff0@flush_thread_4 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `sleep'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `wait'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `flush_thread_run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:501:in `block (2 levels) in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f9f04d48@flush_thread_5 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `sleep'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `wait'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `flush_thread_run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:501:in `block (2 levels) in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f9f04aa0@flush_thread_6 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `sleep'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `wait'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `flush_thread_run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:501:in `block (2 levels) in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f9f047f8@flush_thread_7 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `sleep'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `wait'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1524:in `flush_thread_run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:501:in `block (2 levels) in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f9f04578@enqueue_thread /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1463:in `sleep'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1463:in `enqueue_thread_run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f9f036c8@event_loop /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f9f02278@event_loop /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f9f014e0@event_loop /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f9f00748@event_loop /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f9eff988@event_loop /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f9efebf0@event_loop /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f9efde58@event_loop /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f9efd0c0@event_loop /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f9efc328@event_loop /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23fbfb9c78@event_loop /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23fbfc3bb0@event_loop /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23fbfca820@event_loop /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23fbfd80d8@event_loop /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f174d148@event_loop /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f174c3b0@event_loop /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f133ef70@in_prometheus_server /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:70 sleep> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/webrick-1.7.0/lib/webrick/server.rb:173:in `select'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/webrick-1.7.0/lib/webrick/server.rb:173:in `block in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/webrick-1.7.0/lib/webrick/server.rb:32:in `start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/webrick-1.7.0/lib/webrick/server.rb:160:in `start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/http_server/compat/server.rb:61:in `start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/http_server.rb:75:in `block (2 levels) in http_server_create_http_server'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x00007f23f133c4c8 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/fluent_log_event_router.rb:95 sleep_forever> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/fluent_log_event_router.rb:99:in `pop'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/fluent_log_event_router.rb:99:in `block (2 levels) in start'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/fluent_log_event_router.rb:98:in `loop'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/fluent_log_event_router.rb:98:in `block in start'
  Thread #<Thread:0x00007f23f0f30618 /fluentd/vendor/bundle/ruby/3.1.0/gems/webrick-1.7.0/lib/webrick/utils.rb:158 sleep_forever> status=sleep priority=0
      /fluentd/vendor/bundle/ruby/3.1.0/gems/webrick-1.7.0/lib/webrick/utils.rb:179:in `pop'
      /fluentd/vendor/bundle/ruby/3.1.0/gems/webrick-1.7.0/lib/webrick/utils.rb:179:in `watch'
  GC stat:
      count: 43
      time: 316
      heap_allocated_pages: 1481
      heap_sorted_length: 1481
      heap_allocatable_pages: 0
      heap_available_slots: 605137
      heap_live_slots: 579072
      heap_free_slots: 26065
      heap_final_slots: 0
      heap_marked_slots: 167329
      heap_eden_pages: 1418
      heap_tomb_pages: 63
      total_allocated_pages: 1481
      total_freed_pages: 0
      total_allocated_objects: 1425833
      total_freed_objects: 846761
      malloc_increase_bytes: 6120464
      malloc_increase_bytes_limit: 26290136
      minor_gc_count: 33
      major_gc_count: 10
      compact_count: 0
      read_barrier_faults: 0
      total_moved_objects: 0
      remembered_wb_unprotected_objects: 769
      remembered_wb_unprotected_objects_limit: 1362
      old_objects: 165672
      old_objects_limit: 300710
      oldmalloc_increase_bytes: 23712360
      oldmalloc_increase_bytes_limit: 20132659
  Built-in objects:
   605,137: TOTAL
   242,075: T_STRING
   118,755: T_HASH
    90,726: T_IMEMO
    54,197: T_ARRAY
    33,413: T_DATA
    25,978: FREE
    17,516: T_OBJECT
     8,935: T_MATCH
     4,208: T_RATIONAL
     4,199: T_CLASS
     1,314: T_BIGNUM
     1,071: T_STRUCT
       877: T_REGEXP
       700: T_ICLASS
       653: T_SYMBOL
       406: T_MODULE
       104: T_FILE
         9: T_FLOAT
         1: T_COMPLEX
  All objects:
   179,585: String
    41,707: Hash
    35,930: Array
     7,172: Time
     5,989: Set
     4,340: MatchData
     4,326: Thread::Backtrace
     4,324: File::Stat
     4,289: EOFError
     3,863: Rational
     2,459: Class
     2,383: MIME::Type::Columnar
     1,560: Proc
     1,247: Integer
       873: Regexp
       812: Thread::Mutex
       649: Gem::Requirement
       570: Range
       564: Symbol
       518: Fluent::EventTime
       479: MessagePack::Buffer
       463: Fluent::MultiEventStream
       455: MessagePack::Packer
       454: JSON::Ext::Generator::State
       404: Module
       313: Gem::Dependency
       245: Enumerator
       230: Fluent::Config::ConfigureProxy
       192: Gem::Version
       187: Fluent::Config::Element
       184: Method
       163: Fluent::Config::Section
       159: Monitor
       150: Fluent::Plugin::TailInput::TargetInfo
       142: Fluent::Plugin::Base::State
       125: Gem::StubSpecification::StubLine
       125: Gem::Specification
       125: Gem::StubSpecification
       117: Fluent::Plugin::Buffer::Metadata
       113: WEBrick::HTTPVersion
       104: Fluent::Plugin::LocalMetrics
       103: Encoding
        90: OpenSSL::ASN1::ASN1Data
        60: Thread
        50: OpenSSL::X509::Extension
        45: File
        44: Fiber
        42: Strftime
        31: TCPSocket
        30: Fluent::Plugin::TailInput::TailWatcher::FIFO
        30: Fluent::Plugin::TailInput::TailWatcher::IOHandler
        30: Fluent::Plugin::TailInput::StatWatcher
        30: Fluent::Plugin::TailInput::TimerTrigger
        30: Fluent::Plugin::TailInput::TailWatcher::RotateHandler
        30: Fluent::Plugin::TailInput::TailWatcher
        30: Fluent::Plugin::TailInput::FilePositionEntry
        28: #<Class:0x00007f23f133ea20>
        28: URI::HTTP
        28: URI::Generic
        28: WEBrick::HTTPResponse
        28: WEBrick::HTTPRequest
        26: Fluent::Plugin::Buffer::MemoryChunk
        25: URI::HTTPS
        24: MessagePack::Unpacker
        24: StringIO
        23: Thread::ConditionVariable
        22: Logger::Formatter
        21: Fluent::Log
        21: ServerEngine::DaemonLogger
        21: FFI::Type::Builtin
        20: OpenSSL::ASN1::Sequence
        20: OptionParser::Switch::RequiredArgument
        20: Fluent::PluginLogger
        18: FFI::FunctionType
        18: OpenStruct
        17: OptionParser::Switch::NoArgument
        17: Prometheus::Client::Gauge
        17: Prometheus::Client::LabelSetValidator
        17: Prometheus::Client::DataStores::Synchronized::MetricStore
        16: Coolio::Loop
        16: Fluent::TimeParser
        15: OpenSSL::X509::Certificate
        15: Fluent::PluginHelper::EventLoop::DefaultWatcher
        15: Fluent::PluginHelper::Timer::TimerWatcher
        15: FFI::StructLayout::Number
        14: Object
        14: OpenSSL::Buffering::Buffer
        14: Fluent::Plugin::TailInput::PositionFile
        14: Fluent::Plugin::TailInput
        14: Fluent::Capability
        14: Fluent::Plugin::TailInput::MetricsInfo
        12: Strptime
        10: Fluent::Registry
        10: IPAddr
        10: OpenSSL::ASN1::OctetString
        10: OpenSSL::ASN1::ObjectId
        10: FFI::StructLayout::Function
         9: Float
         8: Fluent::EventRouter::Pipeline::FilterOptimizer
         8: Fluent::EventRouter::Pipeline
         8: UDPSocket
         8: HTTP::Headers
         8: Fluent::Plugin::KubernetesParser
         8: Fluent::Plugin::Output::FlushThreadState
         7: OpenSSL::SSL::SSLSocket
         7: OpenSSL::SSL::SSLContext
         6: UnboundMethod
         6: OptionParser::OptionMap
         6: FFI::Type::Mapped
         6: HTTP::Accept::MediaTypes::MediaRange
         6: Net::HTTPOK
         6: Concurrent::Event
         6: HTTP::CookieJar::HashStore
         6: HTTP::CookieJar
         6: RestClient::Request
         6: RestClient::Resource
         6: FFI::Function
         6: FFI::DynamicLibrary::Symbol
         6: HttpParser::Parser::DataCallback
         5: IO
         5: Gem::Platform
         5: JSON::Ext::Parser
         5: StringScanner
         5: Fluent::EventRouter::Rule
         5: RestClient::Response
         5: Concurrent::Delay
         5: OpenSSL::SSL::Session
         5: Net::BufferedIO
         5: BigDecimal
         5: Socket
         5: Addrinfo
         5: Net::HTTP
         5: Net::HTTP::Get
         5: Netrc
         5: Netrc::TokenArray
         5: Errno::ENOENT
         4: FFI::StructLayout
         4: LruRedux::TTL::ThreadSafeCache
         4: HttpParser::Parser::Callback
         4: FFI::MemoryPointer
         3: OptionParser::List
         3: Fluent::GlobMatchPattern
         3: Oj::Cache
         3: FFI::Enum
         3: Thread::Queue
         2: ThreadGroup
         2: fatal
         2: Refinement
         2: Fluent::EventRouter::MatchCache
         2: Fluent::EventRouter
         2: Fluent::NoMatchMatch
         2: MessagePack::Factory
         2: Random
         2: OptionParser::Switch::PlacedArgument
         2: OptionParser::Switch::OptionalArgument
         2: Fluent::AllMatchPattern
         2: HTTP::Accept::Encodings::ContentCoding
         2: MIME::Types::Container
         2: FFI::StructByReference
         2: HTTP::Options
         2: HTTP::Client
         2: Kubeclient::Common::WatchStream
         2: HttpParser::Settings
         2: HttpParser::Parser
         2: HttpParser::Instance
         2: HTTP::Response::Parser
         2: HTTP::Connection
         2: Addressable::URI
         2: HTTP::URI
         2: HTTP::Timeout::Null
         2: Fluent::Plugin::RegexpParser
         2: HTTP::Response::Body
         2: HTTP::ContentType
         2: HTTP::Response::Status
         2: HTTP::Response
         2: Fluent::OneEventStream
         1: #<Class:0x00007f2401d643a0>
         1: #<Class:0x00007f2401d50c88>
         1: Complex
         1: IOError
         1: Binding
         1: RubyVM
         1: Ractor
         1: NoMemoryError
         1: Gem::PathSupport
         1: SystemStackError
         1: ARGF.class
         1: URI::RFC2396_Parser
         1: URI::RFC3986_Parser
         1: DidYouMean::DeprecatedMapping
         1: Gem::RequestSet::Lockfile::Tokenizer::Token
         1: OptionParser::CompletingHash
         1: Concurrent::LockFreeStack::Node
         1: Fluent::Label
         1: TZInfo::UnaryMinusGlobalStringDeduper
         1: #<Class:0x00007f23ff00ebf8>
         1: Resolv::DNS::Config
         1: Resolv::DNS
         1: Resolv::Hosts
         1: Resolv
         1: Psych::Handler::DumperOptions
         1: Fluent::RootAgent
         1: Concurrent::Promises::InternalStates::Fulfilled
         1: Concurrent::Promises::InternalStates::Reserved
         1: Concurrent::Promises::InternalStates::Pending
         1: #<Class:0x00007f23ff068658>::Config
         1: Fluent::SystemConfig
         1: #<Class:0x00007f23fcf1c850>
         1: #<Class:0x00007f23ff0701c8>
         1: Fluent::Plugin::RecordTransformerFilter::RubyPlaceholderExpander::CleanroomExpander
         1: Fluent::Plugin::RecordTransformerFilter::RubyPlaceholderExpander
         1: Concurrent::ImmediateExecutor
         1: Concurrent::AtomicReference
         1: Concurrent::Utility::ProcessorCounter
         1: OpenSSL::X509::Store
         1: OpenSSL::PKey::DH
         1: Fluent::Plugin::RecordTransformerFilter
         1: Fluent::Compat::NullOutputChain
         1: Fluent::EngineClass
         1: OptionParser
         1: Fluent::Supervisor
         1: Fluent::Plugin::NullOutput
         1: Fluent::Supervisor::LoggerInitializer
         1: MIME::Types
         1: Fluent::Plugin::MemoryBuffer
         1: MIME::Types::WarnLogger::WarnLogDevice
         1: MIME::Types::WarnLogger
         1: FFI::Enums
         1: Fluent::Plugin::RecordModifierFilter
         1: Fluent::GELFOutput
         1: FFI::DynamicLibrary
         1: FFI::Compiler::Platform
         1: Fluent::Plugin::KubernetesMetadataFilter
         1: FFI::StructLayout::Array
         1: FFI::ArrayType
         1: FFI::StructByValue
         1: FFI::StructLayout::Pointer
         1: Kubeclient::Client
         1: KubernetesMetadata::Stats
         1: FFI::Pointer
         1: Prometheus::Client::Registry
         1: Fluent::Plugin::PrometheusInput
         1: Fluent::Plugin::PrometheusOutputMonitorInput
         1: Fluent::Plugin::MonitorAgentInput
         1: Fluent::Plugin::CriParser
         1: Fluent::Plugin::NoneParser
         1: Fluent::Plugin::MultilineParser
         1: Fluent::Plugin::MultilineParser::MultilineRegexpParser
         1: Fluent::Plugin::SyslogParser
         1: Fluent::FluentLogEventRouter
         1: GELF::Notifier
         1: GELF::Transport::UDP
         1: Prometheus::Client::DataStores::Synchronized
         1: Prometheus::Client::Config
         1: Thread::SizedQueue
         1: WEBrick::HTTPServer
         1: WEBrick::Log
         1: Fluent::PluginHelper::HttpServer::Compat::Server
         1: LoadError
         1: WEBrick::HTTPServer::MountTable
         1: TCPServer
         1: WEBrick::Utils::TimeoutHandler
  String 25,056,823 bytes
   Array 2 elements
    Hash 0 pairs

daipom · 2023-04-04T04:38:25Z

daipom
Apr 4, 2023
Maintainer

Please follow the contributing guidelines.

Describe the bug

High memory consumption during the work

To Reproduce

I attached sigdump and logs

Expected behavior

Memory consuption around 50-100 MB without spikes

With this information, we are unable to reproduce this issue and cannot judge if it is a bug or not.

I moved this issue to discussions.

I think we need more information.

For example, how much memory is being consumed and why it is considered unusual.
(Fluentd can consume a lot of memory depending on the configuration and the amount of processing data.)

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fluentd high memory consuption #4126

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Describe the bug

To Reproduce

Expected behavior

Select a reply

Uh oh!

fluentd high memory consuption #4126

Uh oh!

Uh oh!

requiming Apr 3, 2023

Describe the bug

To Reproduce

Expected behavior

Your Environment

Your Configuration

Your Error Log

Additional context

Replies: 1 comment

Uh oh!

daipom Apr 4, 2023 Maintainer

Describe the bug

To Reproduce

Expected behavior

requiming
Apr 3, 2023

daipom
Apr 4, 2023
Maintainer