Fluentd with http throwing ENDOFFILE error #4274

prashant407 · 2023-08-18T06:23:56Z

prashant407
Aug 18, 2023

Describe the bug

We are using fluentd as k8s setup. In the config we are sending data to logstash with http plugin with ssl. But we are getting the below error in the fluentd logs

2023-08-18 06:13:08 +0000 [warn]: #0 failed to flush the buffer. retry_time=1 next_retry_seconds=2023-08-18 06:13:13 +0000 chunk="60329c60f47dd24c9e2dd618d9879eca" error_class=EOFError error="end of file reached"
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/protocol.rb:225:in `rbuf_fill'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/protocol.rb:191:in `readuntil'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/protocol.rb:201:in `readline'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http/response.rb:40:in `read_status_line'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http/response.rb:29:in `read_new'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:1509:in `block in transport_request'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:1506:in `catch'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:1479:in `request'
  2023-08-18 06:13:08 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.2/lib/fluent/plugin/out_http.rb:229:in `block in send_request'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:920:in `start'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:605:in `start'
  2023-08-18 06:13:08 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.2/lib/fluent/plugin/out_http.rb:228:in `send_request'
  2023-08-18 06:13:08 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.2/lib/fluent/plugin/out_http.rb:136:in `write'
  2023-08-18 06:13:08 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.2/lib/fluent/plugin/output.rb:1133:in `try_flush'
  2023-08-18 06:13:08 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.2/lib/fluent/plugin/output.rb:1439:in `flush_thread_run'
  2023-08-18 06:13:08 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.2/lib/fluent/plugin/output.rb:461:in `block (2 levels) in start'
  2023-08-18 06:13:08 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2023-08-18 06:13:13 +0000 [warn]: #0 failed to flush the buffer. retry_time=2 next_retry_seconds=2023-08-18 06:13:40 +0000 chunk="60329c60f47dd24c9e2dd618d9879eca" error_class=EOFError error="end of file reached"
  2023-08-18 06:13:13 +0000 [warn]: #0 suppressed same stacktrace
2023-08-18 06:13:13 +0000 [warn]: #0 failed to flush the buffer. retry_time=3 next_retry_seconds=2023-08-18 06:15:07 +0000 chunk="60329c5c2e8293464e499f083c22a977" error_class=EOFError error="end of file reached"
  2023-08-18 06:13:13 +0000 [warn]: #0 suppressed same stacktrace

Please refer the config as below

</match> is not used.
2023-08-18 06:13:08 +0000 [warn]: parameter 'ssl_key_path' in <match **>
  @type http
  time_as_integer true
  endpoint "http://domain_name:24284"
  serializer json_lines
  add_newline true
  http_method post
  ssl_version TLSv1_2
  ssl_cert_path /fluentd/etc/ssl/client.crt
  ssl_key_path /fluentd/etc/ssl/client.key
  ssl_verify_peer true
  <buffer>
    @type "file"
    path "/var/log/fluentd_application/kubernetes.internal.application.buffer"
    chunk_limit_size 512k
    flush_interval 5s
    total_limit_size 4GB
    flush_thread_count 2
    overflow_action drop_oldest_chunk
    retry_forever true
    retry_wait 5s
    retry_exponential_backoff_base 5
  </buffer>
</match> is not used.
2023-08-18 06:13:08 +0000 [warn]: parameter 'ssl_verify_peer' in <match **>
  @type http
  time_as_integer true
  endpoint "http://domain_name:24284"
  serializer json_lines
  add_newline true
  http_method post
  ssl_version TLSv1_2
  ssl_cert_path /fluentd/etc/ssl/client.crt
  ssl_key_path /fluentd/etc/ssl/client.key
  ssl_verify_peer true
  <buffer>
    @type "file"
    path "/var/log/fluentd_application/kubernetes.internal.application.buffer"
    chunk_limit_size 512k
    flush_interval 5s
    total_limit_size 4GB
    flush_thread_count 2
    overflow_action drop_oldest_chunk
    retry_forever true
    retry_wait 5s
    retry_exponential_backoff_base 5
  </buffer>
</match>

Could someone please help us to resolve the issue? or suggest the ssl fields which can be used for http plugin

To Reproduce

Install the fluentd with http plugin with ssl with the above configuration

Expected behavior

there should not be any error in the connection to http plugin.

Your Environment

- Fluentd version: fluentd 1.11.2
- Operating system:
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
- Kernel version:5.10.179-168.710.amzn2.x86_64

Your Configuration

<source>
      @type monitor_agent
      bind 0.0.0.0
      port 24220
    </source>

    <source>
      @type tail
      #@id in_tail_fluentd_logs
      # @label @fluentdlogs
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
     <parse>
       #@type json
       #time_format %Y-%m-%dT%H:%M:%S.%NZ
       @type multi_format
       <pattern>
         format json
         time_key time
         keep_time_key true
      #time_format %Y-%m-%dT%H:%M:%S.%L %Z
         time_format %Y-%m-%dT%H:%M:%S.%NZ
       </pattern>
       <pattern>
         format none
         time_key time
         keep_time_key true
      #time_format %Y-%m-%dT%H:%M:%S.%L %Z
         time_format %Y-%m-%dT%H:%M:%S.%NZ
       </pattern>
     </parse>
    </source>

    <filter kubernetes.**>
      @type kubernetes_metadata
      @id filter_kube_metadata
      cache_size 20000
      cache_ttl 7200
      watch false
    </filter>

    <filter kubernetes.**>
      @type record_transformer
      enable_ruby true
      <record>
        cluster_name "#{ENV['CLUSTERNAME']}"
        namespace_name ${record["kubernetes"]["namespace_name"]}
        node_name ${record["kubernetes"]["host"]}
        pod_name ${record["kubernetes"]["pod_name"]}
        container_name ${record["kubernetes"]["container_name"]}
        #logstash_apikey "#{ENV['LOGSTASH_APIKEY']}"
        #logstash_apiid "#{ENV['LOGSTASH_APIID']}"
        environment "DEV"
      </record>
      remove_keys stream,containerd_prefix,docker, kubernetes
    </filter>

    <filter kubernetes.**>
      @type grep
      <exclude>
        key message
        pattern /DEBUG/
      </exclude>
    </filter>

    <filter kubernetes.**>
      @type grep
      <exclude>
        key message
        pattern /Debug/
      </exclude>
    </filter>

    <filter kubernetes.**>
      @type grep
      <exclude>
        key message
        pattern /debug/
      </exclude>
    </filter>

    <match kubernetes.**{_kube-system_nginx-ingress}**>
      @type http
      time_as_integer true
      <buffer>
        @type file
        path /var/log/fluentd_system/kubernetes.internal.system.buffer
        chunk_limit_size 512k
        flush_interval 5s
        total_limit_size 4GB
        flush_thread_count "#{ENV['FLUENTD_THREADS']}"
        overflow_action drop_oldest_chunk
        retry_forever true
        retry_wait 5s
        retry_exponential_backoff_base 5
      </buffer>
      endpoint http://domain-name:24284
      serializer json_lines
      add_newline true
      http_method post
      ssl_version TLSv1_2
      ssl_cert_path "/fluentd/etc/ssl/client.crt"
      ssl_key_path "/fluentd/etc/ssl/client.key"
      ssl_verify_peer true
    </match>

    <match kubernetes.**{_kube-public_,_kube-system_,_kube-metrics_,_kube-node-lease_,_ibm-**_,_ibmro-ops_,_nag-mon_}**>
      @type null
    </match>

    <match **>
      @type http
      time_as_integer true
      <buffer>
        @type file
        path /var/log/fluentd_application/kubernetes.internal.application.buffer
        chunk_limit_size 512k
        flush_interval 5s
        total_limit_size 4GB
        flush_thread_count "#{ENV['FLUENTD_THREADS']}"
        overflow_action drop_oldest_chunk
        retry_forever true
        retry_wait 5s
        retry_exponential_backoff_base 5
      </buffer>
      endpoint http://domain-name:24284
      serializer json_lines
      add_newline true
      http_method post
      ssl_version TLSv1_2
      ssl_cert_path "/fluentd/etc/ssl/client.crt"
      ssl_key_path "/fluentd/etc/ssl/client.key"
      ssl_verify_peer true
    </match>

Your Error Log

2023-08-18 06:13:08 +0000 [warn]: #0 failed to flush the buffer. retry_time=1 next_retry_seconds=2023-08-18 06:13:13 +0000 chunk="60329c60f47dd24c9e2dd618d9879eca" error_class=EOFError error="end of file reached"
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/protocol.rb:225:in `rbuf_fill'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/protocol.rb:191:in `readuntil'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/protocol.rb:201:in `readline'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http/response.rb:40:in `read_status_line'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http/response.rb:29:in `read_new'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:1509:in `block in transport_request'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:1506:in `catch'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:1479:in `request'
  2023-08-18 06:13:08 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.2/lib/fluent/plugin/out_http.rb:229:in `block in send_request'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:920:in `start'
  2023-08-18 06:13:08 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:605:in `start'
  2023-08-18 06:13:08 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.2/lib/fluent/plugin/out_http.rb:228:in `send_request'
  2023-08-18 06:13:08 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.2/lib/fluent/plugin/out_http.rb:136:in `write'
  2023-08-18 06:13:08 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.2/lib/fluent/plugin/output.rb:1133:in `try_flush'
  2023-08-18 06:13:08 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.2/lib/fluent/plugin/output.rb:1439:in `flush_thread_run'
  2023-08-18 06:13:08 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.2/lib/fluent/plugin/output.rb:461:in `block (2 levels) in start'
  2023-08-18 06:13:08 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2023-08-18 06:13:13 +0000 [warn]: #0 failed to flush the buffer. retry_time=2 next_retry_seconds=2023-08-18 06:13:40 +0000 chunk="60329c60f47dd24c9e2dd618d9879eca" error_class=EOFError error="end of file reached"
  2023-08-18 06:13:13 +0000 [warn]: #0 suppressed same stacktrace
2023-08-18 06:13:13 +0000 [warn]: #0 failed to flush the buffer. retry_time=3 next_retry_seconds=2023-08-18 06:15:07 +0000 chunk="60329c5c2e8293464e499f083c22a977" error_class=EOFError error="end of file reached"
  2023-08-18 06:13:13 +0000 [warn]: #0 suppressed same stacktrace

Additional context

No response

Answered by daipom

Aug 21, 2023

The documentation is here.
Your Fluentd version is old, but the SSL parameters seem to be not changed.
Please try these parameters.

View full answer

daipom · 2023-08-18T09:18:17Z

daipom
Aug 18, 2023
Maintainer

I have moved this issue to the discussion since this is a question rather than a bug report. (contributing guidelines)

0 replies

daipom · 2023-08-18T09:20:17Z

daipom
Aug 18, 2023
Maintainer

Please confirm the setting.
Some parameters about TLS seem wrong.

Please check these logs and try the correct option name.

parameter 'ssl_key_path' in xxx
...
is not used

2 replies

prashant407 Aug 18, 2023
Author

Thanks for the response @daipom
Could you please suggest us the SSL parameters used for http?
If there is any documentation that would be helpful

daipom Aug 21, 2023
Maintainer

The documentation is here.
Your Fluentd version is old, but the SSL parameters seem to be not changed.
Please try these parameters.

Answer selected by prashant407

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fluentd with http throwing ENDOFFILE error #4274

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Fluentd with http throwing ENDOFFILE error #4274

Uh oh!

prashant407 Aug 18, 2023

Describe the bug

To Reproduce

Expected behavior

Your Environment

Your Configuration

Your Error Log

Additional context

Replies: 2 comments · 2 replies

Uh oh!

daipom Aug 18, 2023 Maintainer

Uh oh!

daipom Aug 18, 2023 Maintainer

Uh oh!

prashant407 Aug 18, 2023 Author

Uh oh!

daipom Aug 21, 2023 Maintainer

prashant407
Aug 18, 2023

Replies: 2 comments 2 replies

daipom
Aug 18, 2023
Maintainer

daipom
Aug 18, 2023
Maintainer

prashant407 Aug 18, 2023
Author

daipom Aug 21, 2023
Maintainer