Connection errors with filebeat as agent and logstash source

[camilisette]

We currently have Filebeat agents picking up logs from different hosts and sending logs to Vector using the logstash source. It works fine for a few days, but after some days of running, we start to see an increase in connection errors in the Filebeat logs.

Errors like:

• "Failed to publish events caused by: read tcp XXX.XX.XXX.XX:XXXX->XXX.XX.XXX.XX:XXXX: i/o timeout"
• "Failed to publish events caused by: write tcp XXX.XX.XXX.XX:XXXX->XXX.XX.XXX.XX:XXXX: use of closed network connection"
• "Failed to publish events caused by: client is not connected"
• "Failed to publish events caused by: EOF filesdf"
• "File cleanup state update error: input outlet closed"
• "Failed to publish events caused by: invalid sequence number received (seq=1024, expected=6)"

Some of the things we noticed:

• With an increase of these errors, we are seeing duplicate logs, we suspect it's because filebeat resends them because either Vector didn't ack or didn't ack in time.
• Not seeing any utilization issues, infact utilization is low. Checked cpu, memory, file descriptor limit, network i/o, storage i/o, etc.
• Important: Restarting the Vector services gets rid of the errors for a couple days, then it resurfaces after it's been running for some days.

Details:
Agent: Filebeat sending logs using the logstash output
Vector: Receives logs using the logstash source
Vector version: 0.42.0, running on EC2

Any ideas on what we could check? Could there be some internal limit in Vector that we're hitting?

[pront]

Hi @camilisette,

It sounds like a stale TCP connection issue. It's hard to diagnose such issues that arise after long running operations. I would use available debugging tools (see here) to get an overview of Vector's internal metrics around the time the errors start appearing vs the metrics during normal operation.

[adam-kral-wb]

Hi, I just wanted to share that we are hitting a similar issue, under a bit different conditions.
When we deploy a new version of vector, we immediately see similar logs from filebeat:

2025-05-13T13:50:11.749Z	ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: invalid sequence number received (seq=680, expected=127)
2025-05-13T13:50:11.749Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2025-05-13T13:50:11.749Z	INFO	[publisher]	pipeline/retry.go:223	  done
2025-05-13T13:50:11.749Z	ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: invalid sequence number received (seq=680, expected=127)
2025-05-13T13:50:11.749Z	INFO	[publisher]	pipeline/retry.go:213	retryer: send wait signal to consumer
2025-05-13T13:50:11.749Z	INFO	[publisher]	pipeline/retry.go:217	  done
2025-05-13T13:50:11.749Z	ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: invalid sequence number received (seq=680, expected=127)
2025-05-13T13:50:11.749Z	INFO	[publisher]	pipeline/retry.go:213	retryer: send wait signal to consumer
2025-05-13T13:50:11.749Z	INFO	[publisher]	pipeline/retry.go:217	  done
2025-05-13T13:50:11.785Z	ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: client is not connected
2025-05-13T13:50:13.110Z	ERROR	[publisher_pipeline_output]	pipeline/output.go:180	failed to publish events: client is not connected
2025-05-13T13:50:13.110Z	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to backoff(async(tcp://some.url:443))
2025-05-13T13:50:13.110Z	INFO	[publisher]	pipeline/retry.go:213	retryer: send wait signal to consumer
2025-05-13T13:50:13.110Z	INFO	[publisher]	pipeline/retry.go:217	  done
2025-05-13T13:50:13.155Z	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to backoff(async(tcp://some.url:443)) established
2025-05-13T13:50:13.155Z	INFO	[publisher]	pipeline/retry.go:213	retryer: send wait signal to consumer
2025-05-13T13:50:13.155Z	INFO	[publisher]	pipeline/retry.go:217	  done
2025-05-13T13:50:13.160Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2025-05-13T13:50:13.160Z	INFO	[publisher]	pipeline/retry.go:223	  done
2025-05-13T13:50:14.213Z	ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: invalid sequence number received (seq=807, expected=217)
2025-05-13T13:50:14.214Z	ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: invalid sequence number received (seq=807, expected=217)
2025-05-13T13:50:14.214Z	ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: invalid sequence number received (seq=807, expected=217)
2025-05-13T13:50:14.214Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2025-05-13T13:50:14.214Z	INFO	[publisher]	pipeline/retry.go:223	  done
2025-05-13T13:50:14.214Z	INFO	[publisher]	pipeline/retry.go:213	retryer: send wait signal to consumer
2025-05-13T13:50:14.214Z	INFO	[publisher]	pipeline/retry.go:217	  done
2025-05-13T13:50:14.214Z	INFO	[publisher]	pipeline/retry.go:213	retryer: send wait signal to consumer
2025-05-13T13:50:14.214Z	INFO	[publisher]	pipeline/retry.go:217	  done
2025-05-13T13:50:14.214Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2025-05-13T13:50:14.214Z	INFO	[publisher]	pipeline/retry.go:223	  done
2025-05-13T13:50:14.238Z	ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: client is not connected
2025-05-13T13:50:14.238Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2025-05-13T13:50:14.239Z	INFO	[publisher]	pipeline/retry.go:223	  done
2025-05-13T13:50:15.296Z	ERROR	[publisher_pipeline_output]	pipeline/output.go:180	failed to publish events: client is not connected
2025-05-13T13:50:15.296Z	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to backoff(async(tcp://some.url:443))
2025-05-13T13:50:15.297Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2025-05-13T13:50:15.297Z	INFO	[publisher]	pipeline/retry.go:223	  done
2025-05-13T13:50:15.361Z	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to backoff(async(tcp://some.url:443)) established

In this case, we switched from logstash to vector, but similar behavior was observed when upgrading to newer vector version.

We see similar symptoms, mainly duplicated logs (around 30-50% increase of our usual traffic).

Ther aren't any out of usual errors on Vector side.

Similar setup as mentioned above:
Agent: Filebeat sending logs using the logstash output - version 7.17.18
Vector: Receives logs using the logstash source
Vector version: 0.46.0
Deployment: multiple vector pods deployed behind a load balancer on kubernetes

Any advise would be appreciated 🙏 In the meantime, I'll try to check if we can get anything more specific with the vector debugging tools.

[adam-kral-wb]

Also might be worth mentioning that we have tls enabled between filebeat and vector

sources:
  logstream:
    type: logstash
    address: 0.0.0.0:443
    tls:
      enabled: true
      ca_file: "${TLS_CA}"
      crt_file: "${TLS_CERT}"
      key_file: "${TLS_KEY}"
      key_pass: "${VECTOR_KEY_PASSPHRASE}"
      verify_certificate: true
      client_metadata_key: "client_metadata"

[camilisette]

Have you tried enabling vector metrics to further investigate? We have enabled what we can on our end, but unfortunately we are not seeing anything, even on the host metrics... So we are stumped on what could be causing this.

[camilisette]

Also, the connection errors maybe can't be fully avoided for a number of reasons. I think it should be okay since filebeat has an at least once delivery guarantee. However, I believe the "at least once delivery" guarantee is also the reason why we are getting duplicated logs....

[camilisette]

Also curious if you've tried using Vector as an agent instead of Filebeat. We are thinking of testing it, but haven't done so yet.

[adam-kral-wb]

Have you tried enabling vector metrics to further investigate

we are able to see increased rate of "writer failed" errors from the logstash source:

source{component_kind="source" component_id=logstream component_type=logstash}:connection{peer_addr=some_addr}: vector::internal_events::tcp: Error writing acknowledgement, dropping connection. error=unexpected EOF error_code="ack_failed" error_type="writer_failed" stage="sending"

the theory is that the acknowledgment between the logstream & vector fails and logstream tries to resend the events. but I don't have anything more tangible yet

Also curious if you've tried using Vector as an agent instead of Filebeat.

Sadly this is not realistic to do in some reasonable time period. The logstreams run "externally" and upgrading them to vector agents is gonna take effort on our customers' side.