Events dropped Vector 0.44.0 hemlchart 0.40.0

[roymartinez-mollie]

I'm running into a connection error that's affecting event processing, and I wanted to share some context for troubleshooting and discussion.

When the issue hits, I see this debug log:

"level": "DEBUG",
"message": "connection error: connection closed before message completed",
"target": "hyper::server::server::new_svc"

Shortly after, we also get this:

"message": "Events dropped",
"internal_log_rate_limit": true,
"target": "vector_common::internal_event::component_events_dropped",
"intentional": false,
"level": "ERROR",
"timestamp": "2025-04-14T14:06:24.760730Z",
"count": 1446,

One thing that stands out is a mismatch between the component_discarded_events_total internal metric and the count reported in the error log.

My concern here is that the dropped events don’t appear to be intentional, and this is especially worrying because we’ve seen noticeable metrics loss when onboarding high-volume projects—without any clear root cause or telemetry from Vector that helps us pinpoint what’s going wrong.

Somewhat related: we’ve also observed symptoms of cardinality loss—tags going missing from metrics even though no VRL was modifying them. This, along with the event loss, eventually led us to offboard one of our larger projects. We did manage to correct the cardinality issue by upping CPU limits and simplifying our pipeline. Originally, the flow was source -> filter -> remap -> remap -> sink, and we simplified it to source -> remap -> sink, dropping redundant VRL blocks in the process. That helped stabilize things, but we’re still in the dark about the root cause of the metric loss and dropped events.

For additional context on our setup:

We're terminating TLS directly at the Vector pod. Traffic comes in via passthrough Network Load Balancers (Layer 4), which don’t do SSL termination but just forward encrypted packets to the pods. The TLS handshake and decryption happen inside the Vector container. This lets us keep end-to-end encryption and preserve client connection info, while still balancing traffic inside the VPC. We are using datadog_agent as soource

Happy to hear thoughts or if anyone’s seen similar behavior.

[pront]

Hello, nothing really stands out from your description / setup.

Can you perhaps have plot the metrics over time? This might provide valuable insights.

https://vector.dev/guides/developer/debugging/#visualizing-and-querying-internal-metrics