What does "Source send cancelled" mean?

[tronboto]

We started seeing this error in the opentelemetry source and I don't really know how to interpret it:

{
  "timestamp": "2025-06-04T16:05:16.841021Z",
  "level": "ERROR",
  "message": "Events dropped",
  "intentional": false,
  "count": 1,
  "reason": "Source send cancelled.",
  "internal_log_rate_limit": true,
  "target": "vector_common::internal_event::component_events_dropped",
  "span": {
    "grpc_method": "Export",
    "grpc_service": "opentelemetry.proto.collector.logs.v1.LogsService",
    "name": "grpc-request"
  },
  "spans": [
    {
      "component_id": "source_opentelemetry",
      "component_kind": "source",
      "component_type": "opentelemetry",
      "name": "source"
    },
    {
      "grpc_method": "Export",
      "grpc_service": "opentelemetry.proto.collector.logs.v1.LogsService",
      "name": "grpc-request"
    }
  ]
}

[pront]

Hi @tronboto, from the source docs:

// It's possible that the caller stops polling this future while it is blocked waiting
// on `self.send()`. When that happens, we use `UnsentEventCount` to correctly emit
// `ComponentEventsDropped` events.

/// UnsentEvents tracks the number of events yet to be sent in the buffer. This is used to
/// increment the appropriate counters when a future is not polled to completion. Particularly,
/// this is known to happen in a Warp server when a client sends a new HTTP request on a TCP
/// connection that already has a pending request.
///
/// If its internal count is greater than 0 when dropped, the appropriate [ComponentEventsDropped]
/// event is emitted.

This can indicate (among other things) a backpressure issue.

[tronboto]

Thanks @pront . Do you mean backpressure coming from the sinks? Because that's not what we're seeing. Any tips on investigating this further?

vector_component_received_events_total

vector_component_discarded_events_total

vector_adaptive_concurrency_back_pressure_bucket

vector_buffer_events (buffer.max_events is 100000)