Vector 0.44 version : vector to vector traffic error

[shamj]

Vector source and Vector destination are on vector 0.44 version. While sending the traffic from vector sink to vector source , seeing the below error: Please advise.
ERROR source{component_kind="source" component_id=vector_vector_data component_type=vector}:grpc-request{grpc_service="vector.Vector" grpc_method="PushEvents"}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1000 reason="Source send cancelled." internal_log_rate_limit=true

[pront]

Hello @shamj, did you update to a new Vector version recently? Did you change your config? This can also be a network instability issue on your environment.

I highly recommend taking a look at our debugging guide if you didn't already: https://vector.dev/guides/developer/debugging/

[shamj]

Are there any limitations on GRPC?

[pront]

Nothing specific for Vector sources. You can try sending sample GRPC curl request to rule out networking failures. Also, inspect the component_errors_total and component_discarded_events_total metrics.

[shamj]

I tried adding debug to logs. Noticed this error on vector source:

"message":"stream error: connection error: stream closed because of a broken pipe",

Also no drop in vector sink , However events are dropped on vector source. Some of the traffic is processed to next transforms/sinks. We got success sending some of the traffic, but when traffic is increased , error is seen.

Component_discarded_events_total shows discarded events on source vector.

[pront]

Sounds like this originates in the GRPC internal and you might be hitting a limit or your environment is misconfigured. Can you try to debug on a clean deployment?

[shamj]

yes, I did redeploy everything, started new pods , still same error