Vector doesnt support high load of traffic (kubernetess_logs)

[ALNBAND]

Hi,

we need to support load of 6000 RPS for our load,
each request is with size of 600 byte.
we are using vector 0.21
machine type https://docs.microsoft.com/en-us/azure/virtual-machines/fsv2-series (we tried Standard_F4s_v2 & Standard_F8s_v2 ) follows the vector sizing guide for CPU intensive machine type.
we are using 3 vector instance on 3 machine (overall 24CPU) ,each vector instance should support 2000RPS.
The sync is k8s logs to a file( read specific pod logs and send it to a file)

The problem:

Currently vector is losing data, for more than 60% of the request we dont see on the file,
the vector pods is not overloaded (we verify in Prometheus) .

when reducing the load to 300RPS per instance all the data is written the the file ( no data loss),
what could be the issue? please advice

[jszwedko]

Just noting for others that we ended up discussing this in Discord here: https://discord.com/channels/742820443487993987/989197470842118164/989206058427879465

[JeanMertz]

I'll mark this as resolved, given that the discussion continue in Discord, but you are welcome to leave a comment if you disagree.