Back Pressure Metric

[satellite-no]

Is there any metric that would show if there is any back-pressure being applied upstream?

I have scenerio where I'm using s3 source to process data then write back to s3 with the s3 sink and if my nodes go from 5 down to 3 source SQS queue startes growing because we arent processing data fast enough. But on all nodes CPU is low, memory is low, disk is low, all metrics from Vector are low and input / output match. The s3 sink just is not writing data fast enough but no disk buffer grows its applying back pressure upstream but I cant find a metric that says "this here is the slowdown".

[pront]

Hi @satellite-no, utilization and buffer_ metrics should be good indicators. These are all internal to Vector though. You can also experiment with https://vector.dev/docs/reference/configuration/sinks/aws_s3/#request settings if you haven't already.

[satellite-no]

Ok, this makes sense and works. The issue I was hitting was I normally use disk buffer but we are hitting a bufferToSmall issue when using disk buffer v2 similar to this 19759 and 18346 and I have to swap from disk buffer to memory buffer which Im just using defaults of 500 events which was causing the "unseen" buffer issues since 500 events in memory is pretty small.

[satellite-no]

When Ingesting from s3 and theres 1 line thats multiple events making it "really large" it will trigger the BuffferToSmall condution which doesnt make sense since my disk buffer is like 1TB.