Impact of Loki 429 Errors on All Tenants in a Shared Vector Pipeline

[LinTechSo]

We are experiencing issues where a 429 (Too Many Requests) error from Grafana Loki is impacting all tenants in a shared Vector pipeline. This happens when one of our tenants generates a high volume of logs, which leads to throttling or rate-limiting on Loki's ingester. The problem is that all tenants in the same Vector pipeline are affected, even though logs are isolated by tenant via the X-Scope-OrgID header.

I thought that adding the X-Scope-OrgID header in the request would isolate the batch request for each tenant. Is that correct?

When one of the tenants gets a 429 error, how does Vector.dev behave for other tenants in the shared Vector pipeline?

Assuming the impact on all tenants is correct behavior in one vector pipeline, and I know that one of the tenants in my Vector pipeline will get a 429 error, what is the best configuration for this scenario? should i seperate the sinks? or should i seperate the vectors or etc

I would appreciate it ifyou could help me figure this out.

[pront]

Hi @LinTechSo, I suspect the following: the X-Scope-OrgID header isolates tenant data on the Loki side, but when logs from multiple tenants are batched together in one request there is no tenant isolation. This is likely why all tenants in your shared Vector pipeline are affected when one tenant triggers a 429 error.

Without having the full context, I can suggest:

• introducing more Vector pipelines and load balance events
• introduce throttling so that your backend isn't overwhelmed