Mimir Distributor OOM

[KrishnaJyothika]

Hi Team,

We are landing in distributors OOMKilled continuously, issue is causing due to one ingester silent restart

1. The issue started with one single ingester silent restart, which didn’t come up immediately due to node issue
2. Distributors attempting to reach the failed ingester causing distributors OOM
3. Distributors are completely going down within 1-2min of ingester restart
4. We can see distributors unavailable in logs and retires at source
5. When few of the distributors are trying to recover – spike in ingestion rate due to retires are further causing OOM in distributors

Distributor logs
ts=2025-02-28T15:13:24.644631208Z caller=pool.go:250 level=warn msg="removing ingester failing healthcheck" addr=10.128.54.5:9095 reason="rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: read tcp 10.128.148.12:47456->10.128.54.5:9095: read: connection reset by peer""
ts=2025-02-28T15:13:33.722860622Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'esobb-mimir-distributor-enterprise-fc74dcccb-2qwsk-b94240a0' from=10.128.143.7:7946"

Complete write path is going down leading to data loss, can someone please provide a solution for this.

[KrishnaJyothika]

Any update on this please?

[aknuds1]

Converting this to a discussion, as there's no Mimir bug AFAICT.

[deniszh]

I also noticed that e.g. during deployments and ingester restart distributors consume more memory. I think that's unavoidable and you just need add extra memory to distributors.
I tried to catch heap during issues but found nothing in it, strange

[gpadidala]

We have updated the memory as recommended, we will monitor and update if there are any changes.