Ingester CPU spikes

[LasseHels]

Hey folks

We run ingesters in three zones. Typically, average CPU consumption per ingester in a zone is ~3. However, we observe that occasionally, CPU will spike to ~6 for a minute or two, and then return to normal without any manual intervention. It looks like this:

In the screenshot above, we see CPU usage soar briefly, even though write requests, read requests and memory series are unchanged.

The pattern is also noticeable on our node level metrics:

All nodes included in the screenshot above run exclusively ingester pods (not counting DaemonSet pods).

Looking at our disk metrics, we see increased read activity on some ingester nodes, but not all:

Write activity looks unchanged.

In the Mimir / Writes Resources dashboard, we can see that it is the CPU usage of all ingesters that spikes:

We also see that memory usage goes up by quite a bit across the three zones. It comes back down again, but not quite to the level it was at pre-spike:

We've taken a look at the following and did not find any obvious smoking guns:

• Ingester logs.
• Node syslogs.
• Eviction logs (i.e., did any evictions take place around the time of the spike).

We'd be curious to get the Mimir team's input on what leads to chase to get to the bottom of the spikes. As you can see from the resources dashboard, our ingesters are currently over-provisioned by quite a bit, and we would like to right-size them, but are concerned about these CPU spikes.

EDIT: Forgot to include our Mimir version 🤦. We run Mimir 2.11.0.

[LasseHels]

Any chance for some input on this issue? We spent some more time investigating this and have come up empty-handed again.