Intermittent recording rule failures during WAL operation

[rishabhkumar92]

Hi Team,

We are seeing intermittent recording rule failures (Ruler -> Ruler QFE -> Ruler Querier -> Ingesters) during WAL operations.
We adjusted head compaction interval to 15 minutes to spread out the WAL operations and thus spreading out the disk write/read bandwidth. Are there any side effect of spreading it to 30 minutes, what are other recommendations to solve for this backpressure issues?

Current Mimir version: 2.14.1
Infrastructure: K8s