Reduce alerts due to temporary compaction #2497
Unanswered
martin-aders
asked this question in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Whenever max disk retention size is reached, a compaction is triggered. The compaction requires additional space temporarily until disk space may be freed. Having this alert fire "for: 1m" may cause false alerts during this compaction period unless you reserve a larger amount of disk size just for the short period when compaction occurs. On a real scenario, a "for: 10m" would be enough to give time to get disk space freed. May be a matter of taste, though. There may be alternative ways to increasing the "for" time too, to cover systems with slower disks.
Log compaction example, showing percentage of free disk from ~40% down to almost 0%:

Code reference: https://github.com/prometheus-operator/kube-prometheus/blob/d04c1e140ad406cd14c0040a2e891d389b69814b/manifests/kubernetesControlPlane-prometheusRule.yaml#L380C1-L380C14
Documentation: https://prometheus.io/docs/prometheus/latest/storage/#compaction
Beta Was this translation helpful? Give feedback.
All reactions