Cannot override Grafana storage volume with PVC using Jsonnet overlays in kube-prometheus

[morkeleb]

It is not possible to override the default emptyDir storage for Grafana with a PersistentVolumeClaim (PVC) using Jsonnet overlays or parameters when using the upstream kubernetes-grafana/grafana.libsonnet mixin (as used in kube-prometheus).

The mixin hardcodes the storageVolume as emptyDir: {} via a local variable:

This local variable is then used in the volumes array for the Deployment, and there is no way to override it via parameters or overlays from a consumer (such as kube-prometheus). Attempts to overlay or set storageVolume or _config.storageVolume in the parameters have no effect.

To Reproduce
Try to overlay the Grafana deployment to use a PVC for /var/lib/grafana by setting storageVolume or _config.storageVolume in the parameters or overlays.
Build manifests with kube-prometheus or your own Jsonnet stack.

Observe that the generated Deployment for Grafana always uses emptyDir for the grafana-storage volume, regardless of overlay or parameter.

Expected behavior
It should be possible to override the storage type for the grafana-storage volume (e.g., to use a PVC) via parameters or overlays, as is possible for Prometheus and Alertmanager in kube-prometheus.

Why this matters
Persistent storage is required for Grafana dashboards, plugins, and state. Not being able to use a PVC means data is lost on pod reschedule or node failure.

Proposed solution
Expose storageVolume as a parameter in the mixin, so it can be overridden by consumers (e.g., via overlays or parameters in kube-prometheus). For example:

This would allow overlays or parameters to set a PVC for persistent storage.