-
Couldn't load subscription status.
- Fork 247
Description
Is your feature request related to a problem? Please describe.
This was discovered on a Slack thread I started to troubleshoot why I wasn't getting any CPU metrics in my krr reports.
Turns out that krr was making queries against Prometheus to gather pods related to k8s objects:
https://github.com/robusta-dev/krr/blob/main/robusta_krr/core/integrations/prometheus/metrics_service/prometheus_metrics_service.py#L268
This code assumes that the labels that should be used are called pod but that can in some instances not be the case. For example, Prometheus has a scraper config called honor_labels which, if set to false, can rewrite the names of labels being used by adding an exported_ prefix to it (so pod becomes exported_pod).
This also applies to other label names such as job_name, namespace and multiple other.
Describe the solution you'd like
I would like to be able to run krr in such a way that it detects whether labels have been rewritten (renamed) and automatically adapt to that.
Describe alternatives you've considered
I've created a PR that solves this for my particular case where it was only the pod label that needed to be adjusted. This was done by providing the label name via an environment variable.
Are you interested in contributing a PR for this?
I don't really feel like I have enough knowledge of what is happening in the load_pods function, or how the label renaming in the Prometheus scraper works, or of PromQL in general to be able to contribute this myself. But if someone can help with that and prefers that I make the PR rather than doing it themselves, I'd be happy to help.
Additional context
One idea I had, since this seems to mostly have something to do with the honor_labels setting of the Prometheus scraper which seemingly renames label in a predictable way (prepending exported_ to the name), was whether krr should provide an option to prefer labels that are named like that if they exist. So if the label pod exists but also the label exported_pod, krr should prefer to use the latter. This could then also apply to other labels. This should mean that if the honor_labels setting is set to false and Prometheus scraper rewrites the label names, krr would pick that up. I don't know if doing this is really feasible though.