Improve performance when querying from Grafana #35
Description
Context
Depending on the volume of data requested, the performance of the backend when interfacing with Grafana can be flaky.
For example, fetching the same visualisation twice with the same time range and variable selections seems to re-trigger a request and response cycle, when ideally the response should be cached.
Proposal
We have identified that the compute usage query, with a time resolution/step of 5m, is the bottleneck.
- ensure the prometheus compute usage query is cached and retrievable from Grafana
- configure a prometheus recording rule to store results from the compute usage query so that it's readily available for fetching from Grafana
Definition of Done
- Dashboard panels take at most 10 seconds to load, especially when readjusting time ranges and variable selectors