Add "User Group Diagnostics" Grafana dashboard #6065
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hey @GeorgianaElena ! Just requested a review from you to just check if you think this is okay to implement across all of our clusters. I think it should be fine, but I would value your infra eng experience here to give a quick green/red light :) |
|
Actually, not ready for review yet, just found a bug! |
There was a problem hiding this comment.
Great work @jnywong <3 ! I didn't pay too much attention to the queries, so this review is more from an infra perspective.
If you look at
infrastructure/deployer/commands/grafana/deploy_dashboards.py
Lines 58 to 72 in a526d62
The dashboards here are deployed on aws only and are categorized as cost-related dashboards.
In my opinion, ideally this work should be instead upstream, in jupyterhub/grafana-dashboards. I believe 2i2c uses that repo most intensively so we shouldn't be impacting lots of people if we were to merge this ourselves.
If you want to be on the safe side, we could also host them under 2i2c for a month lets say, validate that they are correct, then upstream them. In this time, we would also open an upstream issue about this intention.
If we are to go with option two, then we need to update the deployer command and the general structure of this directory because it looks like it's growing to be more than cost-related dashboards.
What do you think?
|
Your infra perspective is exactly what I needed :) Yeah, eventually I would like to upstream this, but like you said, to be on the safe side we can validate and host this on the 2i2c side while we wait. I'll make changes to this PR to update the deployer command then. Thanks! |
This was referenced May 19, 2025
Closed
|
Merging this PR will trigger the following deployment actions. Support deploymentsNo support upgrades will be triggered Staging deployments
Production deployments
|
|
Deployer updated, dashboards moved to 2i2c hosted repo and upstream PR waiting in draft 👍 |
|
this is perfect @jnywong ❤️ I feel comfortable merging your PR upstream as well (I have the rights) if you feel confident too. Just let me know, otherwise feel free to merge this one and maybe also open an internal 2i2c tracking issue to review+merge upstream PR after some testing on our infra to make sure we don't loose track of it. I believe the biggest challenge is having the 2i2c repo diverge and having to maintain and contribute in two places. |
|
Great, thank you @GeorgianaElena! I agree, we do not want these two repos to diverge and I will let you know when #5315 is ready to be merged and upstreamed once I am happy. To not lose track, I have added this task to the DoD in the parent initiative: #5315 |
sunu
pushed a commit
to sunu/infrastructure
that referenced
this pull request
May 27, 2025
* Add new user group diagnostics dashboard and update user diagnostics * Use unescaped $user_name from pod annotation instead of $user_pod from pod label * Add home dir usage panel for groups * Use unescaped username for home dir usage panel * Move dashboards to https://github.com/2i2c-org/grafana-dashboards * Revert variables for cloud cost dashboard * Point deployer to 2i2c hosted dashboards
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a new "User Group Diagnostics" Grafana dashboard that complements the "User Diagnostics Dashboard" to show resource usage aggregated by user group.
Requires jupyterhub-groups-exporter to be set up on the hub for the dashboard to work, but if not, then see below. I have created a fork of the upstream dashboards while we validate implementation across 2i2c-hosted hubs.
For this to universally work across all our infrastructure, I decided to separate the user group dashboard entirely from user name based aggregation. This is because if I combined both user name and user group into one dashboard/PromQL query, then it would break the dashboard for hubs that do not have jupyterhub-groups-exporter set up because the
jupyterhub_user_group_infometric is unavailable to the PromQL. Therefore, if a hub does not have jupyterhub-groups-exporter set up, then the "User Diagnostics" dashboard will work as normal but the "User Group Diagnostics" dashboard will show no data.The "User Diagnostics" dashboard included in this PR differs from the upstream version of user.jsonnet, because the upstream version is technically a "Pod Diagnostics" dashboard. This PR aggregates pod-level data on a per user basis and uses unescaped usernames from metric
kube_pod_annotations, rather than limited charactersets fromkube_pod_labels.Note
Metrics are available as a time series from the date of initially manually deploying the
jupyterhub-groups-exporterservice (therefore some PromQL in this PR is invalid prior to deployment, say, since before 18 May 2025). If you see an execution error in the dashboard, try selecting a more recent time window.Ref: #5983