Skip to content

[Dashboard] Grafana dashboard iframe doesn't set cluster name, causing "No Data" for multi-cluster setups #52579

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hedeesaa opened this issue Apr 24, 2025 · 0 comments
Labels
bug Something that is supposed to be working; but isn't clusters dashboard Issues specific to the Ray Dashboard kuberay Issues for the Ray/Kuberay integration that are tracked on the Ray side triage Needs triage (eg: priority, bug/not-bug, and owning component) usability

Comments

@hedeesaa
Copy link

hedeesaa commented Apr 24, 2025

What happened + What you expected to happen

Issue:

We're running multiple Ray clusters in our Kubernetes environment. After the following merges:

Grafana now supports displaying metrics from multiple clusters in a single dashboard. However, we've encountered a client-side issue where the iframe embeds don't pass the Cluster (cluster name) as a variable. This results in "No Data" or a fallback to the default cluster being shown in the Grafana panels.

For example, the ClusterUtilization panel in the Ray Dashboard references the Cluster variable:

<Box
component="iframe"
title="Cluster Utilization"
sx={{ flex: 1 }}
src={`${grafanaHost}${path}&refresh&timezone=${currentTimeZone}${timeRangeParams}&var-SessionName=${sessionName}`}
frameBorder="0"
/>

Proposed Solution:

Update the dashboard iframes on the client side to include the cluster name in the query string, e.g.:

&var-Cluster=${cluster}

Related Workaround Attempt:

PR ray#51396 attempted to address this by enabling the "Include All" option in the Grafana dashboard. However, the actual kuberay/config/grafana/* configs haven't been updated accordingly.

Suggestion:

Ensure the iframe logic in the dashboard client app sets the appropriate var-Cluster parameter, and update the kuberay/config/grafana templates to reflect the multi-cluster setup.

Versions / Dependencies

Ray version: 2.44.1

Reproduction script

--

Issue Severity

Medium: It is a significant difficulty but I can work around it.

@hedeesaa hedeesaa added bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Apr 24, 2025
@masoudcharkhabi masoudcharkhabi added clusters dashboard Issues specific to the Ray Dashboard kuberay Issues for the Ray/Kuberay integration that are tracked on the Ray side usability labels Apr 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something that is supposed to be working; but isn't clusters dashboard Issues specific to the Ray Dashboard kuberay Issues for the Ray/Kuberay integration that are tracked on the Ray side triage Needs triage (eg: priority, bug/not-bug, and owning component) usability
Projects
None yet
Development

No branches or pull requests

2 participants