Grafana Alerting plugin with multi-tenant Mimir ruler

[arnitolog]

Hello,
I’m trying to integrate the Grafana Alerting plugin with the multi-tenant Mimir ruler.
I’m able to get metrics for any tenant, so looks like datasource is working as expected. But when I tried to get alerts in the Alerting plugin I got an error:

Failed to load rules state from [mimir-delta](https://metrics.test.com/datasources/edit/eb8de08c-96ac-4f0e-8cad-bd87c87c90b1): {"status":"error","data":null,"errorType":"server_error","error":"no valid org id found"}
Failed to load rules config from [mimir-delta](https://metrics.test.com/datasources/edit/eb8de08c-96ac-4f0e-8cad-bd87c87c90b1): {"status":"error","data":null,"errorType":"server_error","error":"no org id"}

from the ruler side, I see the following errors:

ts=2024-01-15T22:44:13.416053602Z caller=api.go:160 level=error user=default|k8s|nginx|rds|sharedservicess msg="error extracting org id from context" err="multiple org IDs present"
ts=2024-01-15T22:44:13.41609064Z caller=logging.go:126 level=warn traceID=1d1fd46ecc4160f8 msg="GET /prometheus/api/v1/rules (500) 90.187µs Response: \"{\\\"status\\\":\\\"error\\\",\\\"data\\\":null,\\\"errorType\\\":\\\"server_error\\\",\\\"error\\\":\\\"no valid org id found\\\"}\" ws: false; Accept-Encoding: gzip; Connection: close; Traceparent: 00-d59bf3e0dd6a7d8f7b86848aab2980c5-e3d4a79ad5b32b13-01; User-Agent: Grafana/10.2.3; X-Forwarded-For: 10.65.65.242; X-Forwarded-Host: mimir-delta-eastus.aks.dev1.test.com; X-Forwarded-Port: 443; X-Forwarded-Proto: https; X-Forwarded-Scheme: https; X-Grafana-Referer: ; X-Real-Ip: 10.65.65.242; X-Request-Id: eb0a84b31ee1e0c7172b254f7667b54c; X-Scheme: https; X-Scope-Orgid: default|k8s|nginx|rds|sharedservicess; "
ts=2024-01-15T22:44:13.459691567Z caller=logging.go:126 level=warn traceID=33ac360a0b9e99a2 msg="GET /prometheus/config/v1/rules?subtype=mimir (500) 74.038µs Response: \"{\\\"status\\\":\\\"error\\\",\\\"data\\\":null,\\\"errorType\\\":\\\"server_error\\\",\\\"error\\\":\\\"no org id\\\"}\" ws: false; Accept-Encoding: gzip; Connection: close; Traceparent: 00-0c48f391c9dae46f5e28dea105c7afa9-57da77f3fc98e333-01; User-Agent: Grafana/10.2.3; X-Forwarded-For: 10.65.65.102; X-Forwarded-Host: mimir-delta-eastus.aks.dev1.test.com; X-Forwarded-Port: 443; X-Forwarded-Proto: https; X-Forwarded-Scheme: https; X-Grafana-Referer: ; X-Real-Ip: 10.65.65.102; X-Request-Id: 148cb729c14d092cd97093c8011626b6; X-Scheme: https; X-Scope-Orgid: default|k8s|nginx|rds|sharedservicess; "

mimir is configured with helm chard and with the federation options in place:

  structuredConfig:
    tenant_federation:
      enabled: true
    ruler:
      tenant_federation:
        enabled: true

and I'm able to see metrics from the recording rules
So the question is what I did wrong or how to properly integrate the Grafana Alerting plugin with multi-tenant Mimir ruler

[rayjasson98]

Just for reference, this is mentioned in this GitHub issue (#8165), and it seems there is no workaround for creating an alert rule that includes multiple tenants.