Prometheus: a way to include custom labels into per-object metrics

[blowfishpro]

Is your feature request related to a problem? Please describe.

We put a lot of queues into each RabbitMQ cluster, many of which are related in various ways. Some of this information can be parsed out of the queue names we generate but this is rather limited. It would be great if we could add per-object custom labels to queue metrics (and e.g. exchange metrics) that we could then use in our observability and monitoring.

Describe the solution you'd like

Custom labels could be set via arguments when a queue/exchange is declared, e.g.

{
  "parameters": {
    "x-prometheus-labels": {
      "custom-label-1": "custom-value-1",
      "custom-label-2": "custom-value-2"
    }
  }
}

Those labels would then be added to the per-queue/exchange metrics generated by the prometheus plugin.

Describe alternatives you've considered

There's some information we can include in e.g. queue names and then parse out later. That's fairly limited though.

Additional context

Could it also be useful to add per-vhost labels that are added to the metrics for all queues and exchanges within that vhost?

[michaelklishin]

@blowfishpro are you aware of nodes and cluster tags? We can probably reuse them and add them as labels instead of introducing a yet another set of tags.

[blowfishpro]

I was not though we already have a way to (externally) add tags at the RabbitMQ cluster level, I was specifically requesting something more granular than that.

[michaelklishin]

@blowfishpro sorry, I'm not sure if I understand.

Are you aware of these two existing tagging mechanisms? Are you suggesting that we use them in the Prometheus plugin?

I am not eager to introduce a yet another tagging mechanism.

We already have cluster tags, node tags, and have had virtual host tags for a decade.

[mkuratczyk]

Regarding vhosts, I think we can indeed use existing vhost tags and emit a rabbitmq_vhost_info metric with these tags as labels.

For queues, in RabbitMQ 4.1, rabbitmq_queue_info metric gets introduced and would be the obvious place to expose queue tags if we decided to introduce them. However, is there a particular reason you need RabbitMQ to expose these labels/tags, instead of your applications doing so (which declare these queues I guess)? We'd need to keep this info in memory and we need to be able to handle lots of queues, so we'd need strict limits (no more than a handful of tags per queue + short keys/values).

We'd also need some conflict resolution mechanism - in 4.1 the metric looks like this:

rabbitmq_queue_info{vhost="/",queue="qq",queue_type="rabbit_quorum_queue",membership="follower"} 1         

In the future we may want to add more information to that metric, which could conflict with user-provided tags. I guess the resolution would need to be "built-in labels win" (a tag with a conflicting name is not exposed as a Prometheus label).