Missing label on consumer lag metrics

[marceloavan]

Hey,

I am setting up the o11y for an environment following the example using the kafka exporter.

I can see the metrics and most of the panels on the dashboards are working, except some related to consumer groups.

Exploring the metrics on Grafana I have the following labels:

kafka_topic_partition_current_offset{
  cluster
  container
  endpoint
  instance
  job
  kubernetes_pod_name
  namespace
  node_ip
  node_name
  partition
  pod
  prometheus
  strimzi_io_cluster
  strimzi_io_component_type
  strimzi_io_kind
  strimzi_io_name
  topic
}

Looking directly on the kafka exporter's /metrics, I can see just topic and partition as labels.

So the label consumergroup (required by the dashboards) is not present.

Important to say:

• I am running a producer and a consumer to be sure the metrics are being provoked
• My metrics config map is exactly this
• And my exporter configuration:

  kafkaExporter:
    groupRegex: .*
    topicRegex: .*

Any thought on what I am missing? Is this behavior known?

[scholzj]

The other labels are probably added by Prometheus when scraping the metrics endpoint. They basically describe where the metrics were scraped from.

[scholzj]

Yeah, I am considering both metrics, one of each panel on my initial screenshot:

sum(kafka_consumergroup_lag{consumergroup=~"$consumergroup",topic=~"$topic", namespace=~"$kubernetes_namespace"}) by (consumergroup, topic)

and

sum(delta(kafka_consumergroup_current_offset{consumergroup=~"$consumergroup",topic=~"$topic", namespace=~"$kubernetes_namespace"}[5m])/300) by (consumergroup, topic)

The variable present on the dashboard is based on kafka_consumergroup_current_offset:

label_values(kafka_consumergroup_current_offset{namespace="$kubernetes_namespace",strimzi_io_cluster="$strimzi_cluster_name"}, consumergroup).

The point is that consumergroup label is not present in any of these metrics.

So, these are metrics related to the consumer groups -> they should have the consumer group label. The question is whether they are missing in the Kafka Exporter or whether they are removed in Prometheus after/while being scraped.

[marceloavan]

Yeah, the issue is not related with the labels, I get confused by the metrics names, you got it right!

The issue is that the exporter is not generating any metric prefixed with kafka_consumergroup*, I am trying to figure it out now.

[scholzj]

As in that you do not see those metrics in your Prometheus? Or with curl in the Kafka Exporter Pod?

[marceloavan]

Tried both... I first tried directly on prometheus and then with curl on kafka exporter pod.

But I figured it out, it is always the client, right? hehe... basically kcat trolled me ignoring the -G parameter and didn't create the consumer group, so the metrics weren't created :( I relied on it too much I didn't check the consumer group per se. After ensuring the consumer group, the metrics now are there.

Thanks for the support again @scholzj, and I am sorry for the confusion (also again).

[scholzj]

Great, glad you figured it out.