[Bug]: Support for Kafka Consumer Group Lag Metrics in KRaft Mode (Strimzi)

[inderanz]

Bug Description

Seems, there is no native support in Strimzi for exposing Kafka consumer group lag metrics (kafka_consumergroup_lag, kafka_consumergroup_current_offset, kafka_consumergroup_lag_seconds) when Kafka is running in KRaft mode (i.e., without ZooKeeper).

These metrics are essential for production monitoring and alerting on streaming pipeline health and consumer behavior. However, they are not exposed via the standard JMX exporter or Strimzi’s metricsConfig.

Kafka Exporter is not supported in Strimzi for KRaft mode clusters.

The JMX exporter does not expose consumer group lag metrics, as they depend on reading committed offsets and comparing them with the latest offsets per partition.

The only method currently available is to manually run kafka-consumer-groups.sh and parse the output, which is not ideal for time-series metrics or Prometheus scraping.

My current config, which is giving all other metrics expect these

...
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
default.replication.factor: 3
min.insync.replicas: 2
socket.request.max.bytes: 400000000 # 400 MB
auto.create.topics.enable: false
metricsConfig:
type: jmxPrometheusExporter
valueFrom:
configMapKeyRef:
name: kafka-metrics
key: kafka-metrics-config.yml

ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
name: kafka-metrics-sit
namespace: ns-pspnpp-integration-sit
data:
kafka-metrics-config.yml: |
lowercaseOutputName: true
rules:
- pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.)><>Value
name: kafka_server_$1_$2
type: GAUGE
labels:
clientId: "$3"
topic: "$4"
partition: "$5"
- pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
name: kafka_server_$1_$2
type: GAUGE
labels:
clientId: "$3"
broker: "$4:$5"
- pattern: kafka.server<type=(.+), cipher=(.+), protocol=(.+), listener=(.+), networkProcessor=(.+)><>connections
name: kafka_server_$1_connections_tls_info
type: GAUGE
labels:
cipher: "$2"
protocol: "$3"
listener: "$4"
networkProcessor: "$5"
- pattern: kafka.server<type=(.+), clientSoftwareName=(.+), clientSoftwareVersion=(.+), listener=(.+), networkProcessor=(.+)><>connections
name: kafka_server_$1_connections_software
type: GAUGE
labels:
clientSoftwareName: "$2"
clientSoftwareVersion: "$3"
listener: "$4"
networkProcessor: "$5"
- pattern: "kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+-total):"
name: kafka_server_$1_$4
type: COUNTER
labels:
listener: "$2"
networkProcessor: "$3"
- pattern: "kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+):"
name: kafka_server_$1_$4
type: GAUGE
labels:
listener: "$2"
networkProcessor: "$3"
- pattern: kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+-total)
name: kafka_server_$1_$4
type: COUNTER
labels:
listener: "$2"
networkProcessor: "$3"
- pattern: kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+)
name: kafka_server_$1_$4
type: GAUGE
labels:
listener: "$2"
networkProcessor: "$3"
- pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w><>MeanRate
name: kafka_$1_$2_$3_percent
type: GAUGE
- pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>Value
name: kafka_$1_$2_$3_percent
type: GAUGE
- pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*, (.+)=(.+)><>Value
name: kafka_$1_$2_$3_percent
type: GAUGE
labels:
"$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
name: kafka_$1_$2_$3_total
type: COUNTER
labels:
"$4": "$5"
"$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
name: kafka_$1_$2_$3_total
type: COUNTER
labels:
"$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
name: kafka_$1_$2_$3_total
type: COUNTER
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
"$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
name: kafka_$1_$2_$3
type: GAUGE
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
name: kafka_$1_$2_$3_count
type: COUNTER
labels:
"$4": "$5"
"$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.), (.+)=(.+)><>(\d+)thPercentile
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
"$6": "$7"
quantile: "0.$8"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
name: kafka_$1_$2_$3_count
type: COUNTER
labels:
"$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.)><>(\d+)thPercentile
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
quantile: "0.$6"
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
name: kafka_$1_$2_$3_count
type: COUNTER
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile
name: kafka_$1_$2_$3
type: GAUGE
labels:
quantile: "0.$4"
- pattern: "kafka.server<type=raft-metrics><>(.+-total|.+-max):"
name: kafka_server_raftmetrics_$1
type: COUNTER
- pattern: "kafka.server<type=raft-metrics><>(current-state): (.+)"
name: kafka_server_raftmetrics_$1
value: 1
type: UNTYPED
labels:
$1: "$2"
- pattern: "kafka.server<type=raft-metrics><>(.+):"
name: kafka_server_raftmetrics_$1
type: GAUGE
- pattern: "kafka.server<type=raft-channel-metrics><>(.+-total|.+-max):"
name: kafka_server_raftchannelmetrics_$1
type: COUNTER
- pattern: "kafka.server<type=raft-channel-metrics><>(.+):"
name: kafka_server_raftchannelmetrics_$1
type: GAUGE
- pattern: "kafka.server<type=broker-metadata-metrics><>(.+):"
name: kafka_server_brokermetadatametrics_$1
type: GAUGE

When am trying to add:

kafkaExporter:
topicRegex: "."
groupRegex: "."
logging: info

its giving error & seems because am using raft mode.

Steps to reproduce

No response

Expected behavior

No response

Strimzi version

Strimzi: v0.42.0 , Kafka: v3.6.2 (KRaft mode)

Kubernetes version

1.31.6-gke.1064001

Installation method

Operator

Infrastructure

GKE

Configuration files and logs

No response

Additional context

No response

[scholzj]

The JMX Exporter was never exposing any lag metrics. That is done by the Kafka Exporter.

[inderanz]

Thanks @scholzj , can you share link for Kafka exporter which I can deploy to capture these metrics and ensure it supports with kraft mode .

[scholzj]

It is part of the Kafka CR (.spec.kafkaExporter) and is covered by the docs. (BTW: This is not a new thing, this was always the way how to get the consumer lag for many years)

[inderanz]

Hi @scholzj

Thanks for the response. will this work with KRaft mode installation ?

Hope you are referring to below config: ?

strimzi-kafka-operator/examples/metrics/kafka-metrics.yaml

Line 76 in 84af936

kafkaExporter:

• If yes, will this work with KRaft mode installation because am getting error... if you confirm I will workout to fix it & also confirm - if there is any dependancy of a specific KAKFA CRD version to be installed?

Some output for you to review:

] $
] $ kubectl get crd kafkas.kafka.strimzi.io -o json | jq '.spec.versions[] | select(.name=="v1beta2") | .schema.openAPIV3Schema.properties.spec.properties.kafka.properties' |grep -i kafkaExporter

~] $ kubectl get crd kafkas.kafka.strimzi.io -o json | jq '.spec.versions[] | select(.name=="v1beta2") | .schema.openAPIV3Schema.properties.spec.properties.kafka.properties' |grep -i exporter
"description": "The container image used for Kafka pods. If the property is not set, the default Kafka image version is determined based on the version configuration. The image names are specifically mapped to corresponding versions in the Cluster Operator configuration. Changing the Kafka image version does not automatically update the image versions for other components, such as Kafka Exporter. ",
"description": "Metrics type. Only 'jmxPrometheusExporter' supported currently.",
"jmxPrometheusExporter"
"description": "ConfigMap entry where the Prometheus JMX Exporter configuration is stored. ",

[inderanz]

kafka: authorization: type: simple config: default.replication.factor: 3 min.insync.replicas: 2 offsets.topic.replication.factor: 3 socket.request.max.bytes: 400000000 ssl.cipher.suites: TLS_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ssl.enabled.protocols: TLSv1.3, TLSv1.2 ssl.protocol: TLSv1.3 transaction.state.log.min.isr: 2 transaction.state.log.replication.factor: 3 image: strimzi/kafka:0.42.0-kafka-3.6.2-amd64 jvmOptions: -Xms: 8G -Xmx: 12G gcLoggingEnabled: true kafkaExporter: topicRegex: ".*" groupRegex: ".*" logging: debug resources: requests: cpu: 200m memory: 256Mi limits: cpu: 500m memory: 512Mi

With above (not complete) , am getting error:

,"operatorLastSuccessfulVersion":"0.42.0"}}": strict decoding error: unknown field "spec.kafka.kafkaExporter"

in my deployed CRD:

$ kubectl get crd kafkas.kafka.strimzi.io -o yaml > kafka-crd.yaml

] $ grep -A 10 'kafkaExporter' kafka-crd.yaml
kafkaExporter:
description: Configuration of the Kafka Exporter. Kafka Exporter can
provide additional metrics, for example lag of consumer group at
topic/partition.
properties:
enableSaramaLogging:
description: Enable Sarama logging, a Go client library used by
the Kafka Exporter.
type: boolean
groupExcludeRegex:
description: Regular expression to specify which consumer groups
[inder_chauhan_anz_com@psre-linux-util-vm-instance-pk1v ~] $

[scholzj]

Yes, the example here will work in KRaft.

Your code is not formatted properly, so I cannto read it. But from the error message:

unknown field "spec.kafka.kafkaExporter"

I'm guessing your YAML is incorrect. The Kafka Exporter section is directly in .spec and not in .spec.kafka.