Strimzi Kafka pods startup failed

[vipul0805]

Hi,
I am just getting started with strimzi and was attempting to create a Kafka cluster using the Strimzi kafka CRD. After applying the CRD the Kafka pods are giving Exception and I am not able to figure out the exact issue.

Strimzi Operator version: 0.33.0
Kafka version: 3.3.2 and 3.3.1

The Kafka CRD is as follows:

        apiVersion: kafka.strimzi.io/v1beta2
        kind: Kafka
        metadata:
          name: strimzi-kafka
        spec:
          kafka:
            version: 3.3.1
            replicas: 3
            listeners:
              - name: plain
                port: 9092
                type: internal
                tls: false
                configuration:
                  useServiceDnsDomain: true
            config:
              offsets.topic.replication.factor: 1
              transaction.state.log.replication.factor: 1
              transaction.state.log.min.isr: 1
              default.replication.factor: 2
              min.insync.replicas: 2
              inter.broker.protocol.version: "3.3"
            storage:
              type: jbod
              volumes:
              - id: 0
                type: persistent-claim
                size: 100Gi
                class: gp2
                deleteClaim: false
            metricsConfig:
              type: jmxPrometheusExporter
              valueFrom:
                configMapKeyRef:
                  name: kafka-metrics
                  key: kafka-metrics-config.yml
            rack:
              topologyKey: topology.kubernetes.io/zone
            resources:
              requests:
                memory: 8Gi
                cpu: "2"
              limits:
                memory: 8Gi
                cpu: "3"
          zookeeper:
            replicas: 3
            storage:
              type: persistent-claim
              size: 100Gi
              class: gp2
              deleteClaim: false
            metricsConfig:
              type: jmxPrometheusExporter
              valueFrom:
                configMapKeyRef:
                  name: kafka-metrics
                  key: zookeeper-metrics-config.yml
            resources:
              requests:
                memory: 8Gi
                cpu: "2"
              limits:
                memory: 8Gi
                cpu: "3"          
          entityOperator:
            tlsSidecar:
              resources:
                requests:
                  cpu: 200m
                  memory: 64Mi
                limits:
                  cpu: 500m
                  memory: 128Mi
            template:
              pod:
                affinity:
                  nodeAffinity:
                    requiredDuringSchedulingIgnoredDuringExecution:
                      nodeSelectorTerms:
                      - matchExpressions:
                        - key: errors.ea.com/linux-distro
                          operator: In
                          values:
                          - bottlerocket
            topicOperator:
              resources:
                requests:
                  memory: 512Mi
                  cpu: "1"
                limits:
                  memory: 512Mi
                  cpu: "1"
            userOperator:
              resources:
                requests:
                  memory: 512Mi
                  cpu: "1"
                limits:
                  memory: 512Mi
                  cpu: "1"
          kafkaExporter:
            groupRegex: ".*"
            topicRegex: ".*"
            
      - apiVersion: v1
        kind: ConfigMap
        metadata:
          name: kafka-metrics
          labels:
            app: strimzi
        data:
          kafka-metrics-config.yml: |
            # See https://github.com/prometheus/jmx_exporter for more info about JMX Prometheus Exporter metrics
            lowercaseOutputName: true
            rules:
            # Special cases and very specific rules
            - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
              name: kafka_server_$1_$2
              type: GAUGE
              labels:
              clientId: "$3"
              topic: "$4"
              partition: "$5"
            - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
              name: kafka_server_$1_$2
              type: GAUGE
              labels:
              clientId: "$3"
              broker: "$4:$5"
            - pattern: kafka.server<type=(.+), cipher=(.+), protocol=(.+), listener=(.+), networkProcessor=(.+)><>connections
              name: kafka_server_$1_connections_tls_info
              type: GAUGE
              labels:
                cipher: "$2"
                protocol: "$3"
                listener: "$4"
                networkProcessor: "$5"
            - pattern: kafka.server<type=(.+), clientSoftwareName=(.+), clientSoftwareVersion=(.+), listener=(.+), networkProcessor=(.+)><>connections
              name: kafka_server_$1_connections_software
              type: GAUGE
              labels:
                clientSoftwareName: "$2"
                clientSoftwareVersion: "$3"
                listener: "$4"
                networkProcessor: "$5"
            - pattern: "kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+):"
              name: kafka_server_$1_$4
              type: GAUGE
              labels:
              listener: "$2"
              networkProcessor: "$3"
            - pattern: kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+)
              name: kafka_server_$1_$4
              type: GAUGE
              labels:
              listener: "$2"
              networkProcessor: "$3"
            # Some percent metrics use MeanRate attribute
            # Ex) kafka.server<type=(KafkaRequestHandlerPool), name=(RequestHandlerAvgIdlePercent)><>MeanRate
            - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>MeanRate
              name: kafka_$1_$2_$3_percent
              type: GAUGE
            # Generic gauges for percents
            - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>Value
              name: kafka_$1_$2_$3_percent
              type: GAUGE
            - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*, (.+)=(.+)><>Value
              name: kafka_$1_$2_$3_percent
              type: GAUGE
              labels:
                "$4": "$5"
            # Generic per-second counters with 0-2 key/value pairs
            - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
              name: kafka_$1_$2_$3_total
              type: COUNTER
              labels:
                "$4": "$5"
                "$6": "$7"
            - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
              name: kafka_$1_$2_$3_total
              type: COUNTER
              labels:
                "$4": "$5"
            - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
              name: kafka_$1_$2_$3_total
              type: COUNTER
            # Generic gauges with 0-2 key/value pairs
            - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
              name: kafka_$1_$2_$3
              type: GAUGE
              labels:
                "$4": "$5"
                "$6": "$7"
            - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
              name: kafka_$1_$2_$3
              type: GAUGE
              labels:
                "$4": "$5"
            - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
              name: kafka_$1_$2_$3
              type: GAUGE
            # Emulate Prometheus 'Summary' metrics for the exported 'Histogram's.
            # Note that these are missing the '_sum' metric!
            - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
              name: kafka_$1_$2_$3_count
              type: COUNTER
              labels:
                "$4": "$5"
                "$6": "$7"
            - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile
              name: kafka_$1_$2_$3
              type: GAUGE
              labels:
                "$4": "$5"
                "$6": "$7"
                quantile: "0.$8"
            - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
              name: kafka_$1_$2_$3_count
              type: COUNTER
              labels:
                "$4": "$5"
            - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile
              name: kafka_$1_$2_$3
              type: GAUGE
              labels:
                "$4": "$5"
                quantile: "0.$6"
            - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
              name: kafka_$1_$2_$3_count
              type: COUNTER
            - pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile
              name: kafka_$1_$2_$3
              type: GAUGE
              labels:
                quantile: "0.$4"
          zookeeper-metrics-config.yml: |
            # See https://github.com/prometheus/jmx_exporter for more info about JMX Prometheus Exporter metrics
            lowercaseOutputName: true
            rules:
            # replicated Zookeeper
            - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+)><>(\\w+)"
              name: "zookeeper_$2"
              type: GAUGE
            - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+)><>(\\w+)"
              name: "zookeeper_$3"
              type: GAUGE
              labels:
                replicaId: "$2"
            - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(Packets\\w+)"
              name: "zookeeper_$4"
              type: COUNTER
              labels:
                replicaId: "$2"
                memberType: "$3"
            - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(\\w+)"
              name: "zookeeper_$4"
              type: GAUGE
              labels:
                replicaId: "$2"
                memberType: "$3"
            - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+), name3=(\\w+)><>(\\w+)"
              name: "zookeeper_$4_$5"
              type: GAUGE
              labels:
                replicaId: "$2"
                memberType: "$3"

The Exception is:

STRIMZI_BROKER_ID=0
Preparing truststore for replication listener
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/kafka/cluster.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore for replication listener is complete
Looking for the right CA
Found the right CA: /opt/kafka/cluster-ca-certs/ca.crt
Preparing keystore for replication and clienttls listener
Preparing keystore for replication and clienttls listener is complete
Preparing truststore for client authentication
Adding /opt/kafka/client-ca-certs/ca.crt to truststore /tmp/kafka/clients.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore for client authentication is complete
Starting Kafka with configuration:
##############################
##############################
# This file is automatically generated by the Strimzi Cluster Operator
# Any changes to this file will be ignored and overwritten!
##############################
##############################

##########
# Broker ID
##########
broker.id=0
node.id=0

##########
# Rack ID
##########
broker.rack=us-east-1a

##########
# Zookeeper
##########
zookeeper.connect=strimzi-kafka-zookeeper-client:2181
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
zookeeper.ssl.client.enable=true
zookeeper.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
zookeeper.ssl.keystore.password=[hidden]
zookeeper.ssl.keystore.type=PKCS12
zookeeper.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
zookeeper.ssl.truststore.password=[hidden]
zookeeper.ssl.truststore.type=PKCS12

##########
# Kafka message logs configuration
##########
log.dirs=/var/lib/kafka/data-0/kafka-log0

##########
# Control Plane listener
##########
listener.name.controlplane-9090.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
listener.name.controlplane-9090.ssl.keystore.password=[hidden]
listener.name.controlplane-9090.ssl.keystore.type=PKCS12
listener.name.controlplane-9090.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
listener.name.controlplane-9090.ssl.truststore.password=[hidden]
listener.name.controlplane-9090.ssl.truststore.type=PKCS12
listener.name.controlplane-9090.ssl.client.auth=required

##########
# Replication listener
##########
listener.name.replication-9091.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
listener.name.replication-9091.ssl.keystore.password=[hidden]
listener.name.replication-9091.ssl.keystore.type=PKCS12
listener.name.replication-9091.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
listener.name.replication-9091.ssl.truststore.password=[hidden]
listener.name.replication-9091.ssl.truststore.type=PKCS12
listener.name.replication-9091.ssl.client.auth=required

##########
# Listener configuration: PLAIN-9092
##########

##########
# Common listener configuration
##########
listeners=CONTROLPLANE-9090://0.0.0.0:9090,REPLICATION-9091://0.0.0.0:9091,PLAIN-9092://0.0.0.0:9092
advertised.listeners=CONTROLPLANE-9090://strimzi-kafka-kafka-0.strimzi-kafka-kafka-brokers.strimzi-kafka.svc:9090,REPLICATION-9091://strimzi-kafka-kafka-0.strimzi-kafka-kafka-brokers.strimzi-kafka.svc:9091,PLAIN-9092://strimzi-kafka-kafka-0.strimzi-kafka-kafka-brokers.strimzi-kafka.svc.cluster.local:9092
listener.security.protocol.map=CONTROLPLANE-9090:SSL,REPLICATION-9091:SSL,PLAIN-9092:PLAINTEXT
control.plane.listener.name=CONTROLPLANE-9090
inter.broker.listener.name=REPLICATION-9091
sasl.enabled.mechanisms=
ssl.endpoint.identification.algorithm=HTTPS

##########
# User provided configuration
##########
default.replication.factor=1
inter.broker.protocol.version=3.3
min.insync.replicas=1
offsets.topic.replication.factor=1
transaction.state.log.min.isr=1
transaction.state.log.replication.factor=1
log.message.format.version=3.3
Configuring Java heap: -Xms4294967296 -Xmx4294967296
+ exec /usr/bin/tini -w -e 143 -- /opt/kafka/bin/kafka-server-start.sh /tmp/strimzi.properties
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:491)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:503)
Caused by: java.lang.NullPointerException: Cannot invoke "java.util.Map.size()" because "map" is null
	at java.base/java.util.TreeMap.putAll(TreeMap.java:314)
	at java.base/java.util.TreeMap.<init>(TreeMap.java:187)
	at io.prometheus.jmx.JmxCollector.loadConfig(JmxCollector.java:251)
	at io.prometheus.jmx.JmxCollector.<init>(JmxCollector.java:91)
	at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:29)
	... 6 more
*** java.lang.instrument ASSERTION FAILED ***: "result" with message agent load/premain call failed at src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422
FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed

I am not able to figure out that which map is it referring to Cannot invoke "java.util.Map.size()" because "map" is null.

Any help will be appreciated. Thanks

[scholzj]

I think this is because your metrics YAML file is invalid (at least for Kafka). Check this diff from the Strimzi example:

@@ -82,15 +82,15 @@ data:
       name: kafka_server_$1_$2
       type: GAUGE
       labels:
-       clientId: "$3"
-       topic: "$4"
-       partition: "$5"
+      clientId: "$3"
+      topic: "$4"
+      partition: "$5"
     - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
       name: kafka_server_$1_$2
       type: GAUGE
       labels:
-       clientId: "$3"
-       broker: "$4:$5"
+      clientId: "$3"
+      broker: "$4:$5"
     - pattern: kafka.server<type=(.+), cipher=(.+), protocol=(.+), listener=(.+), networkProcessor=(.+)><>connections
       name: kafka_server_$1_connections_tls_info
       type: GAUGE
@@ -111,14 +111,14 @@ data:
       name: kafka_server_$1_$4
       type: GAUGE
       labels:
-       listener: "$2"
-       networkProcessor: "$3"
+      listener: "$2"
+      networkProcessor: "$3"
     - pattern: kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+)
       name: kafka_server_$1_$4
       type: GAUGE
       labels:
-       listener: "$2"
-       networkProcessor: "$3"
+      listener: "$2"
+      networkProcessor: "$3"
     # Some percent metrics use MeanRate attribute
     # Ex) kafka.server<type=(KafkaRequestHandlerPool), name=(RequestHandlerAvgIdlePercent)><>MeanRate
     - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>MeanRate

Some of the labels fields are missaligned and that is causing the error.

[scholzj]

Where exactly is the indentation wrong? Can you point to the exact lines? I compared it with your file to find your indentation issues as you see in the diff above.

[vipul0805]

Yes, The configmap which I was using had 0 indentations at the labels level as can be seen in the diff. The configmap in the repo has only 1 indent at the labels level at the same lines as the diff which you have shared. Although 1 indent will work too but it should be 2 indents.

[scholzj]

Ah, ok. It does not really matter how much it is indented for the validity, but yes, it is weird with the single-space indentation. If you want, feel free to open a PR to fix this -> but this needs to be fixed in packaging/examples/metrics/... and not in examples/metrics/....

[vipul0805]

Sure. I will raise a PR.

[scholzj]

Thanks.