Migration to KRaft mode stuck

[Davee02]

Hi Community!

I would appreciate your help. I'm currently in the process of migrating from Zookeeper to KRaft mode using the upgrade guide in the docs, but the process seems to be stuck at step 5 where the metadata state should be updated.

Current Setup

• Kubernetes 1.33.0
• Strimzi version: 0.45.0
• Kafka version: 3.9.0
• Kafka Nodepools:
  • broker role with 3 replicas (newly created for the migration, as mentioned in the docs`
  • controller role with 3 replicas

Steps I've performed

1. Downgrade running Strimzi operator from 0.46.0 to 0.45.0 because I missed the deprecation notice
2. Create new Kafka broker Nodepool
3. Add annotation strimzi.io/node-pools: enabled to the existing Kafka (cluster) resource
4. Update Kafka version from 3.8.0 to 3.9.0 because I wanted to use the latest Zookeeper supporting version to ease the migration
5. Create controller Nodepool
6. Enable KRaft migration in the Kafka resource by setting the strimzi.io/kraft annotation to migration

Current State
Kafka cluster with 3 replicas:

NAME    DESIRED KAFKA REPLICAS   DESIRED ZK REPLICAS   READY   WARNINGS
kafka   3                        3                     True

Two Kafka Nodepools:

NAME         DESIRED REPLICAS
controller   3
kafka        3

3 running Zookeeper, 3 running broker, 3 running `controller:

NAME                                     READY   STATUS    RESTARTS   AGE
kafka-controller-3                       1/1     Running   0          3m45s
kafka-controller-4                       1/1     Running   0          3m44s
kafka-controller-5                       1/1     Running   0          12h
kafka-kafka-0                            1/1     Running   0          3m38s
kafka-kafka-1                            1/1     Running   0          2m12s
kafka-kafka-2                            1/1     Running   0          91s
kafka-zookeeper-0                        1/1     Running   0          63m
kafka-zookeeper-1                        1/1     Running   0          63m
kafka-zookeeper-2                        1/1     Running   0          63m

Problem
According to step 5 in the migration guide, Strimzi should now begin setting the metadata state on the Kafka resource. This however, doesn't happen.
In the cluster-operator, I can see that it repeatedly wants to transition the state but then fails to connect to node -1:

2025-06-12 05:50:03 INFO  KafkaMetadataStateManager:96 - Reconciliation #931(watch) Kafka(name-of-application/kafka): Transitioning metadata state from [ZooKeeper] to [KRaftMigration] with strimzi.io/kraft annotation [migration]
2025-06-12 05:50:04 INFO  CrdOperator:123 - Reconciliation #931(watch) Kafka(name-of-application/kafka): Status of Kafka kafka in namespace name-of-application has been updated
2025-06-12 05:50:04 INFO  AbstractOperator:520 - Reconciliation #934(watch) Kafka(name-of-application/kafka): Kafka kafka in namespace name-of-application was MODIFIED
2025-06-12 05:50:04 INFO  AbstractOperator:546 - Reconciliation #931(watch) Kafka(name-of-application/kafka): reconciled
2025-06-12 05:50:04 INFO  AbstractOperator:266 - Reconciliation #932(watch) Kafka(name-of-application/kafka): Kafka kafka will be checked for creation or modification
2025-06-12 05:50:07 WARN  NetworkClient:849 - [AdminClient clientId=adminclient-3044] Connection to node -1 (kafka-kafka-bootstrap.name-of-application.svc.cluster.local/10.0.125.157:9091) could not be established. Node may not be available.
2025-06-12 05:50:07 WARN  NetworkClient:849 - [AdminClient clientId=adminclient-3044] Connection to node -1 (kafka-kafka-bootstrap.name-of-application.svc.cluster.local/10.0.125.157:9091) could not be established. Node may not be available.
2025-06-12 05:50:08 INFO  KafkaMetadataStateManager:96 - Reconciliation #932(watch) Kafka(name-of-application/kafka): Transitioning metadata state from [ZooKeeper] to [KRaftMigration] with strimzi.io/kraft annotation [migration]
2025-06-12 05:50:08 INFO  CrdOperator:123 - Reconciliation #932(watch) Kafka(name-of-application/kafka): Status of Kafka kafka in namespace name-of-application has been updated
2025-06-12 05:50:08 INFO  AbstractOperator:520 - Reconciliation #935(watch) Kafka(name-of-application/kafka): Kafka kafka in namespace name-of-application was MODIFIED
2025-06-12 05:50:08 INFO  AbstractOperator:546 - Reconciliation #932(watch) Kafka(name-of-application/kafka): reconciled
2025-06-12 05:50:08 INFO  AbstractOperator:266 - Reconciliation #933(watch) Kafka(name-of-application/kafka): Kafka kafka will be checked for creation or modification
2025-06-12 05:50:10 WARN  NetworkClient:849 - [AdminClient clientId=adminclient-3048] Connection to node -1 (kafka-kafka-bootstrap.name-of-application.svc.cluster.local/10.0.125.157:9091) could not be established. Node may not be available.
2025-06-12 05:50:11 WARN  NetworkClient:849 - [AdminClient clientId=adminclient-3050] Connection to node -1 (kafka-kafka-bootstrap.name-of-application.svc/10.0.125.157:9091) could not be established. Node may not be available.
2025-06-12 05:50:11 WARN  NetworkClient:849 - [AdminClient clientId=adminclient-3050] Connection to node -1 (kafka-kafka-bootstrap.name-of-application.svc/10.0.125.157:9091) could not be established. Node may not be available.
2025-06-12 05:50:11 WARN  NetworkClient:849 - [AdminClient clientId=adminclient-3051] Connection to node -1 (kafka-kafka-bootstrap.name-of-application.svc/10.0.125.157:9091) could not be established. Node may not be available.
2025-06-12 05:50:12 WARN  NetworkClient:849 - [AdminClient clientId=adminclient-3051] Connection to node -1 (kafka-kafka-bootstrap.name-of-application.svc/10.0.125.157:9091) could not be established. Node may not be available.
2025-06-12 05:50:12 INFO  KafkaMetadataStateManager:96 - Reconciliation #933(watch) Kafka(name-of-application/kafka): Transitioning metadata state from [ZooKeeper] to [KRaftMigration] with strimzi.io/kraft annotation [migration]
2025-06-12 05:50:12 INFO  AbstractOperator:520 - Reconciliation #936(watch) Kafka(name-of-application/kafka): Kafka kafka in namespace name-of-application was MODIFIED
2025-06-12 05:50:12 INFO  CrdOperator:123 - Reconciliation #933(watch) Kafka(name-of-application/kafka): Status of Kafka kafka in namespace name-of-application has been updated
2025-06-12 05:50:12 INFO  AbstractOperator:546 - Reconciliation #933(watch) Kafka(name-of-application/kafka): reconciled
2025-06-12 05:50:12 INFO  AbstractOperator:266 - Reconciliation #934(watch) Kafka(name-of-application/kafka): Kafka kafka will be checked for creation or modification

This happens over and over.
The service kafka-kafka-bootstrap exists:

NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE   SELECTOR
kafka-kafka-bootstrap   ClusterIP   10.0.125.157   <none>        9091/TCP,9092/TCP   23h   strimzi.io/cluster=kafka,strimzi.io/kind=Kafka,strimzi.io/name=kafka-kafka

Pods matching this selector:

NAME                 READY   STATUS    RESTARTS   AGE
kafka-controller-3   1/1     Running   0          33m
kafka-controller-4   1/1     Running   0          33m
kafka-controller-5   1/1     Running   0          13h
kafka-kafka-0        1/1     Running   0          33m
kafka-kafka-1        1/1     Running   0          31m
kafka-kafka-2        1/1     Running   0          30m

Question
Has anybody a suggestion on what the problem could be or how I could continue to troubleshoot? Any help is appreciated.

CRDs
Kafka:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  annotations:
    strimzi.io/kraft: migration
    strimzi.io/node-pools: enabled
  labels:
    kustomize.toolkit.fluxcd.io/name: infrastructure-configs
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: kafka
  namespace: name-of-application
spec:
  entityOperator:
    topicOperator:
      watchedNamespace: name-of-application
  kafka:
    config:
      auto.create.topics.enable: "false"
      default.replication.factor: 3
      group.min.session.timeout.ms: 1000
      inter.broker.protocol.version: "3.9"
      min.insync.replicas: 2
      offsets.topic.replication.factor: 3
      transaction.state.log.min.isr: 2
      transaction.state.log.replication.factor: 3
    listeners:
    - configuration:
        useServiceDnsDomain: true
      name: plain
      port: 9092
      tls: false
      type: internal
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: kafka-metrics-config.yaml
          name: kafka-metrics
    replicas: 3
    resources:
      limits:
        memory: 3Gi
      requests:
        cpu: "1"
        memory: 2Gi
    storage:
      size: 16Gi
      type: persistent-claim
    template:
      pod:
        priorityClassName: mission-critical-components
    version: 3.9.0
  kafkaExporter:
    groupRegex: .*
    topicRegex: .*
  zookeeper:
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: zookeeper-metrics.yaml
          name: zookeeper-metrics
    replicas: 3
    resources:
      limits:
        memory: 512Mi
      requests:
        cpu: "0.1"
        memory: 256Mi
    storage:
      size: 1Gi
      type: persistent-claim
    template:
      pod:
        priorityClassName: mission-critical-components
status:
  clusterId: 4YQBUHtHQuuvflrbPqwijg
  conditions:
  - lastTransitionTime: "2025-06-12T06:16:47.576183787Z"
    status: "True"
    type: Ready
  listeners:
  - addresses:
    - host: kafka-kafka-bootstrap.name-of-application.svc.cluster.local
      port: 9092
    bootstrapServers: kafka-kafka-bootstrap.name-of-application.svc.cluster.local:9092
    name: plain
  observedGeneration: 8

broker Nodepool:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  creationTimestamp: "2025-06-11T11:15:19Z"
  generation: 2
  labels:
    kustomize.toolkit.fluxcd.io/name: infrastructure-configs
    kustomize.toolkit.fluxcd.io/namespace: flux-system
    strimzi.io/cluster: kafka
  name: kafka
  namespace: name-of-application
  resourceVersion: "2039562188"
  uid: 7d51a02b-ec3d-4710-9d4a-8b86789c25f7
spec:
  replicas: 3
  roles:
  - broker
  storage:
    size: 16Gi
    type: persistent-claim
status:
  clusterId: 4YQBUHtHQuuvflrbPqwijg
  conditions: []
  labelSelector: strimzi.io/cluster=kafka,strimzi.io/name=kafka-kafka,strimzi.io/kind=Kafka,strimzi.io/pool-name=kafka
  nodeIds:
  - 0
  - 1
  - 2
  observedGeneration: 2
  replicas: 3
  roles:
  - broker

controller Nodepool:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  creationTimestamp: "2025-06-11T12:17:08Z"
  generation: 1
  labels:
    kustomize.toolkit.fluxcd.io/name: infrastructure-configs
    kustomize.toolkit.fluxcd.io/namespace: flux-system
    strimzi.io/cluster: kafka
  name: controller
  namespace: name-of-application
  resourceVersion: "2039653478"
  uid: 7ee04965-5c25-4e35-9c68-bf02742d2b83
spec:
  replicas: 3
  resources:
    limits:
      memory: 512Mi
    requests:
      cpu: "0.1"
      memory: 256Mi
  roles:
  - controller
  storage:
    size: 1Gi
    type: persistent-claim
  template:
    pod:
      priorityClassName: mission-critical-components
status:
  clusterId: 4YQBUHtHQuuvflrbPqwijg
  conditions: []
  labelSelector: strimzi.io/cluster=kafka,strimzi.io/name=kafka-kafka,strimzi.io/kind=Kafka,strimzi.io/pool-name=controller
  nodeIds:
  - 3
  - 4
  - 5
  observedGeneration: 1
  replicas: 3
  roles:
  - controller

[ppatierno]

Some thoughts ...

• Strimzi version: 0.45.5 .... there is no such version :-)
• when you migrated your ZooKeeper based cluster to use Kafka node pools you should remove the replicas and storage fields from your Kafka CR (anyway it isn't the cause of your issue).
• Is this conditions: ] correct in the KafkaNodePool resources? It's really weird.
• can you check logs from brokers and controllers to see if there are errors (i.e. on the 10.0.125.157 where the operator is trying to connect).

[ppatierno]

No, the 9091 is the internal listener used by Kafka for replication and by the operator to connect internally to the brokers and controllers.

[Davee02]

I noticed that after some time, one of the controllers will start to repeatedly log the following message every minute:

2025-06-12 10:01:26,922 INFO [KRaftMigrationDriver id=3] No brokers are known to KRaft, waiting for brokers to register. (org.apache.kafka.metadata.migration.KRaftMigrationDriver) [controller-3-migration-driver-event-handler]
2025-06-12 10:02:26,922 INFO [KRaftMigrationDriver id=3] No brokers are known to KRaft, waiting for brokers to register. (org.apache.kafka.metadata.migration.KRaftMigrationDriver) [controller-3-migration-driver-event-handler]
2025-06-12 10:03:26,078 INFO [KRaftMigrationDriver id=3] No brokers are known to KRaft, waiting for brokers to register. (org.apache.kafka.metadata.migration.KRaftMigrationDriver) [controller-3-migration-driver-event-handler]

The other two controllers don't have this message:

2025-06-12 08:43:06,038 INFO [SnapshotGenerator id=4] Creating new KRaft snapshot file snapshot 00000000000000144461-0000000418 because we have waited at least 60 minute(s). (org.apache.kafka.image.publisher.SnapshotGenerator) [kafka-4-snapshot-generator-event-handler]
2025-06-12 08:43:06,044 INFO [SnapshotEmitter id=4] Successfully wrote snapshot 00000000000000144461-0000000418 (org.apache.kafka.image.publisher.SnapshotEmitter) [kafka-4-snapshot-generator-event-handler]
2025-06-12 09:43:06,320 INFO [SnapshotGenerator id=4] Creating new KRaft snapshot file snapshot 00000000000000151660-0000000418 because we have waited at least 60 minute(s). (org.apache.kafka.image.publisher.SnapshotGenerator) [kafka-4-snapshot-generator-event-handler]
2025-06-12 09:43:06,328 INFO [SnapshotEmitter id=4] Successfully wrote snapshot 00000000000000151660-0000000418 (org.apache.kafka.image.publisher.SnapshotEmitter) [kafka-4-snapshot-generator-event-handler]

2025-06-12 08:19:25,759 INFO [SnapshotGenerator id=5] Creating new KRaft snapshot file snapshot 00000000000000141621-0000000418 because we have waited at least 60 minute(s). (org.apache.kafka.image.publisher.SnapshotGenerator) [kafka-5-snapshot-generator-event-handler]
2025-06-12 08:19:25,788 INFO [SnapshotEmitter id=5] Successfully wrote snapshot 00000000000000141621-0000000418 (org.apache.kafka.image.publisher.SnapshotEmitter) [kafka-5-snapshot-generator-event-handler]
2025-06-12 09:19:26,018 INFO [SnapshotGenerator id=5] Creating new KRaft snapshot file snapshot 00000000000000148820-0000000418 because we have waited at least 60 minute(s). (org.apache.kafka.image.publisher.SnapshotGenerator) [kafka-5-snapshot-generator-event-handler]
2025-06-12 09:19:26,031 INFO [SnapshotEmitter id=5] Successfully wrote snapshot 00000000000000148820-0000000418 (org.apache.kafka.image.publisher.SnapshotEmitter) [kafka-5-snapshot-generator-event-handler]

Furthermore, I'm running k8s 1.33.0 and already had to implement the workaround mentioned here to get the cluster operator running. Could there be further incompatibilities with this?

[ppatierno]

The #11386 is totally unrelated.
So the active controller is waiting for brokers to register and start the migration. It's normal the other controllers are not printing anything. Could you try to remove the strimzi.io/kraft: migration annotation and re-apply. If it doesn't work, try to roll a broker manually and see if it starts registering with the controller (looking at the active controller log) ?

[Davee02]

I was able to find the culprit while trying to remove the replicas and storage fields from the Kafka CR like you suggested in your initial answer. The update was rejected because

# kafkas.kafka.strimzi.io "kafka" was not valid:
# * spec.kafka.replicas: Required value
# * spec.kafka.storage: Required value

This made me wonder if perhaps my CRDs are out of date, as the cluster has been running for about 2 years now with automatic updates applied by the renovate bot.
After I updated the CRDs with the command kubectl apply -f https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.45.0/strimzi-crds-0.45.0.yaml, the migration continued successfully to the state KRaftPostMigration. I was able to complete the migration successfully, and it's now running fully in KRaft mode!

Thank you for your time, @ppatierno.

Tl;dr: Updating the CRDs as mentioned in the helm chart's readme solved it for me.

[ppatierno]

Glad you fixed the issue!