[Bug]: Active controller frequently changes — instability in KRaft controller quorum

[Mrmel94]

Bug Description

Hi,

I'm running Apache Kafka 4.0.0 IV3 with Strimzi on Kubernetes, using the KRaft mode with a dedicated controller quorum (3 controller nodes, 3 broker nodes). I'm using ArgoCD to deploy helm chart of strimzi.

This is a freshly installed cluster, and I've noticed that the active controller changes very frequently (on kafka-ui and logs), even when there is no workload or broker restarts.

The logs show that controller leadership changes regularly, despite the absence of crashes or visible errors (e.g., no stepping down due to failures). Example log entries for controllers :

2025-07-03 13:06:51 INFO  [quorum-controller-4-event-handler] EventPerformanceMonitor:174 - [QuorumController id=4] In the last 60000 ms period, 385 controller events were completed, which took an average of 9.70 ms each. The slowest event was writeNoOpRecord(1088212380), which took 37.71 ms.
2025-07-03 13:07:41 INFO  [quorum-controller-4-event-handler] ScramControlManager:345 - [QuorumController id=4] Replayed UserScramCredentialRecord modifying existing entry for kafkaui-user with mechanism SCRAM_SHA_512.
2025-07-03 13:07:41 INFO  [quorum-controller-4-event-handler] ScramControlManager:345 - [QuorumController id=4] Replayed UserScramCredentialRecord modifying existing entry for workidkafkauser with mechanism SCRAM_SHA_512.
2025-07-03 13:07:41 INFO  [quorum-controller-3-event-handler] ScramControlManager:345 - [QuorumController id=3] Replayed UserScramCredentialRecord modifying existing entry for kafkaui-user with mechanism SCRAM_SHA_512.
2025-07-03 13:07:41 INFO  [quorum-controller-3-event-handler] ScramControlManager:345 - [QuorumController id=3] Replayed UserScramCredentialRecord modifying existing entry for workidkafkauser with mechanism SCRAM_SHA_512.
2025-07-03 13:07:41 INFO  [quorum-controller-5-event-handler] ScramControlManager:345 - [QuorumController id=5] Replayed UserScramCredentialRecord modifying existing entry for kafkaui-user with mechanism SCRAM_SHA_512.
2025-07-03 13:07:41 INFO  [quorum-controller-5-event-handler] ScramControlManager:345 - [QuorumController id=5] Replayed UserScramCredentialRecord modifying existing entry for workidkafkauser with mechanism SCRAM_SHA_512.
2025-07-03 13:07:42 INFO  [quorum-controller-4-event-handler] ScramControlManager:345 - [QuorumController id=4] Replayed UserScramCredentialRecord modifying existing entry for kafkaui-user with mechanism SCRAM_SHA_512.
2025-07-03 13:07:42 INFO  [quorum-controller-4-event-handler] ScramControlManager:345 - [QuorumController id=4] Replayed UserScramCredentialRecord modifying existing entry for workidkafkauser with mechanism SCRAM_SHA_512.
2025-07-03 13:07:42 INFO  [quorum-controller-5-event-handler] ScramControlManager:345 - [QuorumController id=5] Replayed UserScramCredentialRecord modifying existing entry for kafkaui-user with mechanism SCRAM_SHA_512.
2025-07-03 13:07:42 INFO  [quorum-controller-5-event-handler] ScramControlManager:345 - [QuorumController id=5] Replayed UserScramCredentialRecord modifying existing entry for workidkafkauser with mechanism SCRAM_SHA_512.
2025-07-03 13:07:42 INFO  [quorum-controller-3-event-handler] ScramControlManager:345 - [QuorumController id=3] Replayed UserScramCredentialRecord modifying existing entry for kafkaui-user with mechanism SCRAM_SHA_512.
2025-07-03 13:07:42 INFO  [quorum-controller-3-event-handler] ScramControlManager:345 - [QuorumController id=3] Replayed UserScramCredentialRecord modifying existing entry for workidkafkauser with mechanism SCRAM_SHA_512.
2025-07-03 13:07:51 INFO  [quorum-controller-4-event-handler] EventPerformanceMonitor:174 - [QuorumController id=4] In the last 60000 ms period, 388 controller events were completed, which took an average of 9.65 ms each. The slowest event was alterUserScramCredentials(186000098), which took 37.58 ms.
2025-07-03 13:08:51 INFO  [quorum-controller-4-event-handler] EventPerformanceMonitor:174 - [QuorumController id=4] In the last 60000 ms period, 384 controller events were completed, which took an average of 9.71 ms each. The slowest event was writeNoOpRecord(43926563), which took 43.24 ms.
2025-07-03 13:09:42 INFO  [quorum-controller-4-event-handler] ScramControlManager:345 - [QuorumController id=4] Replayed UserScramCredentialRecord modifying existing entry for kafkaui-user with mechanism SCRAM_SHA_512.
2025-07-03 13:09:42 INFO  [quorum-controller-4-event-handler] ScramControlManager:345 - [QuorumController id=4] Replayed UserScramCredentialRecord modifying existing entry for workidkafkauser with mechanism SCRAM_SHA_512.
2025-07-03 13:09:42 INFO  [quorum-controller-5-event-handler] ScramControlManager:345 - [QuorumController id=5] Replayed UserScramCredentialRecord modifying existing entry for kafkaui-user with mechanism SCRAM_SHA_512.
2025-07-03 13:09:42 INFO  [quorum-controller-5-event-handler] ScramControlManager:345 - [QuorumController id=5] Replayed UserScramCredentialRecord modifying existing entry for workidkafkauser with mechanism SCRAM_SHA_512.
2025-07-03 13:09:42 INFO  [quorum-controller-3-event-handler] ScramControlManager:345 - [QuorumController id=3] Replayed UserScramCredentialRecord modifying existing entry for kafkaui-user with mechanism SCRAM_SHA_512.
2025-07-03 13:09:42 INFO  [quorum-controller-3-event-handler] ScramControlManager:345 - [QuorumController id=3] Replayed UserScramCredentialRecord modifying existing entry for workidkafkauser with mechanism SCRAM_SHA_512.

and broker logs :

2025-07-03 11:02:41 INFO  [broker-2-to-controller-forwarding-channel-manager] NetworkClient:1072 - [NodeToControllerChannelManager id=2 name=forwarding] Node 4 disconnected.
2025-07-03 11:29:42 INFO  [broker-2-to-controller-forwarding-channel-manager] NetworkClient:1072 - [NodeToControllerChannelManager id=2 name=forwarding] Node 4 disconnected.
2025-07-03 11:46:23 INFO  [kafka-0-snapshot-generator-event-handler] SnapshotGenerator:279 - [SnapshotGenerator id=0] Creating new KRaft snapshot file snapshot 00000000000001057305-0000000005 because we have waited at least 60 minute(s).
2025-07-03 11:46:23 INFO  [kafka-0-snapshot-generator-event-handler] SnapshotEmitter:155 - [SnapshotEmitter id=0] Successfully wrote snapshot 00000000000001057305-0000000005
2025-07-03 11:47:04 INFO  [kafka-2-snapshot-generator-event-handler] SnapshotGenerator:279 - [SnapshotGenerator id=2] Creating new KRaft snapshot file snapshot 00000000000001057388-0000000005 because we have waited at least 60 minute(s).
2025-07-03 11:47:04 INFO  [kafka-2-snapshot-generator-event-handler] SnapshotEmitter:155 - [SnapshotEmitter id=2] Successfully wrote snapshot 00000000000001057388-0000000005
2025-07-03 11:47:47 INFO  [kafka-1-snapshot-generator-event-handler] SnapshotGenerator:279 - [SnapshotGenerator id=1] Creating new KRaft snapshot file snapshot 00000000000001057477-0000000005 because we have waited at least 60 minute(s).
2025-07-03 11:47:47 INFO  [kafka-1-snapshot-generator-event-handler] SnapshotEmitter:155 - [SnapshotEmitter id=1] Successfully wrote snapshot 00000000000001057477-0000000005
2025-07-03 11:49:42 INFO  [broker-1-to-controller-forwarding-channel-manager] NetworkClient:1072 - [NodeToControllerChannelManager id=1 name=forwarding] Node 4 disconnected.
2025-07-03 12:01:45 INFO  [kafka-scheduler-2] LocalLog:644 - [LocalLog partition=strimzi.cruisecontrol.partitionmetricsamples-25, dir=/var/lib/kafka/data-0/kafka-log0/strimzi.cruisecontrol.partitionmetricsamples-25] Rolled new log segment at offset 23875 in 3 ms.
2025-07-03 12:01:45 INFO  [kafka-scheduler-2] ProducerStateManager:442 - [ProducerStateManager partition=strimzi.cruisecontrol.partitionmetricsamples-25] Wrote producer snapshot at offset 23875 with 1 producer ids in 0 ms.
2025-07-03 12:01:45 INFO  [kafka-scheduler-2] UnifiedLog:66 - [UnifiedLog partition=strimzi.cruisecontrol.partitionmetricsamples-25, dir=/var/lib/kafka/data-0/kafka-log0] Incremented log start offset to 23875 due to segment deletion
2025-07-03 12:01:45 INFO  [kafka-scheduler-2] UnifiedLog:66 - [UnifiedLog partition=strimzi.cruisecontrol.partitionmetricsamples-25, dir=/var/lib/kafka/data-0/kafka-log0] Deleting segment LogSegment(baseOffset=0, size=404864, lastModifiedTime=1751540470434, largestRecordTimestamp=1751540465313) due to log retention time 3600000ms breach based on the largest record timestamp in the segment
2025-07-03 12:01:46 INFO  [ReplicaFetcherThread-0-0] UnifiedLog:66 - [UnifiedLog partition=strimzi.cruisecontrol.partitionmetricsamples-25, dir=/var/lib/kafka/data-0/kafka-log2] Incremented log start offset to 23875 due to leader offset increment
2025-07-03 12:01:46 INFO  [ReplicaFetcherThread-0-0] UnifiedLog:66 - [UnifiedLog partition=strimzi.cruisecontrol.partitionmetricsamples-25, dir=/var/lib/kafka/data-0/kafka-log1] Incremented log start offset to 23875 due to leader offset increment
2025-07-03 12:02:26 INFO  [kafka-scheduler-5] LocalLog:644 - [LocalLog partition=strimzi.cruisecontrol.partitionmetricsamples-25, dir=/var/lib/kafka/data-0/kafka-log2/strimzi.cruisecontrol.partitionmetricsamples-25] Rolled new log segment at offset 23875 in 3 ms.
2025-07-03 12:02:26 INFO  [kafka-scheduler-5] UnifiedLog:66 - [UnifiedLog partition=strimzi.cruisecontrol.partitionmetricsamples-25, dir=/var/lib/kafka/data-0/kafka-log2] Deleting segment LogSegment(baseOffset=0, size=404864, lastModifiedTime=1751540470432, largestRecordTimestamp=1751540465313) due to log retention time 3600000ms breach based on the largest record timestamp in the segment
2025-07-03 12:02:45 INFO  [kafka-scheduler-5] LocalLog:860 - [LocalLog partition=strimzi.cruisecontrol.partitionmetricsamples-25, dir=/var/lib/kafka/data-0/kafka-log0/strimzi.cruisecontrol.partitionmetricsamples-25] Deleting segment files LogSegment(baseOffset=0, size=404864, lastModifiedTime=1751540470434, largestRecordTimestamp=1751540465313)
2025-07-03 12:02:45 INFO  [kafka-scheduler-5] LogSegment:801 - Deleted log /var/lib/kafka/data-0/kafka-log0/strimzi.cruisecontrol.partitionmetricsamples-25/00000000000000000000.log.deleted.
2025-07-03 12:02:45 INFO  [kafka-scheduler-5] LogSegment:801 - Deleted offset index /var/lib/kafka/data-0/kafka-log0/strimzi.cruisecontrol.partitionmetricsamples-25/00000000000000000000.index.deleted.
2025-07-03 12:02:45 INFO  [kafka-scheduler-5] LogSegment:801 - Deleted time index /var/lib/kafka/data-0/kafka-log0/strimzi.cruisecontrol.partitionmetricsamples-25/00000000000000000000.timeindex.deleted.
2025-07-03 12:03:09 INFO  [kafka-scheduler-0] LocalLog:644 - [LocalLog partition=strimzi.cruisecontrol.partitionmetricsamples-25, dir=/var/lib/kafka/data-0/kafka-log1/strimzi.cruisecontrol.partitionmetricsamples-25] Rolled new log segment at offset 23875 in 1 ms.
2025-07-03 12:03:09 INFO  [kafka-scheduler-0] UnifiedLog:66 - [UnifiedLog partition=strimzi.cruisecontrol.partitionmetricsamples-25, dir=/var/lib/kafka/data-0/kafka-log1] Deleting segment LogSegment(baseOffset=0, size=404864, lastModifiedTime=1751540470432, largestRecordTimestamp=1751540465313) due to log retention time 3600000ms breach based on the largest record timestamp in the segment
2025-07-03 12:03:26 INFO  [kafka-scheduler-3] LocalLog:860 - [LocalLog partition=strimzi.cruisecontrol.partitionmetricsamples-25, dir=/var/lib/kafka/data-0/kafka-log2/strimzi.cruisecontrol.partitionmetricsamples-25] Deleting segment files LogSegment(baseOffset=0, size=404864, lastModifiedTime=1751540470432, largestRecordTimestamp=1751540465313)
2025-07-03 12:03:26 INFO  [kafka-scheduler-3] LogSegment:801 - Deleted log /var/lib/kafka/data-0/kafka-log2/strimzi.cruisecontrol.partitionmetricsamples-25/00000000000000000000.log.deleted.
2025-07-03 12:03:26 INFO  [kafka-scheduler-3] LogSegment:801 - Deleted offset index /var/lib/kafka/data-0/kafka-log2/strimzi.cruisecontrol.partitionmetricsamples-25/00000000000000000000.index.deleted.
2025-07-03 12:03:26 INFO  [kafka-scheduler-3] LogSegment:801 - Deleted time index /var/lib/kafka/data-0/kafka-log2/strimzi.cruisecontrol.partitionmetricsamples-25/00000000000000000000.timeindex.deleted.
2025-07-03 12:04:09 INFO  [kafka-scheduler-5] LocalLog:860 - [LocalLog partition=strimzi.cruisecontrol.partitionmetricsamples-25, dir=/var/lib/kafka/data-0/kafka-log1/strimzi.cruisecontrol.partitionmetricsamples-25] Deleting segment files LogSegment(baseOffset=0, size=404864, lastModifiedTime=1751540470432, largestRecordTimestamp=1751540465313)
2025-07-03 12:04:09 INFO  [kafka-scheduler-5] LogSegment:801 - Deleted log /var/lib/kafka/data-0/kafka-log1/strimzi.cruisecontrol.partitionmetricsamples-25/00000000000000000000.log.deleted.
2025-07-03 12:04:09 INFO  [kafka-scheduler-5] LogSegment:801 - Deleted offset index /var/lib/kafka/data-0/kafka-log1/strimzi.cruisecontrol.partitionmetricsamples-25/00000000000000000000.index.deleted.
2025-07-03 12:04:09 INFO  [kafka-scheduler-5] LogSegment:801 - Deleted time index /var/lib/kafka/data-0/kafka-log1/strimzi.cruisecontrol.partitionmetricsamples-25/00000000000000000000.timeindex.deleted.
2025-07-03 12:41:42 INFO  [broker-2-to-controller-forwarding-channel-manager] NetworkClient:1072 - [NodeToControllerChannelManager id=2 name=forwarding] Node 4 disconnected.
2025-07-03 12:46:23 INFO  [kafka-0-snapshot-generator-event-handler] SnapshotGenerator:279 - [SnapshotGenerator id=0] Creating new KRaft snapshot file snapshot 00000000000001064595-0000000005 because we have waited at least 60 minute(s).
2025-07-03 12:46:23 INFO  [kafka-0-snapshot-generator-event-handler] SnapshotEmitter:155 - [SnapshotEmitter id=0] Successfully wrote snapshot 00000000000001064595-0000000005
2025-07-03 12:47:04 INFO  [kafka-2-snapshot-generator-event-handler] SnapshotGenerator:279 - [SnapshotGenerator id=2] Creating new KRaft snapshot file snapshot 00000000000001064678-0000000005 because we have waited at least 60 minute(s).
2025-07-03 12:47:04 INFO  [kafka-2-snapshot-generator-event-handler] SnapshotEmitter:155 - [SnapshotEmitter id=2] Successfully wrote snapshot 00000000000001064678-0000000005
2025-07-03 12:47:47 INFO  [kafka-1-snapshot-generator-event-handler] SnapshotGenerator:279 - [SnapshotGenerator id=1] Creating new KRaft snapshot file snapshot 00000000000001064769-0000000005 because we have waited at least 60 minute(s).
2025-07-03 12:47:47 INFO  [kafka-1-snapshot-generator-event-handler] SnapshotEmitter:155 - [SnapshotEmitter id=1] Successfully wrote snapshot 00000000000001064769-0000000005
2025-07-03 12:57:42 INFO  [broker-2-to-controller-forwarding-channel-manager] NetworkClient:1072 - [NodeToControllerChannelManager id=2 name=forwarding] Node 4 disconnected.
2025-07-03 13:05:42 INFO  [broker-1-to-controller-forwarding-channel-manager] NetworkClient:1072 - [NodeToControllerChannelManager id=1 name=forwarding] Node 4 disconnected.

Steps to reproduce

Deploy a fresh cluster with this kafka.yaml

kind: KafkaNodePool
metadata:
  name: controller
  labels:
    strimzi.io/cluster: wid-kafka
spec:
  replicas: 3
  roles:
    - controller
  resources:
      requests:
        memory: 2Gi
        cpu: 500m
      limits:
        memory: 2Gi
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 20Gi
        kraftMetadata: shared
        deleteClaim: false
---

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: broker
  labels:
    strimzi.io/cluster: wid-kafka
spec:
  replicas: 3
  roles:
    - broker
  resources:
      requests:
        memory: 2Gi
        cpu: 500m
      limits:
        memory: 2Gi
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 20Gi
        kraftMetadata: shared
        deleteClaim: false
---

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: wid-kafka
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - wid-kafka
                topologyKey: "kubernetes.io/hostname"
    authorization:
      type: simple
    version: 4.0.0
    metadataVersion: 4.0-IV3
    logging:
      type: inline
      loggers:
        kafka.root.logger.level: INFO
    listeners:

      - name: internal 
        port: 9093
        type: internal
        tls: false
        authentication:
          type: scram-sha-512

      - name: external
        port: 9092
        type: loadbalancer
        tls: false
        authentication:
          type: scram-sha-512
        configuration:
          bootstrap:
            loadBalancerIP: 10.40.2.80
    config:
      auto.create.topics.enable: "false"
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
  entityOperator:
    topicOperator: {}
    userOperator: {}
  cruiseControl:
    autoRebalance:
      - mode: add-brokers
        template:
          name: my-add-brokers-rebalancing-template
      - mode: remove-brokers
        template:
          name: my-remove-brokers-rebalancing-template

---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: my-add-brokers-rebalancing-template
  annotations:
    strimzi.io/rebalance-template: "true"
# no goals specified, using the default goals from the Cruise Control configuration
spec: {}

---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: my-remove-brokers-rebalancing-template
  annotations:
    strimzi.io/rebalance-template: "true"
# no goals specified, using the default goals from the Cruise Control configuration
spec: {}

Expected behavior

The active controller should remain stable and only change in the event of a failure, maintenance event, or leadership transfer initiated by configuration.

Strimzi version

0.46.1

Kubernetes version

Kubernetes 1.28.7

Installation method

Helm chart with argocd

Infrastructure

Tanzu Kubernetes

Configuration files and logs

The zip report generated by report.sh is attached to this issue

report-03-07-2025_13-37-33.zip

Additional context

No response

[scholzj]

Please keep in mind that this is fully in Apache Kafka's control. Not much Strimzi can do about this. An issue like this could be due to a Kafka issue. But could also be caused by some environmental factors. Such as network stability, resourcing issues etc. You can easily try some of them - e.g. giving it more CPU resources. While others might be harder to debug.

[Mrmel94]

Hello, thank you for your reply, I've already tried to increase the CPU / memory resources of my controllers / brokers but without success... The purpose of my message was to know if it's not my kafka.yaml deployment that has a problem or if someone else had already had a similar problem starting from a clean installation of strimzi.

I already have several other applications and have never had a network or IO problem. But since I have the impression that there's hardly anyone else in my case, I think the problem does indeed lie with my infrastructure.