Election timed out before receiving sufficient vote responses to become candidate. EpochElection(voterStates={ 3=UNRECORDED, 4=GRANTED, 5=UNRECORDED })

[lyd100822]

k8s version 1.28.15 operator version 0.46.0 kafka 4.0.0
The controller keeps reporting an error "2025-06-30 10:31:01 INFO [kafka-4-raft-io-thread] KafkaRaftClient:3136 - [RaftManager id=4] Election timed out before receiving sufficient vote responses to become candidate. Current epoch election state: EpochElection(voterStates={3=VoterState(replicaKey=ReplicaKey(id=3, directoryId=), state=UNRECORDED), 4=VoterState(replicaKey=ReplicaKey(id=4, directoryId=), state=GRANTED), 5=VoterState(replicaKey=ReplicaKey(id=5, directoryId=), state=UNRECORDED)})
", and the broker keeps reporting an error
“2025-06-30 10:53:55 ERROR [main] BrokerServer:72 - [BrokerServer id=2] Received a fatal error while waiting for the controller to acknowledge that we are caught up
java.util.concurrent.CancellationException: null
at java.base/java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2478) ~[?:?]
at kafka.server.BrokerLifecycleManager$ShutdownEvent.run(BrokerLifecycleManager.scala:638) ~[kafka_2.13-4.0.0.jar:?]
at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:191) ~[kafka-server-common-4.0.0.jar:?]
at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
2025-06-30 10:53:56 INFO [main] BrokerServer:66 - [BrokerServer id=2] Transition from STARTING to STARTED
2025-06-30 10:53:56 INFO [kafka-2-metadata-loader-event-handler] MetadataLoader:233 - [MetadataLoader id=2] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet.
2025-06-30 10:53:56 INFO [broker-2-lifecycle-manager-event-handler] NodeToControllerChannelManagerImpl:66 - Node to controller channel manager for heartbeat shutdown
2025-06-30 10:53:56 INFO [kafka-2-metadata-loader-event-handler] MetadataLoader:233 - [MetadataLoader id=2] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet.
2025-06-30 10:53:56 INFO [kafka-2-raft-outbound-request-thread] NetworkClient:943 - [RaftManager id=2] Disconnecting from node 5 due to socket connection setup timeout. The timeout value is 30000 ms.
2025-06-30 10:53:56 ERROR [main] BrokerServer:82 - [BrokerServer id=2] Fatal error during broker startup. Prepare to shutdown
java.lang.RuntimeException: Received a fatal error while waiting for the controller to acknowledge that we are caught up
at org.apache.kafka.server.util.FutureUtils.waitWithLogging(FutureUtils.java:73) ~[kafka-server-common-4.0.0.jar:?]
at kafka.server.BrokerServer.startup(BrokerServer.scala:564) ~[kafka_2.13-4.0.0.jar:?]
at kafka.server.KafkaRaftServer.$anonfun$startup$2(KafkaRaftServer.scala:96) ~[kafka_2.13-4.0.0.jar:?]
at kafka.server.KafkaRaftServer.$anonfun$startup$2$adapted(KafkaRaftServer.scala:96) ~[kafka_2.13-4.0.0.jar:?]
at scala.Option.foreach(Option.scala:437) [scala-library-2.13.15.jar:?]
at kafka.server.KafkaRaftServer.startup(KafkaRaftServer.scala:96) [kafka_2.13-4.0.0.jar:?]
at kafka.Kafka$.main(Kafka.scala:97) [kafka_2.13-4.0.0.jar:?]
at kafka.Kafka.main(Kafka.scala) [kafka_2.13-4.0.0.jar:?]
Caused by: java.util.concurrent.CancellationException
at java.base/java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2478) ~[?:?]
at kafka.server.BrokerLifecycleManager$ShutdownEvent.run(BrokerLifecycleManager.scala:638) ~[kafka_2.13-4.0.0.jar:?]
at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:191) ~[kafka-server-common-4.0.0.jar:?]
at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
2025-06-30 10:53:56 INFO [main] BrokerServer:66 - [BrokerServer id=2] Transition from STARTED to SHUTTING_DOWN
2025-06-30 10:53:56 INFO [main] BrokerServer:66 - [BrokerServer id=2] shutting down
2025-06-30 10:53:56 INFO [main] SocketServer:66 - [SocketServer listenerType=BROKER, nodeId=2] Stopping socket server request processors”

[lyd100822]

[lyd100822]

my-cluster-kafka-log.zip

[scholzj]

I guess the brokers are not working because the controllers are not working. But it is not really clear to me from the logs what the issue is.

Perhaps a good start would be to share more details. Some obvious things are missing:

• The custom resources with the configuration
• Operator logs
• Environment ... e.g. what type of storage you use etc.
• Scenario how did it happened ... is it a new cluster? Or existing cluster? If it is new, did you make sure there are no pre-existing PVs/PVCs?

[lyd100822]

The template file used is temporary storage.

cat kafka-ephemeral.yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
name: controller
labels:
strimzi.io/cluster: my-cluster
spec:
replicas: 3
roles:
- controller
storage:
type: jbod
volumes:
- id: 0
type: ephemeral
kraftMetadata: shared

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
name: broker
labels:
strimzi.io/cluster: my-cluster
spec:
replicas: 3
roles:
- broker
storage:
type: jbod
volumes:
- id: 0
type: ephemeral
kraftMetadata: shared

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: my-cluster
annotations:
strimzi.io/node-pools: enabled
strimzi.io/kraft: enabled
spec:
kafka:
livenessProbe:
initialDelaySeconds: 1800
periodSeconds: 30
timeoutSeconds: 60
failureThreshold: 10
readinessProbe:
initialDelaySeconds: 180
periodSeconds: 30
timeoutSeconds: 60
failureThreshold: 10
version: 4.0.0
metadataVersion: 4.0-IV3
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
default.replication.factor: 3
min.insync.replicas: 2
entityOperator:
topicOperator: {}
userOperator: {}

[lyd100822]

operator-log
strimzi-cluster-operator-log.zip

[lyd100822]

Does it have anything to do with the network plugin? I currently use calico locally and Cilium in production. This is the only difference.

[scholzj]

I use Calico for my clusters without problems. But it will always depend on how you configure it. The networking things such as MTU might play a role. But also all kind of intrusive features might be source of problems I guess. (it is not really lear which of those does and doesn't work for you)

It might be also worth trying persistent storage if that helps - you will anyway need it for anything serious.