Issue with External Bootstrap Server in Strimzi Kafka Cluster

[amiteka1300]

Hi Team,

We’re currently running a Strimzi Kafka cluster and have observed that we’re able to list Kafka topics using localhost:9092. However, when using the external bootstrap server, we’re encountering the following error :

org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: findAllBrokers
Wrapped by: org.apache.kafka.common.KafkaException: Failed to find brokers to send ListGroups
Additionally, we’ve noticed that the cluster ID sometimes changes, and metadata inconsistencies are observed as well.
here is our config

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: kafka-node-pool-1
  namespace: kafka-front
  labels:
    strimzi.io/cluster: kafka-cluster
spec:
  template: 
    pod:
      tolerations:
        - key: "env/kafka"
          operator: "Exists"
          effect: "NoSchedule"  
      affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
               - matchExpressions:
                  - key: karpenter.sh/nodepool
                    operator: In
                    values:
                      - kafka-on-demand
  replicas: 1
  roles:
    - controller
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        class: gp3
        size: 30Gi
        deleteClaim: false
  resources:
    requests:
      memory: 4Gi
      cpu: 500m
    limits:
      memory: 8Gi
      cpu: 1   
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-cluster
  namespace: kafka-front
  labels:
    strimzi.io/cluster: kafka-cluster
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 3.8.0
    metadataVersion: "3.8"
    template:
      externalBootstrapService:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
            service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-xxxxxxx1, subnet-xxxxxxx2
            service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
            service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
      perPodService:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
            service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-xxxxxxx1, subnet-xxxxxxx2
            service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
            service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip

    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: external
        port: 9094
        type: loadbalancer
        tls: false
        configuration:
          bootstrap:
            alternativeNames: 
              - kafka-bootstrap.dev.example.com
              - kafka-bootstrap.staging.example.com
    resources:
      requests:
        memory: 4Gi
        cpu: 500m
      limits:
        memory: 8Gi
        cpu: "1"   
    jvmOptions:
      -Xms: 2048m
      -Xmx: 2048m        
    logging:
      type: inline
      loggers:
        kafka.root.logger.level: INFO            

    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      num.partitions: 1
      num.recovery.threads.per.data.dir: 1
      log.retention.hours: 48
      log.segment.bytes: 500000000
      log.retention.check.interval.ms: 300000
      num.network.threads: 3
      num.io.threads: 8
      socket.send.buffer.bytes: 102400
      socket.receive.buffer.bytes: 102400
      socket.request.max.bytes: 104857600
      group.initial.rebalance.delay.ms: 0
      zookeeper.connection.timeout.ms: 6000
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
  zookeeper:
    replicas: 1
    storage:
      type: persistent-claim
      size: 50Gi
      deleteClaim: false
    resources:
      requests:
        cpu: 1
        memory: 1Gi
      limits:
        cpu: 2
        memory: 4Gi
    logging:
      type: inline
      loggers:
        zookeeper.root.logger: "INFO"

  entityOperator:
    topicOperator: {}
    userOperator: {}
    template:
      pod:
        tolerations:
          - key: "env/kafka"
            operator: "Exists"
            effect: "NoSchedule"
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kafka-metrics
  namespace: kafka-front
  labels:
    app: strimzi
data:
  kafka-metrics-config.yml: |    
    lowercaseOutputName: true
    rules:
    # [Rules kept unchanged – they are generic metric patterns not needing sanitization]


Has anyone experienced a similar issue, or is there a recommended solution to fix this?

Thanks,
Amit

[scholzj]

You should never run any commands from the broker pods. You should always run them from your own external Pods. So as a result, you should never use any localhost address but use the address you have in the Kafka CR status section. I think that is all what can be said without understanding what exactly are you doing.

[amiteka1300]

We are currently using our testing account to test something. When attempting to list Kafka topics using the bootstrap server kafka-bootstrap.dev.example.com (which is also specified in the Kafka configuration), we encounter the following error:

org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: findAllBrokers
Wrapped by: org.apache.kafka.common.KafkaException: Failed to find brokers to send ListGroups.
Could you please help us understand what might be causing this issue?

[im-konge]

We are currently using our testing account to test something.

That's not saying much what you are actually doing. What command you are using or what configuration you are using etc.
Also the error you posted is not much saying either.

[amiteka1300]

Hi @im-konge ,

Thank you for your response.
We have two issues.

Setup:

We are running kafka on EKS with Karpenter Provisioning.

Strimzi Version: 0.45.0
Kafka Version: 3.8.0

1. We had a Cluster running for over a year and half. We were making some changes to move Kafka to dedicated nodes and accidentally reapplied some additional kafka configurations and load balancer configurations. Kafka wasn’t running for a while and after deleting and reapplying the above right config, kafka nodepool pods started running but we got the following error:

Exception in thread "main" java.lang.RuntimeException: Invalid cluster.id in: /var/lib/kafka/data-0/kafka-log0/meta.properties. Expected bpP6rckOTUS94PxUKgMRTA, but read n-nzUFCFTiyTIuF5aSsYjQ

We resolved it by going into the Pod and deleting the metadata file and reapplying the configuration which prompted kafka to re create the metadata file which Caused the below issue?


3. When we connect to kafka using the k8 service or localhost on port 9092 from within the cluster , it works fine. But when we try to access kafka on the external-bootstrap advertised URL(Kafka-bootstrap.example.com:9094 like mentioned in the above config) via the provisioned Network Load Balancer, it doesn’t connect and we are not able to list topics or send/receive any data and we get timed out. Need some help in debugging and resolving this. Thanks in advance. Let us know if any more info is needed.

Error while executing topic command :

./kafka-topics.sh --bootstrap-server Kafka-bootstrap.example.com:9094 --list
Timed out waiting to send the call. Call: listTopics
[2025-08-06 08:14:42,202] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: listTopics
(org.apache.kafka.tools.TopicCommand)

[scholzj]

Re 1) This was disucssed many times in other discussions and I think it is covered also in docs. You have to properly set the cluster Id when recorvering the cluster from only the PVs/PVCs. Please search the docs and the previous discussions.

Re 2) That suggests some networking issue. Maybe wrong DNS setup or some other networking problem?

[amiteka1300]

Hi @scholzj ,
Thanks for the help . Error was on our target group end. somehow it was not registering kafka nodes in the target group, so we were getting this error.