[Bug]: entity-operator keeps failing, both user-operator as topic-operator. #11675

dj-wasabi · 2025-07-21T13:37:26Z

dj-wasabi
Jul 21, 2025

Bug Description

For example, the user-operator fails with log:

2025-07-21 13:09:51 INFO  Main:76 - UserOperator configuration is UserOperatorBuilderConfig{
        namespace='monitoring'
        reconciliationIntervalMs=60000
        kafkaBootstrapServers='${KAKFA_NAME}-kafka-bootstrap:9091'
        labels=`Labels{strimzi.io/cluster=${KAKFA_NAME}}'
        caCertSecretName='${KAKFA_NAME}-clients-ca-cert'
        caKeySecretName='${KAKFA_NAME}-clients-ca'
        clusterCaCertSecretName='${KAKFA_NAME}-cluster-ca-cert'
        euoKeySecretName='${KAKFA_NAME}-entity-user-operator-certs'
        euoCertName='entity-operator.crt'
        euoKeyName='entity-operator.key'
        caNamespace='monitoring'
        secretPrefix=''
        clientsCaValidityDays=365
        clientsCaRenewalDays=30
        aclsAdminApiSupported=false
        scramPasswordLength=32
        maintenanceWindows=`null'
        kafkaAdminClientConfiguration=`{}'
        operationTimeoutMs=300000
        workQueueSize=1024
        controllerThreadPoolSize=50
        cacheRefresh=15000
        batchQueueSize=1024
        batchMaxBlockSize=100
        batchMaxBlockTime=100
        userOperationsThreadPoolSize=4
        featureGates='FeatureGates(DummyFeatureGate=false)'}
Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [Secret]  with name: [${KAKFA_NAME}-cluster-ca-cert]  in namespace: [monitoring]  failed.
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:139)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.requireFromServer(BaseOperation.java:195)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:149)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:98)
        at io.strimzi.operator.common.operator.resource.concurrent.AbstractNamespacedResourceOperator.get(AbstractNamespacedResourceOperator.java:316)
        at io.strimzi.operator.user.Main.getSecret(Main.java:173)
        at io.strimzi.operator.user.Main.createAdminClient(Main.java:147)
        at io.strimzi.operator.user.Main.main(Main.java:82)
Caused by: java.io.IOException
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:504)
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleGet(OperationSupport.java:467)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleGet(BaseOperation.java:792)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.requireFromServer(BaseOperation.java:193)
        ... 6 more
Caused by: java.net.ConnectException
        at java.net.http/jdk.internal.net.http.common.Utils.toConnectException(Utils.java:1083)
        at java.net.http/jdk.internal.net.http.PlainHttpConnection.connectAsync(PlainHttpConnection.java:198)
        at java.net.http/jdk.internal.net.http.PlainHttpConnection.checkRetryConnect(PlainHttpConnection.java:230)
        at java.net.http/jdk.internal.net.http.PlainHttpConnection.lambda$connectAsync$1(PlainHttpConnection.java:206)
        at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
        at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.nio.channels.ClosedChannelException
        at java.base/sun.nio.ch.SocketChannelImpl.ensureOpen(SocketChannelImpl.java:195)
        at java.base/sun.nio.ch.SocketChannelImpl.beginConnect(SocketChannelImpl.java:760)
        at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:848)
        at java.net.http/jdk.internal.net.http.PlainHttpConnection.lambda$connectAsync$0(PlainHttpConnection.java:183)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:569)
        at java.net.http/jdk.internal.net.http.PlainHttpConnection.connectAsync(PlainHttpConnection.java:185)
        ... 9 more

And yet, the secret does exist:

$ kubectl get secrets ${KAKFA_NAME}-cluster-ca-cert
NAME                            TYPE     DATA   AGE
${KAKFA_NAME}-cluster-ca-cert   Opaque   3      3m18s

The pod keeps on failing. Similar with the topic-operator

2025-07-21 13:08:32 INFO  AppInfoParser:125 - Kafka version: 4.0.0
2025-07-21 13:08:32 INFO  AppInfoParser:126 - Kafka commitId: 0ebdee20e38bf984
2025-07-21 13:08:32 INFO  AppInfoParser:127 - Kafka startTimeMs: 1753103312052
2025-07-21 13:08:33 INFO  TopicOperatorMain:91 - TopicOperator 0.47.0 is starting
2025-07-21 13:08:33 INFO  TopicOperatorMain:97 - Installing shutdown hook
2025-07-21 13:08:33 INFO  TopicOperatorMain:99 - Starting health and metrics
2025-07-21 13:08:33 INFO  TopicOperatorMain:101 - Starting queue
2025-07-21 13:08:33 INFO  TopicOperatorMain:116 - Starting informer
2025-07-21 13:08:33 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkatopics' with unstable version 'v1beta2'
2025-07-21 13:09:09 ERROR Reflector:168 - listSyncAndWatch failed for kafka.strimzi.io/v1beta2/namespaces/monitoring/kafkatopics, will stop
java.util.concurrent.CompletionException: java.net.ConnectException
        at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:368) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:377) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1152) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: java.net.ConnectException
        at jdk.internal.net.http.common.Utils.toConnectException(Utils.java:1083) ~[java.net.http:?]
        at jdk.internal.net.http.PlainHttpConnection.connectAsync(PlainHttpConnection.java:198) ~[java.net.http:?]
        at jdk.internal.net.http.PlainHttpConnection.checkRetryConnect(PlainHttpConnection.java:230) ~[java.net.http:?]
        at jdk.internal.net.http.PlainHttpConnection.lambda$connectAsync$1(PlainHttpConnection.java:206) ~[java.net.http:?]
        at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911) ~[?:?]
        ... 5 more
Caused by: java.nio.channels.ClosedChannelException
        at sun.nio.ch.SocketChannelImpl.ensureOpen(SocketChannelImpl.java:195) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.beginConnect(SocketChannelImpl.java:760) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:848) ~[?:?]
        at jdk.internal.net.http.PlainHttpConnection.lambda$connectAsync$0(PlainHttpConnection.java:183) ~[java.net.http:?]
        at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
        at jdk.internal.net.http.PlainHttpConnection.connectAsync(PlainHttpConnection.java:185) ~[java.net.http:?]
        at jdk.internal.net.http.PlainHttpConnection.checkRetryConnect(PlainHttpConnection.java:230) ~[java.net.http:?]
        at jdk.internal.net.http.PlainHttpConnection.lambda$connectAsync$1(PlainHttpConnection.java:206) ~[java.net.http:?]
        at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911) ~[?:?]
        ... 5 more
2025-07-21 13:09:09 INFO  TopicOperatorMain:176 - isAlive returning false because informer is not running
2025-07-21 13:09:09 INFO  TopicOperatorMain:176 - isAlive returning false because informer is not running
Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:109)
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:102)
        at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:176)
        at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:186)
        at io.fabric8.kubernetes.client.informers.impl.DefaultSharedIndexInformer.run(DefaultSharedIndexInformer.java:188)
        at io.strimzi.operator.topic.TopicOperatorMain.start(TopicOperatorMain.java:117)
        at io.strimzi.operator.topic.TopicOperatorMain.main(TopicOperatorMain.java:166)
Caused by: java.net.ConnectException
        at java.net.http/jdk.internal.net.http.common.Utils.toConnectException(Utils.java:1083)
        at java.net.http/jdk.internal.net.http.PlainHttpConnection.connectAsync(PlainHttpConnection.java:198)
        at java.net.http/jdk.internal.net.http.PlainHttpConnection.checkRetryConnect(PlainHttpConnection.java:230)
        at java.net.http/jdk.internal.net.http.PlainHttpConnection.lambda$connectAsync$1(PlainHttpConnection.java:206)
        at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
        at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
        Suppressed: java.lang.Throwable: waiting here
                at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:175)
                at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:186)
                at io.fabric8.kubernetes.client.informers.impl.DefaultSharedIndexInformer.run(DefaultSharedIndexInformer.java:188)
                at io.strimzi.operator.topic.TopicOperatorMain.start(TopicOperatorMain.java:117)
                at io.strimzi.operator.topic.TopicOperatorMain.main(TopicOperatorMain.java:166)
Caused by: java.nio.channels.ClosedChannelException
        at java.base/sun.nio.ch.SocketChannelImpl.ensureOpen(SocketChannelImpl.java:195)
        at java.base/sun.nio.ch.SocketChannelImpl.beginConnect(SocketChannelImpl.java:760)
        at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:848)
        at java.net.http/jdk.internal.net.http.PlainHttpConnection.lambda$connectAsync$0(PlainHttpConnection.java:183)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:569)
        at java.net.http/jdk.internal.net.http.PlainHttpConnection.connectAsync(PlainHttpConnection.java:185)
        ... 9 more
2025-07-21 13:09:10 INFO  TopicOperatorMain:176 - isAlive returning false because informer is not running

Other than that, no useful (error) message.

Steps to reproduce

The following takes places in a K3d (k3d cluster create mycluster --agents 4, as I can not deploy this yet to our test cluster)

Install the operator helm chart 0.47.```strimzi-kafka-operator: replicas: 1 # watchNamespaces:

strimzi-kafka-operator:
  replicas: 1
  # watchNamespaces:
  #   - monitoring

  rbac:
    create: yes

There is no difference in installing both strimzi-operator and kafka cluster in the same or different namespaces.

Install the manifests, I excluded the log config and the metrics, these are taken from the examples.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: {{ .Values.name }}
  namespace: {{ .Values.namespace }}
  annotations:
    strimzi.io/node-pools: {{ .Values.nodepools }}
    strimzi.io/kraft: {{ .Values.kraft }}
spec:
  kafka:
    template:
      clusterCaCert:
        metadata:
          labels:
            monitor-cert: "true"
      podDisruptionBudget:
        maxUnavailable: 2
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                  - key: {{ .Values.nodelabel.key }}
                    operator: In
                    values:
                    - {{ .Values.nodelabel.value }}
    version: {{ .Values.version }}
    jvmOptions:
      -Xms: 2048m
      -Xmx: 2048m
    logging:
      type: external
      valueFrom:
        configMapKeyRef:
          name: customconfigmap
          key: kafka-log4j.properties
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 3
      default.replication.factor: 3
      min.insync.replicas: 2
      ssl.cipher.suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384"
      ssl.enabled.protocols: "TLSv1.2, TLSv1.3"
      ssl.protocol: "TLSv1.3"
      auto.create.topics.enable: false
      log.message.timestamp.type: LogAppendTime
      message.max.bytes: 100000000
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: prometheus-metrics
          key: kafka-metrics-config.yml
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"
  entityOperator:
    template:
      topicOperatorContainer:
        env:
        - name: STRIMZI_USE_ZOOKEEPER_TOPIC_STORE
          value: "true"
    topicOperator:
      watchedNamespace: {{ .Values.namespace }}
      reconciliationIntervalMs: 60000
      logging:
        type: inline
        loggers:
          rootLogger.level: "INFO"
      resources:
        requests:
          memory: 512Mi
          cpu: "500m"
        limits:
          memory: 756Mi
          cpu: "1"
    userOperator:
      watchedNamespace: {{ .Values.namespace }}
      reconciliationIntervalMs: 60000
      logging:
        type: inline
        loggers:
          rootLogger.level: "INFO"
      resources:
        requests:
          memory: 512Mi
          cpu: "500m"
        limits:
          memory: 756Mi
          cpu: "1"

nodepools:

---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: kafka
  labels:
    strimzi.io/cluster: {{ .Values.name }}
spec:
  replicas: {{ .Values.replicas.brokers }}
  roles:
    - broker
  resources:
    requests:
      memory: {{ .Values.kafka.broker.requestMemory }}
      cpu: {{ .Values.kafka.broker.requestCpu }}
    limits:
      memory: {{ .Values.kafka.broker.limitMemory }}
      cpu: {{ .Values.kafka.broker.limitCpu }}
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: {{ .Values.storagesize.broker }}
        deleteClaim: false
  template:
    pod:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: {{ .Values.nodelabel.key }}
                    operator: In
                    values:
                      - {{ .Values.nodelabel.value }}
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: {{ .Values.name }}-controllers
  labels:
    strimzi.io/cluster: {{ .Values.name }}
spec:
  replicas: {{ .Values.replicas.controllers | default 3 }}
  roles:
    - controller
  resources:
    requests:
      memory: {{ .Values.kafka.controller.requestMemory }}
      cpu: {{ .Values.kafka.controller.requestCpu }}
    limits:
      memory: {{ .Values.kafka.controller.limitMemory }}
      cpu: {{ .Values.kafka.controller.limitCpu }}
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: {{ .Values.storagesize.controller }}
        deleteClaim: false
  template:
    pod:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: {{ .Values.nodelabel.key }}
                    operator: In
                    values:
                      - {{ .Values.nodelabel.value }}

Add them to be sure, network policies:


---

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-to-{{ .Values.name }}
  namespace: {{ .Values.namespace }}
spec:
  egress:
  - ports:
    - port: 9090
      protocol: TCP
    - port: 9091
      protocol: TCP
    - port: 9092
      protocol: TCP
    to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: {{ .Values.namespace }}
      podSelector:
        matchLabels:
          strimzi.io/name: {{ .Values.name }}-kafka
  podSelector: {}
  policyTypes:
  - Egress

---

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: {{ .Values.name }}-entity-operator
  namespace: {{ .Values.namespace }}
spec:
  egress:
  - {}
  ingress:
  - ports:
    - port: 8080
      protocol: TCP
  - ports:
    - port: 8081
      protocol: TCP
  podSelector:
    matchLabels:
      strimzi.io/name: {{ .Values.name }}-entity-operator
  policyTypes:
  - Ingress
  - Egress

---

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: {{ .Values.name }}-kafka-exporter
  namespace: {{ .Values.namespace }}
spec:
  ingress:
  - ports:
    - port: 9404
      protocol: TCP
  podSelector:
    matchLabels:
      strimzi.io/name: {{ .Values.name }}-kafka-exporter
  policyTypes:
  - Ingress

---

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: {{ .Values.name }}-network-policy-kafka
  namespace: {{ .Values.namespace }}
spec:
  egress:
  - ports:
    - port: 9091
      protocol: TCP
    to:
    - podSelector:
         matchLabels:
          strimzi.io/name: {{ .Values.name }}-kafka
  ingress:
  - from:
    - podSelector:
        matchLabels:
          strimzi.io/name: {{ .Values.name }}-kafka
    ports:
    - port: 9090
      protocol: TCP
  - from:
    - namespaceSelector: {}
      podSelector:
        matchLabels:
          strimzi.io/kind: cluster-operator
    - podSelector:
         matchLabels:
          strimzi.io/name: {{ .Values.name }}-kafka
    - podSelector:
        matchLabels:
          strimzi.io/name: {{ .Values.name }}-entity-operator
    - podSelector:
        matchLabels:
          strimzi.io/name: {{ .Values.name }}-kafka-exporter
    ports:
    - port: 9091
      protocol: TCP
  - from:
    - namespaceSelector: {}
      podSelector:
        matchLabels:
          strimzi.io/kind: cluster-operator
    ports:
    - port: 8443
      protocol: TCP
  - ports:
    - port: 9092
      protocol: TCP
  - ports:
    - port: 9404
      protocol: TCP
  podSelector:
    matchLabels:
      strimzi.io/name: {{ .Values.name }}-kafka
  policyTypes:
  - Ingress
  - Egress

The values.yaml:

---
name: ${KAKFA_NAME}
namespace: monitoring

nodepools: enabled
kraft: enabled

replicas:
  brokers: 3
  controllers: 3

storagesize:
  broker: 12Gi
  controller: 8Gi

nodelabel:
 key: lifecycle
 value: OnDemand

instancelabel: ${KAKFA_NAME}
instancename: kafka

version: "3.9.0"

kafka:
  broker:
    requestMemory: 2Gi
    requestCpu: "1"
    limitMemory: 2Gi
    limitCpu: "2"
  controller:
    requestMemory: 2Gi
    requestCpu: "1"
    limitMemory: 2Gi
    limitCpu: "2"

Expected behavior

To have them running, (possibly) without any restarts:

NAME                                                        READY   STATUS             RESTARTS         AGE
kafka-logging-entity-operator-59f4ccf666-lbm7z              0/2     CrashLoopBackOff   17 (3m57s ago)   24m

Strimzi version

0.47

Kubernetes version

1.31

Installation method

Helm charts

Infrastructure

3d

Configuration files and logs

2025-07-21 13:24:31 INFO  ClusterOperator:135 - Triggering periodic reconciliation for namespace monitoring
2025-07-21 13:24:31 INFO  AbstractOperator:266 - Reconciliation #62(timer) Kafka(monitoring/${KAKFA_NAME}): Kafka ${KAKFA_NAME} will be checked for creation or modification
2025-07-21 13:24:32 INFO  KafkaRoller:751 - Reconciliation #62(timer) Kafka(monitoring/${KAKFA_NAME}): Dynamic update of pod ${KAKFA_NAME}-kafka-0/0 was successful.
2025-07-21 13:24:32 INFO  KafkaRoller:751 - Reconciliation #62(timer) Kafka(monitoring/${KAKFA_NAME}): Dynamic update of pod ${KAKFA_NAME}-kafka-1/1 was successful.
2025-07-21 13:24:32 INFO  KafkaRoller:751 - Reconciliation #62(timer) Kafka(monitoring/${KAKFA_NAME}): Dynamic update of pod ${KAKFA_NAME}-kafka-2/2 was successful.
2025-07-21 13:24:34 INFO  AbstractWatchManager:447 - Unknown Watch error received 8 times without progress, will reconnect if possible
java.util.concurrent.CompletionException: java.net.ConnectException
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1141) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
        at io.fabric8.kubernetes.client.jdkhttp.JdkHttpClientImpl.lambda$buildWebSocketDirect$5(JdkHttpClientImpl.java:333) ~[io.fabric8.kubernetes-httpclient-jdk-7.3.1.jar:?]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
        at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:844) ~[?:?]
        at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: java.net.ConnectException
        at jdk.internal.net.http.common.Utils.toConnectException(Utils.java:1083) ~[java.net.http:?]
        at jdk.internal.net.http.PlainHttpConnection.connectAsync(PlainHttpConnection.java:198) ~[java.net.http:?]
        at jdk.internal.net.http.PlainHttpConnection.checkRetryConnect(PlainHttpConnection.java:230) ~[java.net.http:?]
        at jdk.internal.net.http.PlainHttpConnection.lambda$connectAsync$1(PlainHttpConnection.java:206) ~[java.net.http:?]
        at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        ... 1 more
Caused by: java.nio.channels.ClosedChannelException
        at sun.nio.ch.SocketChannelImpl.ensureOpen(SocketChannelImpl.java:195) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.beginConnect(SocketChannelImpl.java:760) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:848) ~[?:?]
        at jdk.internal.net.http.PlainHttpConnection.lambda$connectAsync$0(PlainHttpConnection.java:183) ~[java.net.http:?]
        at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
        at jdk.internal.net.http.PlainHttpConnection.connectAsync(PlainHttpConnection.java:185) ~[java.net.http:?]
        at jdk.internal.net.http.PlainHttpConnection.checkRetryConnect(PlainHttpConnection.java:230) ~[java.net.http:?]
        at jdk.internal.net.http.PlainHttpConnection.lambda$connectAsync$1(PlainHttpConnection.java:206) ~[java.net.http:?]
        at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        ... 1 more
2025-07-21 13:25:31 INFO  AbstractOperator:401 - Reconciliation #62(timer) Kafka(monitoring/${KAKFA_NAME}): Reconciliation is in progress
2025-07-21 13:25:37 INFO  AbstractWatchManager:447 - Unknown Watch error received 11 times without progress, will reconnect if possible
java.util.concurrent.CompletionException: java.net.ConnectException

keeps on going.

Additional context

Trying to make it work locally on K3d before we go to the process in deploying it to AWS, but due to errors we are not there yet. Sometimes when I delete the pod or do a small change somewhere and do a helm upgrade it might work ?!?!

im-konge · 2025-07-21T13:54:46Z

im-konge
Jul 21, 2025
Maintainer

First thing was the name ${KAKFA_NAME} - I guess it's some kind of typo, but it seems that you have it everywhere else, right?
Also, I see some ZK related values (the env var) that should be IMHO deleted.
Another thing is the namespace you have there which should be watched -> is it a correct one?

Maybe if you could share the Kafka CR without templated stuff, it would be great. IDK if for example ${KAKFA_NAME} does some mess there or not.

11 replies

dj-wasabi Jul 22, 2025
Author

Thank you!

I have deleted the network policies and now it works fine without any issues. Still have to figure out why these policies are blocking needed stuff.

scholzj Jul 22, 2025
Maintainer

So, I do not know exactly what your SDN is etc. But typically, network policies allow by default if no policy exists. But when you specify some network policy matching some Pod, then everything not allowed in it will be disallowed. Strimzi normally uses only Ingress policies, so they define who can talk to the Pods. But not who the Pods can talk to.

So maybe your custom policies - which include Egress as well - block the Egress to the Kubernetes API from the Entity Operator Pod? Just a guess as I never used Egress policies to be honest.

dj-wasabi Jul 22, 2025
Author

Yeah, I am not a fan of the egress policies, but this should be the first project to enable them. But without using the egress it does work. I'll discuss with my team on how to proceed with this. Thank you!

scholzj Jul 22, 2025
Maintainer

So, I guess you will have to make sure that the Egress allows talking to the Kube API as well. This would be needed by the:

Cluster operator
Entity operator
And in case you use rack awareness or node ports also for the Kafka pods

dj-wasabi Jul 22, 2025
Author

Thank you!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

[Bug]: entity-operator keeps failing, both user-operator as topic-operator. #11675

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 11 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Strimzi

[Bug]: entity-operator keeps failing, both user-operator as topic-operator. #11675

Uh oh!

dj-wasabi Jul 21, 2025

Bug Description

Steps to reproduce

Expected behavior

Strimzi version

Kubernetes version

Installation method

Infrastructure

Configuration files and logs

Additional context

Replies: 1 comment · 11 replies

Uh oh!

im-konge Jul 21, 2025 Maintainer

Uh oh!

dj-wasabi Jul 22, 2025 Author

Uh oh!

scholzj Jul 22, 2025 Maintainer

Uh oh!

dj-wasabi Jul 22, 2025 Author

Uh oh!

scholzj Jul 22, 2025 Maintainer

Uh oh!

dj-wasabi Jul 22, 2025 Author

dj-wasabi
Jul 21, 2025

Replies: 1 comment 11 replies

im-konge
Jul 21, 2025
Maintainer

dj-wasabi Jul 22, 2025
Author

scholzj Jul 22, 2025
Maintainer

dj-wasabi Jul 22, 2025
Author

scholzj Jul 22, 2025
Maintainer

dj-wasabi Jul 22, 2025
Author