0.27.1 Topic Operator can't connect to zookeeper

[aamargant]

Hello,

I have deployed a 0.27.1 cluster operator and have 3 kafka brokers along with its 3 zookeeper nodes cluster running with no issues at all.
The topic operator pod waits for the brokers to be ready before it initialize, but when it starts it is not able to connect to the zookeeper-client service, here is the error:

2022-05-13 14:15:26 INFO  ZooKeeper:1619 - Session: 0x0 closed
2022-05-13 14:15:26 DEBUG ZkClient:628 - ignoring event '{None | null}' since shutdown triggered
2022-05-13 14:15:26 DEBUG ZkClient:1282 - Closing ZkClient...done
2022-05-13 14:15:26 DEBUG ZkClient:659 - Leaving process event
2022-05-13 14:15:26 INFO  ClientCnxn:578 - EventThread shut down for session: 0x0
2022-05-13 14:15:27 ERROR Main:55 - Error deploying Session
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server 'kafka-zookeeper-client:2181' with timeout of 18000 ms
	at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1233) ~[com.101tec.zkclient-0.11.jar:?]
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:157) ~[com.101tec.zkclient-0.11.jar:?]
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:131) ~[com.101tec.zkclient-0.11.jar:?]
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98) ~[com.101tec.zkclient-0.11.jar:?]
	at io.strimzi.operator.topic.zk.Zk.createSync(Zk.java:36) ~[io.strimzi.topic-operator-0.27.1.jar:0.27.1]
	at io.strimzi.operator.topic.zk.Zk.lambda$create$0(Zk.java:27) ~[io.strimzi.topic-operator-0.27.1.jar:0.27.1]
	at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at java.lang.Thread.run(Thread.java:829) [?:?]

I have these services running:

kafka-kafka-0 10.224.74.29 9094:32550/TCP                                        
kafka-kafka-1 10.224.249.85   9094:31184/TCP                                           
kafka-kafka-2 10.224.235.6 9094:30430/TCP                                                                 
kafka-kafka-brokers  9090/TCP,9091/TCP,9092/TCP,9093/TCP                       
kafka-kafka-external-bootstrap  10.224.190.175   9094:30114/TCP
kafka-zookeeper-client   10.224.74.148 2181/TCP
kafka-zookeeper-nodes  2181/TCP,2888/TCP,3888/TCP

These deployments:

kafka-entity-operator              1/1     1            1          
strimzi-cluster-operator           1/1     1            1           
strimzi-topic-operator             0/1     1            0          

These replica set

kafka-entity-operator-844c9bbf64              1         1         1     
strimzi-cluster-operator-7fc6fbf995           1         1         1      
strimzi-cluster-operator-867b66dbb5           0         0         0      
strimzi-topic-operator-7665ddcc46             1         1         0      
strimzi-topic-operator-97db859d8              0         0         0      

These statefulset:

kafka-kafka       3/3    
kafka-zookeeper   3/3   

The cluster operator created two policies:

kafka-network-policy-kafka
kafka-network-policy-zookeeper

The latter reads:

Name:         kafka-network-policy-zookeeper
Namespace:    default
Created on:   2022-05-12 14:28:14 +0000 UTC
Labels:    app.kubernetes.io/instance=kafka
              app.kubernetes.io/managed-by=strimzi-cluster-operator
              app.kubernetes.io/name=zookeeper
              app.kubernetes.io/part-of=strimzi-kafka
              strimzi.io/cluster=kafka
              strimzi.io/kind=Kafka
              strimzi.io/name=strimzi
Annotations:  <none>
Spec:
  PodSelector:     strimzi.io/name=kafka-zookeeper
  Allowing ingress traffic:
    To Port: 2888/TCP
    To Port: 3888/TCP
    From:
      PodSelector: strimzi.io/name=kafka-zookeeper
    ----------
    To Port: 2181/TCP
    From:
      PodSelector: strimzi.io/name=kafka-kafka
    From:
      PodSelector: strimzi.io/name=kafka-zookeeper
    From:
      PodSelector: strimzi.io/name=kafka-entity-operator
    From:
      PodSelector: strimzi.io/kind=cluster-operator
    From:
      PodSelector: strimzi.io/name=kafka-cruise-control
  Not affecting egress traffic
  Policy Types: Ingress

Looks like there is no topic operator from port 2181 being shown so I guess I am missing something or done something worng.
This issue is similar to this one #3356 although I have the:

  entityOperator:
    topicOperator: {}
    userOperator: {}

which is what solved the problem to the other issue so I am not sure what I am missing.

[scholzj]

When using Strimzi deployed Kafka cluster, you should use the Topic operator deployed as part of the Kafka cluster (i.e. in the Entity Operator pod). You should not deploy the standalone topic operator.

[aamargant]

but am I not deploying the topic operator pod in the entity operator pod already with these lines

 entityOperator:
    topicOperator: {}
    userOperator: {}

?

[scholzj]

You do. This topicOperator: {} enables the topic operator. If you remove it, it will not be deployed. In any case, you cannot deploy it separately.