The zk mode kafka cluster deployed by strimzi operator encounters election problems between zk in production

[lyd100822]

k8sversion

1.20 operator 0.27.1 kafka 3.0.0

[see-quick]

Hi @lyd100822,

You are using an ancient version of the operator, which is no longer supported. I am sorry. Is there a possibility that you can upgrade to the supported versions 0.45.x with ZK support?

Moreover, it's very hard when you don't share full logs and just a screen to troubleshoot the issue. So please, if it's possible, upgrade to 0.45.0 and then if the issue persists, share full logs (i.e., cluster operator, kafka....)

Thanks.

[lyd100822]

Can you provide the following troubleshooting ideas at present? The production environment must be triggered. Thank you very much

[im-konge]

This is the log of Zookeeper startup failure

But from which startup? After the upgrade or still on the Kafka 3.0.0? From the logs it's still Kafka 3.0.0.

Can you provide the following troubleshooting ideas at present?

It's hard to remember that old version and what issues were there. In the last versions of Kafka we experienced issues with ZooKeeper that wasn't able to form a quorum. Which seems to be the same issue here as well. But that's just guessing, as I mentioned already, nobody remembers what complications were in version couple of years old and unsupported.
Also, wasn't there some change regarding Kubernetes cluster, for example some update that broke it? It can be networking issue.

It works fine when a single node is replaced.

What does this mean? If you have production env with single ZK node, then it's wrong from that point on. You should have more than just one ZK pod and Kafka broker. Single-node setup is only for testing purposes.

[lyd100822]

The production environment Zookeeper 3 nodes will report an error when started. The log is what I sent you. Try to replace a single node and it will start normally.

[lyd100822]

The original Kafka version was 3.0.0. By default, the production restarted Zookeeper suddenly and then it could not start.

[im-konge]

Try to replace a single node and it will start normally.

So you replaced (how?) one of the nodes (randomly?) and it started normally?

By default, the production restarted Zookeeper suddenly and then it could not start.

There should be some reason why it was restarted. Otherwise it wouldn't be restarted at all.
Or was it restarted because of some certificate renewal or something?

Anyway, as I mentioned before, you can check older issues, threads or posts if someone from the community experienced the same at that version. This (short) log shows FMPOV that the quorum wasn't form in some way, it can be networking issue, it can be something else. You would need to provide full logs and configuration of Kafka, but I'm afraid that nobody will know what was in version 0.27.1, when now is 0.46.0 out. We support just the latest one + currently 0.45.x as it's last version with ZK support.