leaders are changing frequently even if the kafka brokers are up - strimzi/operator:0.45.0

[zoheb341]

Hi Team,

Kafka broker pods along with controller pods are up/running. But randomly/frequently leader is changing within seconds. PVC is having storage available

strimzi-cluster-operator:
Container ID: containerd://54fca6260be7e22459ca15fbc0bd93a0f3380fe9b64c628dfafd19bee0b134da
Image: quay.io/strimzi/operator:0.45.0

Containers:
kafka:
Container ID: containerd://1416474020ee3a13a11e266277d0c0075e2d150342278e5ea8f4763a5b0d6703
Image: quay.io/strimzi/kafka:0.45.0-kafka-3.9.0

Strimzi version: 0.45.0
Kafka Broker version: 3.9.0

[im-konge]

Hi, the used images will not say much to us, if you want us to have a look on it, you should start with attaching logs and your CR yamls. Also, is it happening after upgrade or why do you see it just now?

[zoheb341]

team has restarted and the logs are too high. Is there a way, I can search with a keyword in logs. So that the logs length will be less. In events, nothing specific to storage.
KafkaNodePools.zip

[im-konge]

it's hard to say what keywords to pick. The logs are there for debug, otherwise we cannot help you much.

[zoheb341]

KafkaNodePools CR I've attached, to see any clue. I will collect logs as well
KafkaNodePools.zip

[zoheb341]

broker-logs.zip
Attached broker logs, hope we will get some clue out of it. Looks like lot of disconnections

[im-konge]

thanks, could you please add logs from controllers as well?