Strimzi reconcilliation takes ages to complete

[nasabah-old-account]

We have 300+ Kafka clusters managed by Strimzi, but it takes more than 30 min to reconcile a cluster.

As an example, recently we made changes to Kafka object to upgrade the Kafka version, but we saw this log 18 min after applying the changes:

Kafka upgrade from 2.8.0 to 3.0.0

and zookeeper-0 started to be rolled 10 min after the previous event. The other Zookeeper and Kafka instances are currently still waiting to be rolled for the upgrade process.

Any idea how to speed up the process? Thank you.

[scholzj]

I'm not aware of anyone running it at this scale TBH. It is certainly not something we test for. I would normally suggest to make sure you have enough resources, but I guess adding more resources might not be enough. You would probably need to shard the operator. Install more instances of it, label the custom resources and have each instance monitor only some Kafka clusters based on the labels. The STRIMZI_CUSTOM_RESOURCE_SELECTOR environment variable in the Cluster Operator deployment can be used to configure the selector (see more details in the docs).

Why do you run 300+ Kafka clusters? Is that some development environment with one cluster per developer or something like that?