Cruise Control Data/Metrics persistence

[prateekkohli21]

Hi,

Just wanted to know where Cruise Control store its data which is needed for rebalancing.

I can see some topics created inside Strimzi kafka which contains cruise control data:
strimzi.cruisecontrol.metrics
strimzi.cruisecontrol.modeltrainingsamples
strimzi.cruisecontrol.partitionmetricsamples

Is there any other data store also present for Cruise Control?
Like does it store any data in Zookeeper/(any database inside Cruise Control) or just the topics created inside Strimzi Kafka?

[scholzj]

It stores the data only in the Kafka topics. It does not use ZooKeeper and does not have any storage on its own.

[ppatierno]

Looking at the code again, the "bad" news is that nothing is going to happen ... and I think it's bad.
The KafkaRebalance will go away because of the deletion (of course) but the rebalancing will continue on the Cruise Control side, missing any task update to report back the user the corresponding status.
At the end, it will try to update the KafkaRebalance status but it won't be there so I would expect an error/exception in the operator log.
I think we should investigate more, try and fix it.
In my opinion, deleting the KafkaRebalance resource should trigger stopping the rebalance execution and not failing when it tries to update the status. To be clear, because rebalancing happens in batches, stopping the rebalancing means that the current rebalancing batch will continue until the end (there is no way to stop it) but the next batches won't happen.
@kyguy @scholzj wdyt?

[scholzj]

In my opinion, deleting the KafkaRebalance resource should trigger stopping the rebalance execution and not failing when it tries to update the status. To be clear, because rebalancing happens in batches, stopping the rebalancing means that the current rebalancing batch will continue until the end (there is no way to stop it) but the next batches won't happen.

I think stopping the rebalance would make sense. But no strong opinions on that. I don't think it is a big issue if it continues 🤷. The issue if you let it continue is I guess that it might keep running for a long time.

But it certainly should not error when the KafkaRebalance is missing.

Will you open an issue for it?

[kyguy]

The KafkaRebalance will go away because of the deletion (of course) but the rebalancing will continue on the Cruise Control side, missing any task update to report back the user the corresponding status.
At the end, it will try to update the KafkaRebalance status but it won't be there so I would expect an error/exception in the operator log.

My understanding of what would happen is the same as Paolo's. If Paolo doesn't beat me to it, I will confirm the behavior locally and open an issue for it

[ppatierno]

Actually I was wrong, on each polling iteration (5 secs by default) to get the current rebalancing task status, the operator tries to retrieve the KafkaRebalance resource to check that it's still in place.
If it's not there anymore, it stops the timer which does the polling (so not retrieving the rebalancing status anymore) and print a log like Rebalance resource was deleted, stopping the request time (but at DEBUG level). The rebalancing continues but we don't get any feedback (unless you look at the Cruise Control log).
I see two things here ...

1. I will open a PR because imho the above log should be at WARN level and clearer about what's going to happen.
2. If we want to stop the current rebalancing batch instead, I will open an issue to track and potentially working on it.

@scholzj @kyguy wdyt?

[scholzj]

Sounds reasonable to me.