KafkaConnector Auto Restart Back-off - Disable or Reset

[treyhendon]

First, thank you for such a great operator!

We use Debezium based Connectors to pass data from multiple Postgresql and MongoDB databases between hundreds of services and our data warehousing project.

Currently, we have a custom Kubernetes Cron Job that runs every 5 min to add the restart connector annotation to any failed KafkaConnector CRs. We'd love to move entirely to the newer autoRestart setting (with no limit), however, the back-off algorithm concerns me a little.

Would it be possible to either:

1. Add a disableBackoff option (per KafkaConnector)? or
2. Reset the back-off logic after x-min have passed?
   • i.e. After 1440 min, consider the restart count 0.

The reason I ask is that 95% of the time our restarts are due to network interruptions (the other 5% are single messages larger than we allow in Kafka). We see around two network interruptions a week (running in AKS). If the restart count is continually applied to the back-off logic, we'd be up to the 1 hour wait in only a few weeks.

Maybe something like this already exists, but I've not found it in the documentation or seen it while reviewing the code. Thoughts?

[scholzj]

The back-off is not accumulating forever. It is reset once the connector runs after the current backoff time. So if you have a network outage on Monday for several hours, it attempts to restart the connector several times until it eventually succeeds. And once it is running after e.g. one hour (the exact time depends on how many times it took to get the connector running), the counter resets. So when you have another network outage on Thursday, it starts again from 0. You should see it in the .status section of the custom resource.

If it works differently for you, then it would be a bug.

[treyhendon]

Great to know! We'll begin testing in our lower environments and monitoring the status of each CR. If we see something wrong, I'll report it. Thanks!