Confirming Desired Stopped Kafka Connector Behavior

[colechiumento-coreweave]

Overview

TL;DR I noticed a consistent attempt to update Task configs when all KafkaConnectors are stopped. These updates are causing excessive Worker rebalancing. I am unsure if this was a bug or desired behavior.

Troubleshooting

When running a Kafka Connect cluster managed by Strimzi, I noticed an excessive amount of Worker Rebalances accruing in the cluster and causing some CPU heartburn. During troubleshooting, I disabled the only Kafka Connector I had running at the time - the Lenses.io HTTP Sink Connector - to take a look. Specifically, I modified the state of the KafkaConnector like so:

$ KUBE_EDITOR="nano" k edit kafkaconnector/my-http-sink -n my-kafkaconnect-namespace

... yaml below ...
spec:
  autoRestart:
    enabled: true
  class: io.lenses.streamreactor.connect.http.sink.HttpSinkConnector
...
  state: stopped # previously running
  tasksMax: 1

When the rebalancing behavior (and CPU heartburn) persisted, I started to troubleshoot and noticed a few things:

• logs indicate an empty Task config update triggers these rebalances

2025-05-27 23:35:50,034 INFO [Worker clientId=XXX:8083, groupId=mygroup] Tasks [] configs updated (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [KafkaBasedLog Work Thread - XXXconnect.cluster.configs]
2025-05-27 23:35:50,035 INFO [Worker clientId=XXX:8083, groupId=mygroup] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-XXX:8083-1]

• the XXXconnect.cluster.configs topic shows the same message repeated:

$ ./kafka-console-consumer.sh --bootstrap-server $BOOTSTRAP_SERVER --group my-group --topic XXXconnect.cluster.configs --from-beginning --property print.key=true

commit-my-http-sink	{"tasks":0}
target-state-my-http-sink	{"state":"PAUSED","state.v2":"STOPPED"}
commit-my-http-sink	{"tasks":0}
target-state-my-http-sink	{"state":"PAUSED","state.v2":"STOPPED"}
commit-my-http-sink	{"tasks":0}
target-state-my-http-sink	{"state":"PAUSED","state.v2":"STOPPED"}
commit-my-http-sink	{"tasks":0}
target-state-my-http-sink	{"state":"PAUSED","state.v2":"STOPPED"}
commit-my-http-sink	{"tasks":0}
target-state-my-http-sink	{"state":"PAUSED","state.v2":"STOPPED"}

• the XXXconnect.cluster.configs topic configuration supports compaction (as expected):

$ ./kafka-configs.sh --bootstrap-server $BOOTSTRAP_SERVER --entity-type topics --entity-name XXXconnect.cluster.configs --describe

Dynamic configs for topic XXXconnect.cluster.configs are:
  cleanup.policy=compact sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:cleanup.policy=compact, DEFAULT_CONFIG:log.cleanup.policy=delete}

Problem Statement

To me, this looks like one of three things:

• an incorrect configuration at the topic level by me (or some other similar form of user error)
• a reconciliation loop gone awry that is causing excessive rebalancing
• the desired behavior of a reconciliation loop that I (and others) should have a strategy for managing

My goal here would ideally be to get this rebalancing under control, and also to have a better/more complete understanding of what is driving this (specifically what is pushing these Task updates to the topic).

Please let me know if you need any additional information. I didn't think the full YAML dump would be necessary for this issue, but I am happy to provide more configuration information to help debug.

[scholzj]

You should probably elaborate a bit more on what is the problem you are seeing and share the full configurations and logs. I do not think it is really clear what rebalancing are you talking about, what exactly is happening etc.