KafkaConnect - Introduce annotation to ignore KafkaConnector state

[swapsCAPS]

Hi all!

Would it be possible to start, stop and pause Connectors by using the Connect API while still using strimzi.io/use-connector-resources: true?
The operator will currently set whatever state is configured on the KafkaConnector, so if a user uses the Connect API, or a tool like AKHQ to change the Connector state it will quickly be reset to its original state in the spec.

Could a new annotation be introduced like strimzi.io/sync-connector-resources-state: false (which would default to true) to tell the operator to ignore whatever state the KafkaConnector has and just sync the rest of the spec. I could also imagine some sort of enum like strimzi.io/use-connector-resources: true | "full" | "spec_only" | "etc."

I can see how this would be potentially confusing, where an operator is managing some part of a resource, but isn't managing another... However, it would make it easier to pause and resume Connectors for folks who don't have access to the k8s cluster.

Would love to hear your thoughts on this

[scholzj]

I do not think there are plans for anything like that.

[swapsCAPS]

Regardless of if there are plans at the moment. Would it be a desirable feature and good practice architecturally? Wouldn't mind investing some time on this, but of course would like to know if something like this would even be accepted.
Was not sure if I should use this Discussions - Ideas section, or immediately create an [Enhancement] issue in strimzi-kafka-operator.

[scholzj]

I guess it depends on what kind of answer you want. In the discussions, you will get informal answers from different people. My answer wasn't exactly direct, sorry for that. But it can also be written as I personally do not think this is an interesting idea.

If you open an enhancement, it will go through a more formal triage. It might be rejected, accepted, or request a proposal. For a feature like this, I would not expect it to be accepted right away, as this would be a significant change. So it would be either rejected or request a proposal. The proposal is a more formal process than the triage. So assuming it is not rejected, you will need to write a proposal to describe the benefits, use cases, but also the technical details of how will it be implemented. And the proposal will be voten upon to decide if it will be accepted or not.

[scholzj]

(It is, of course, up to you on how you want to proceed. You can also simply open the proposal right away. But the triage might give you an indication of the chances of having the proposal accepted.)

[scholzj]

Also to clarify why I do not think this is a great idea -> it adds a lot of complexity and confusion to how the connector operator works. The complexity also means a lot of effort in the long term. Also with tools such as AKHQ, any change you do will be reset by Strimzi. Whether it pausing a connector, changing its settings, recnfiguring a Kafka broker. So this would create a very different pattern of how Strimzi works.

[swapsCAPS]

Thanks @scholzj. I think we share some of the same worries. From a high level, having two systems managing the same resource seems like an error prone process. And would increase the amount of future edge cases to handle. That's also exactly why I was looking for some feedback on the idea. I am not an expert on k8s operators and was unsure if this is common practice.
Thanks for the clarification on the Strimzi proposal process and for the informal triage process 🙏🏻 😁

Cheers!