Kafka Connect upgrade path to 0.46.0

[seanshi007]

Hi, from my understanding since this PR https://github.com/strimzi/strimzi-kafka-operator/pull/11062/files gets rolled out as part of 0.46.0, kafka connect images built from previous strimzi versions is not able to continue to run under 0.46.0, what's suggested upgrade strategy?

In our production, we are using strimzi 0.43.0 and kafka connect images are built using strimzi 0.43.0 as well. We are trying to upgrade strimzi to 0.46.0, which has features that we wanted (dnsConfig). However, our images are not able to run when strimzi upgrades because how the connect worker configuration files are generated is completely changed and seems not backward compatible. We need to have minimum downtime as well, i.e. we simply can't just upgrade strimzi and then upgrade all images, that downtime is not acceptable for our production scale.

[ppatierno]

AFAIU you are using a custom Kafka Connect image (based on Strimzi Kafka image from 0.43.0) for adding connectors/plugins.
I guess you are setting the custom image in the operator Deployment (to the corresponding env var) or within the KafkaConnect custom resource. Either way, I would assume the following should work for you:

• upgrade the operator to 0.46.0 --> because you have Kafka Connect image set (in any of the above ways), the operator should not take care of them or rolling them. So the Kafka Connect pods based on old 0.43.0 based images are still running.
• update the Kafka Connect image to a new one based on the 0.46.0 image --> the operator would start reconciling the Kafka Connect pods and restarting one by one with the new image.

I would try this in a test env first.

[scholzj]

Assuming you configured the container image in te KafkaConnect CR, I think the easiest thing to do is to:

• Pause the reconciliation of your Connect clusters while still on Strimzi 0.45
• Upgrade Strimzi to 0.46 while the Connect clusters are paused
• Update the image in the KafkaConnect CR to set the new container image
• Unpause the reconciliation

While the reconciliation is paused, the Connect will continue to work as before, but will not be operated. So you should try to minimize the time while the reconciliation is paused. But in general, it should not cause any issues unless you keep it paused for examle for days.

[seanshi007]

Thanks for the suggestions. I think we found a way that may be the best to suits our upgrade strategy (non-invasive, not hurry). The key reason old images aren't working with new operator is that the new operator removes a few environment variables that were used to generated by the old operator. In the connect cluster yaml, we can always add those environment variables so that the old image can work with both old and new operator, which allows us to upgrade the operator only and then upgrade the images whenever we are comfortable with.

For example, KAFKA_CONNECT_CONFIGURATION is removed in the new operator, but if we set the connect cluster yaml to something like below, then old image works with new operator.

template:
connectContainer:
env:
- name: KAFKA_CONNECT_CONFIGURATION
value: |
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter

[scholzj]

That is absolute hack which might have many bad consequences. The solution I suggested is much more safe and predictable.

…

On Wed, May 7, 2025, 01:18 Sean Shi ***@***.***> wrote: Thanks for the suggestions. I think we found a way that may be the best to suits our upgrade strategy (non-invasive, not hurry). The key reason old images aren't working with new operator is that the new operator removes a few environment variables that were used to generated by the old operator. In the connect cluster yaml, we can always add those environment variables so that the old image can work with both old and new operator, which allows us to upgrade the operator only and then upgrade the images whenever we are comfortable with. For example, KAFKA_CONNECT_CONFIGURATION is removed in the new operator, but if we set the connect cluster yaml to something like below, then old image works with new operator. template: connectContainer: env: - name: KAFKA_CONNECT_CONFIGURATION value: | key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter — Reply to this email directly, view it on GitHub <#11412 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABLFOR6KMAIFDCT3VATWSPT25FGM5AVCNFSM6AAAAAB4RB6S7SVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMBVGY2DEMI> . You are receiving this because you commented.Message ID: <strimzi/strimzi-kafka-operator/repo-discussions/11412/comments/13056421@ github.com>

[seanshi007]

The solution you suggested can't allow us to upgrade the images slowly, for example, currently in our product, we release new image the first week (so any new customers coming in will use the new image) and then upgrade existing pods the week after. We simply can't pause the reconciliation for many days with your solution.

[ppatierno]

I agree with Jakub. The env var is not there to be used in that way, so you are doing it in a unconventional not supported way.

[scholzj]

The solution you suggested can't allow us to upgrade the images slowly, for example, currently in our product, we release new image the first week (so any new customers coming in will use the new image) and then upgrade existing pods the week after. We simply can't pause the reconciliation for many days with your solution.

Well, the hack you want to use does not guarantee you anything. Will the operator actually operate such operand? Will something amke it go crazy and start rolling it in the loop? Who knows. I also don't understand why you need more time. You can have the new image prepared upfront. You do not need to start with it only after the upgrade. This way, you are taking a huuuge risk.