Auto-restart kafka connect pod if ConfigMap or Secrets (external configuration) have changed.

[ness-safinapharas]

Hi,

We use the Strimzi Operator to build a Kafka Cluster and Kafka Connect as a CRD. We are utilizing a Camel HTTP sink connector. We have a certificate key that needs to be passed to the sink connector via JVM arguments. The certificates are stored in Vault and mounted to a specific path, and they are rotated every 7 days. We are using a Kubernetes Operator for ConfigMap and Secrets, which allows the certificates to refresh at the mounted path. However, the pod using the certificates is not restarting automatically. Is there a way to restart a KafkaConnect (CRD) pod when Secrets or ConfigMap are changed?

We tried a reloader library, but this kind of library works with Deployments. Here we are using Custom Resources for KafkaConnect. Also is there any way to use the side car in KafkaConnect CRD resource? This is a very urgent issue. Can someone help?

[scholzj]

Strimzi does not support sidecars nor any reloading of the external secrets or config maps. Sorry.

[ness-safinapharas]

What is the suggested approach to solve this problem?

[scholzj]

I don't know. I'm not aware of anyone raising a problem like this before. You would need to use some external mechanism I guess? I'm not sure what exactly are you doing and why. Configuring TLS certificates through JVM does not seem to be very secure but I do not think Kafka Connect has any ways to automatically reload the connector configuration either so it probably does not matter.

[ness-safinapharas]

We are using the Camel open-source HTTP Sink Connector (latest version 4.4.2), where the mTLS keystore information can only be configured in Kafka Connect JVM arguments. The remote endpoint's authorization type is mTLS. Therefore, once the certificate is rotated, we need to restart the pod to refresh the JVM arguments.

Additionally, in the case of ConfigMap changes, the mounted file reflects the latest changes. However, the pod does not automatically restart, causing the running pod to retain the older values.

[scholzj]

We are using the Camel open-source HTTP Sink Connector (latest version 4.4.2), where the mTLS keystore information can only be configured in Kafka Connect JVM arguments. The remote endpoint's authorization type is mTLS. Therefore, once the certificate is rotated, we need to restart the pod to refresh the JVM arguments.

That makes it automatically used with all other connectors and the Connect cluster itself. So that is why this is not optimal. It is not like Connect would reload the certificate if done any other way. But it stands in contrast with the relatively often certificate rotation.

Additionally, in the case of ConfigMap changes, the mounted file reflects the latest changes. However, the pod does not automatically restart, causing the running pod to retain the older values.

This is your use-case. But there are be many other use-cases where you do not want to restart everything just because something changed in some ConfigMap or Secret. So Strimzi has no way to know what it is that is being mounted and what the right thing to do about it is. So as I said, you would need to trigger it from the outside.

[ness-safinapharas]

Thank you for your quick responses.

[Navirash]

Hi @ness-safinapharas @scholzj

I'm currently facing a similar issue with a RabbitMQ Kafka connector. In my case, certificates are managed by cert-manager, and they are mounted into the KafkaConnect pod via Secrets. When cert-manager renews the certificates, the Secret is updated, but the KafkaConnect pod doesn't automatically restart or reload the new certificate.

You mentioned a similar setup — i juste want to know how you finally handled the automatic restart of the KafkaConnect pod after certificate renewal ? Did you use a sidecar, an operator, or some custom logic to trigger the restart when the mounted cert changes ?

Any details about your solution would be very helpful

Thanks a lot