Upgrading Kafka from 3.8.0 to 3.9.0 fails during creation of custom cert keystore in broker

[avinash-platformatory]

I'm upgrading Kafka from 3.8.0 to 3.9.0 and noticed that the broker fails to create the keystore for the custom listener with custom certs.From the pod description, the volume is mounted from the secret tls-kafka-external and the secret has the key tls.key. It is an unencrypted private key that worked with the previous version. The controller pods were upgraded fine but the broker pods fail with the following log statements -

Creating keystore /tmp/kafka/custom-external-9094.keystore.p12
Could not find private key from -inkey file from /opt/kafka/certificates/custom-external-9094-certs/tls.key

The external listener is configured as follows -

     listeners:
     - name: external
        port: 9094
        type: ingress
        tls: true
        configuration:
          brokerCertChainAndKey:
            secretName: tls-kafka-external
            certificate: tls.crt
            key: tls.key
        authentication: # Omitted for brevity 

The cluster was working with version 3.8.0 but fails when upgraded to 3.9.0. The cluster works if started with 3.9.0 with the exact same configuration/certs but not during upgrade.

The cluster uses a custom image that adds tiered storage support. The base image version is set appropriately. The Dockerfile for reference -

FROM quay.io/strimzi/kafka:latest-kafka-3.9.0

COPY build/distributions/tiered-storage-for-apache-kafka-*.tgz /opt/tiered-storage/

COPY storage/s3/build/distributions/s3-*.tgz /opt/tiered-storage/

USER root

RUN mkdir -p /opt/kafka/libs/tiered-storage-s3/core && \
    mkdir -p /opt/kafka/libs/tiered-storage-s3/s3 && \
    tar xf /opt/tiered-storage/tiered-storage-for-apache-kafka-*.tgz --strip-components=1 -C /opt/kafka/libs/tiered-storage-s3/core/ && \
    tar xf /opt/tiered-storage/s3-*.tgz --strip-components=1 -C /opt/kafka/libs/tiered-storage-s3/s3/ && \
    rm /opt/tiered-storage/tiered-storage-for-apache-kafka-*.tgz && \
    rm /opt/tiered-storage/s3-*.tgz

USER 1001

RUN mkdir /home/kafka/kafka-tiered-storage-cache

Strimzi version - 0.45.0

[scholzj]

This is wrong:

FROM quay.io/strimzi/kafka:latest-kafka-3.9.0

You have to use image for the Kafka version and Strimzi version you use. Not latest.

[avinash-platformatory]

Thanks for pointing that out.
The issue persists even after updating the image to use the appropriate strimzi version (quay.io/strimzi/kafka:0.45.0-kafka-3.9.0). :/
A fresh installation of the cluster with the 3.9.0 image works with the same operator and configuration, including the certs but fails during upgrade.

[scholzj]

Are you sure you are using the updated image? Does the same certificate work for you with the standard Strimzi image? (i.e. without the tiered storage and custom container image)

[avinash-platformatory]

Sorry, not a Strimzi issue. The certificates were wrongly configured during update. I was automating through Python that broke the certificates during upgrades.

[scholzj]

Thanks for the update. I'm glad you managed to figure it out.