KafkaConnect in CrashloopBackOff with Custom image and operator 0.47.0 Kafka 3.9.0 error FileNotFoundException log4j

[totochichi]

Hello all, I am trying to install Kafka connect with Strimzi operator 0.47.0 and Kafka 3.9.0, using custom plugin, but my Connect pods are stuck in CrashLoopBackOff with the error message log4j:ERROR Could not read configuration file from URL [file:/opt/kafka/custom-config/log4j.properties].
I am not aware of any changes from Kafka 3.8.0 to 3.9.0 that could have altered how log4j config is read. I ensured strimzi Operator is consisted with the base image being used in either scenario.
Has anyone seen something similar to this?

The image with Connect version 0.45.0-kafka-3.8.0 worked well on Strimzi 0.45.0, but I am trying to check backward compatibility if I was to upgrade to v47 and use K8s ImageVolume in the future.

Below is the DockerImage template I'm using, with strimzi Connect version updated based on the current Operator.

ARG CONFLUENT_CLI_VERSION=3.64.3
ARG STRIMZI_CONNECT_VERSION=0.47.0-kafka-3.9.0

FROM <PRIVATE_REGISTRY>/confluentinc/confluent-cli:$CONFLUENT_CLI_VERSION as builder
ENV http_proxy="http://<PROXY_URL>:8000" \
      https_proxy="http://<PROXY_URL>:8000" \
      no_proxy="localhost,172.17.0.1,172.20.0.1,<INTERNAL_DOMAINS>,127.0.0.1,10.43.0.1"

ARG plugins

RUN mkdir -p /tmp/plugins
WORKDIR /tmp/plugins

RUN for plugin in $plugins; do \
    echo "$plugin"; \
    confluent connect plugin install $plugin --plugin-directory /tmp/plugins --force || true; \
    done;

RUN ls -l /tmp/plugins

FROM <PRIVATE_REGISTRY>/strimzi/kafka:$STRIMZI_CONNECT_VERSION

USER root:root
COPY --from=builder /tmp/plugins/ /opt/kafka/plugins/
USER 1001

Here is the KafkaConnect pods logs. Not sure why it cannot see the log4j.properties as I was able to cat it when exec-ed into the kafka broker container.

# Bootstrap servers
##########
bootstrap.servers=<KAFKA_BOOTSTRAP>:9093

##########
# Config providers
##########
config.providers=file,strimzienv,strimzifile,strimzidir,strimzisecrets
config.providers.strimzienv.class=org.apache.kafka.common.config.provider.EnvVarConfigProvider
config.providers.strimzienv.param.allowlist.pattern=.*
config.providers.strimzifile.class=org.apache.kafka.common.config.provider.FileConfigProvider
config.providers.strimzidir.class=org.apache.kafka.common.config.provider.DirectoryConfigProvider
config.providers.strimzidir.param.allowed.paths=/opt/kafka
config.providers.strimzisecrets.class=io.strimzi.kafka.KubernetesSecretConfigProvider

##########
# Provided configurations
##########
offset.storage.topic=<CONNECT_OFFSETS_TOPIC>
value.converter=org.apache.kafka.connect.json.JsonConverter
config.storage.topic=<CONNECT_CONFIGS_TOPIC>
key.converter=org.apache.kafka.connect.json.JsonConverter
group.id=<CONNECT_GROUP_ID>
status.storage.topic=<CONNECT_STATUS_TOPIC>
config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider
config.storage.replication.factor=3
connector.client.config.override.policy=All
key.converter.schemas.enable=false
offset.flush.interval.ms=30000
offset.flush.timeout.ms=6000
offset.storage.replication.factor=3
status.storage.replication.factor=3
value.converter.schemas.enable=false

##########
# REST Listeners
##########
rest.advertised.host.name=${strimzienv:ADVERTISED_HOSTNAME}
rest.advertised.port=8083

##########
# Plugins
##########
plugin.path=/opt/kafka/plugins

##########
# TLS / SSL
##########
ssl.truststore.certificates=${strimzisecrets:<NAMESPACE>/<TRUSTED_CERTS>:*.crt}
ssl.truststore.type=PEM
producer.ssl.truststore.certificates=${strimzisecrets:<NAMESPACE>/<TRUSTED_CERTS>:*.crt}
producer.ssl.truststore.type=PEM
consumer.ssl.truststore.certificates=${strimzisecrets:<NAMESPACE>/<TRUSTED_CERTS>:*.crt}
consumer.ssl.truststore.type=PEM
admin.ssl.truststore.certificates=${strimzisecrets:<NAMESPACE>/<TRUSTED_CERTS>:*.crt}
admin.ssl.truststore.type=PEM

##########
# Authentication configuration
##########
ssl.keystore.certificate.chain=${strimzisecrets:<NAMESPACE>/<KEYSTORE>:tls.crt}
ssl.keystore.key=${strimzisecrets:<NAMESPACE>/<KEYSTORE>:tls.key}
ssl.keystore.type=PEM
producer.ssl.keystore.certificate.chain=${strimzisecrets:<NAMESPACE>/<KEYSTORE>:tls.crt}
producer.ssl.keystore.key=${strimzisecrets:<NAMESPACE>/<KEYSTORE>:tls.key}
producer.ssl.keystore.type=PEM
consumer.ssl.keystore.certificate.chain=${strimzisecrets:<NAMESPACE>/<KEYSTORE>:tls.crt}
consumer.ssl.keystore.key=${strimzisecrets:<NAMESPACE>/<KEYSTORE>:tls.key}
consumer.ssl.keystore.type=PEM
admin.ssl.keystore.certificate.chain=${strimzisecrets:<NAMESPACE>/<KEYSTORE>:tls.crt}
admin.ssl.keystore.key=${strimzisecrets:<NAMESPACE>/<KEYSTORE>:tls.key}
admin.ssl.keystore.type=PEM

##########
# Additional information
##########
consumer.client.rack=${strimzidir:/opt/kafka/init:rack.id}

##########
# Kafka Security protocol
##########
security.protocol=SSL
producer.security.protocol=SSL
consumer.security.protocol=SSL
admin.security.protocol=SSL
...

+ exec /usr/bin/tini -w -e 143 -- /opt/kafka/bin/connect-distributed.sh /tmp/strimzi-connect.properties
2025-07-28 12:52:44.884 | main | INFO | io.prometheus.jmx.JavaAgent | Starting ...
2025-07-28 12:52:45.220 | main | INFO | io.prometheus.jmx.JavaAgent | HTTP enabled [true]
2025-07-28 12:52:45.220 | main | INFO | io.prometheus.jmx.JavaAgent | HTTP host:port [0.0.0.0:9404]
2025-07-28 12:52:45.221 | main | INFO | io.prometheus.jmx.JavaAgent | Starting HTTPServer ...
2025-07-28 12:52:45.269 | main | INFO | io.prometheus.jmx.JavaAgent | HTTPServer started
2025-07-28 12:52:45.270 | main | INFO | io.prometheus.jmx.JavaAgent | OpenTelemetry enabled [false]
2025-07-28 12:52:45.270 | main | INFO | io.prometheus.jmx.JavaAgent | Running ...
log4j:ERROR Could not read configuration file from URL [file:/opt/kafka/custom-config/log4j.properties].
java.io.FileNotFoundException: /opt/kafka/custom-config/log4j.properties (No such file or directory)
    at java.base/java.io.FileInputStream.open0(Native Method)
    ...
log4j:ERROR Ignoring configuration file [file:/opt/kafka/custom-config/log4j.properties].
log4j:WARN No appenders could be found for logger (org.apache.kafka.connect.cli.AbstractConnectCli).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
...

[im-konge]

Hey, you are using this custom image with Strimzi 0.45.0?

[totochichi]

connect_config.txt
PFA our connect config.
I checked our config and we only specify image, and since the image is created with Strimzi 0.47.0 and kafka 3.9.0, wouldn't it ensure the proper Kafka version being used?
From the documentation it said Strimzi Operator will default to the Cluster Operator Kafka version, do you know which Kafka version 0.47.0 defaults to? I presume Kafka 4.0 otherwise it would not have complained. But for my curiosity would be good to know.

[im-konge]

It is 4.0.0 yes for Strimzi 0.47.0 (as the latest supported Kafka version)

[im-konge]

So could you try to specify the 3.9.0 as version in the KafkaConnect?
I will discuss with other maintainers that we should add some note or something when we are still supporting 3.9.x together with 4.0.x that you should specify the version in the spec for KafkaConnect (as for other things like Kafka the images specified in the Deployment file of the operator are used).

[totochichi]

Thanks for confirming, yes I updated the CR in place and it does not complain about missing log4j properties anymore.

[im-konge]

Great :)