[Bug]: KafkaConnect strimzi resource doesn't work in 0.46.0 #11489

dancb10 · 2025-05-29T19:05:40Z

dancb10
May 29, 2025

Bug Description

I have configured the strimzi operator and the crds for version 0.46, deployed Kafka and now trying to deploy KafkaConnect as follows

---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: scylla-to-elasticsearch
  namespace: kafka
  annotations:
  # use-connector-resources configures this KafkaConnect
  # to use KafkaConnector resources to avoid
  # needing to call the Connect REST API directly
    strimzi.io/use-connector-resources: "true"
spec:
  version: 4.0.0
  replicas: 1
  image: localhost:32000/kafka-connect:1.0.3
  bootstrapServers: scylla-kafka-elasticsearch-kafka-bootstrap:9093
  config:
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    # -1 means it will use the default replication factor configured in the broker
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1
    config.providers: file
    config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider
  resources:
    requests:
      cpu: 500m
      memory: 512Mi
    limits:
      cpu: 2
      memory: 4Gi
  logging:
    type: inline
    loggers:
      log4j.rootLogger: "INFO"
      log4j.logger.io.debezium: "DEBUG"
      log4j.logger.io.debezium.connector.base.ChangeEventQueue: "INFO"
      log4j.logger.org.apache.kafka.clients.NetworkClient: "WARN"
      log4j.logger.org.apache.kafka.connect.runtime.rest.RestServer: "WARN"
  template:
    connectContainer:
      env:
        - name: JAVA_TOOL_OPTIONS
          value: "-Dlog4j.configuration=file:/opt/kafka/custom-config/log4j.properties"

But no matter what I've tried, the bootstrap servers is not populated and the container crashes because of missing log4j configuration:
Starting Kafka Connect with configuration:

Bootstrap servers

bootstrap.servers=

REST Listeners

rest.port=8083
rest.advertised.host.name=scylla-to-elasticsearch-connect-0.scylla-to-elasticsearch-connect.kafka.svc
rest.advertised.port=8083

Plugins

plugin.path=/opt/kafka/plugins

Provided configuration

security.protocol=PLAINTEXT
producer.security.protocol=PLAINTEXT
consumer.security.protocol=PLAINTEXT
admin.security.protocol=PLAINTEXT

Additional configuration

consumer.client.rack=

Configuring Java heap: -Xms3221225472 -Xmx3221225472

exec /usr/bin/tini -w -e 143 -- /opt/kafka/bin/connect-distributed.sh /tmp/strimzi-connect.properties
Picked up JAVA_TOOL_OPTIONS: -Dlog4j.configuration=file:/opt/kafka/custom-config/log4j.properties
log4j:ERROR Could not read configuration file from URL [file:/opt/kafka/custom-config/log4j.properties].
java.io.FileNotFoundException: /opt/kafka/custom-config/log4j.properties (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
at java.base/java.io.FileInputStream.(FileInputStream.java:157)
at java.base/java.io.FileInputStream.(FileInputStream.java:111)
at java.base/sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:86)
at java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:189)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:532)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:485)
at org.apache.log4j.LogManager.(LogManager.java:115)
at org.slf4j.impl.Reload4jLoggerFactory.(Reload4jLoggerFactory.java:67)
at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72)
at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45)
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
at org.apache.kafka.connect.cli.AbstractConnectCli.(AbstractConnectCli.java:46)
log4j:ERROR Ignoring configuration file [file:/opt/kafka/custom-config/log4j.properties].
log4j:WARN No appenders could be found for logger (org.apache.kafka.connect.cli.AbstractConnectCli).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Steps to reproduce

No response

Expected behavior

No response

Strimzi version

0.46.0

Kubernetes version

v1.33.1

Installation method

Helm

Infrastructure

Bare-metal

Configuration files and logs

No response

Additional context

No response

scholzj · 2025-05-29T20:18:25Z

scholzj
May 29, 2025
Maintainer

The stuff around the Log4j property makes no sense. You should definitely remove it.

And even more important, you should share how you build the localhost:32000/kafka-connect:1.0.3 image -> make sure that the FROM section of your Dockerfile corresponds to container image corresponding to the Strimzi and Kafka version you are using.

0 replies

dancb10 · 2025-05-30T07:53:12Z

dancb10
May 30, 2025
Author

So i tried and moved past this error, here is my Dockerfile

FROM quay.io/strimzi/kafka:latest-kafka-4.0.0

# Switch to root user temporarily to create and modify directories
USER root:root

# Create a directory for keystore and truststore and adjust permissions
RUN mkdir -p /opt/kafka/security && \
    chown -R 1001:1001 /opt/kafka/security

# Create plugin directories for connectors
RUN mkdir -p /opt/kafka/plugins/elasticsearch
COPY ./connector-elasticsearch/ /opt/kafka/plugins/elasticsearch/
RUN mkdir -p /opt/kafka/plugins/scylladb
COPY ./connector-scylladb/ /opt/kafka/plugins/scylladb/

# Set permissions for non-root user
RUN chown -R 1001:1001 /opt/kafka/plugins

# Switch back to non-root user
USER 1001

Then I created a KafkaUser

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: scylla-kafka-elasticsearch
  namespace: kafka
  labels:
    strimzi.io/cluster: scylla-kafka-elasticsearch
spec:
  authentication:
    type: tls

But then when i deploy the KafkaConnect

---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: scylla-kafka-elasticsearch
  namespace: kafka
  annotations:
  # use-connector-resources configures this KafkaConnect
  # to use KafkaConnector resources to avoid
  # needing to call the Connect REST API directly
    strimzi.io/use-connector-resources: "true"
spec:
  version: 4.0.0
  replicas: 1
  image: localhost:32000/kafka-connect:1.0.4
  bootstrapServers: scylla-kafka-elasticsearch-kafka-bootstrap:9093
  tls:
    trustedCertificates:
      - secretName: scylla-kafka-elasticsearch
        certificate: ca.crt
  config:
    bootstrap.servers: scylla-kafka-elasticsearch-kafka-bootstrap:9093
    # SSL config
    security.protocol: SSL
    ssl.truststore.type: PEM
    ssl.truststore.certificates: /mnt/ssl/ca.crt
    ssl.keystore.certificate.chain: /mnt/ssl/user.crt
    ssl.keystore.key: /mnt/ssl/user.key
    ssl.keystore.key.password: kG4inNj5T1G4OQohSZtPXSoRoqL2IFev

    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    # -1 means it will use the default replication factor configured in the broker
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1
    
    config.providers: file
    config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider
  resources:
    requests:
      cpu: 500m
      memory: 512Mi
    limits:
      cpu: 2
      memory: 4Gi
  template:
    pod:
      volumes:
        - name: ssl-cert-volume
          secret:
            secretName: scylla-kafka-elasticsearch
    connectContainer:
      volumeMounts:
        - name: ssl-cert-volume
          mountPath: /mnt/ssl
          readOnly: true
      env:
      - name: KAFKA_LOG4J_LOGGERS
        value: "org.apache.kafka.common.network.SslTransportLayer=DEBUG, org.apache.kafka.common.security.ssl.SslFactory=DEBUG"

But whatever I try seems like i keep getting

Caused by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:383) ~[?:?]
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326) ~[?:?]
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321) ~[?:?]
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1351) ~[?:?]
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1226) ~[?:?]
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1169) ~[?:?]
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) ~[?:?]
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) ~[?:?]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) ~[?:?]
	at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:444) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:533) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:382) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:302) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:548) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.network.Selector.poll(Selector.java:486) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:645) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1560) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1491) ~[kafka-clients-4.0.0.jar:?]
	at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) ~[?:?]
	at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
	at java.base/sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1329) ~[?:?]
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1226) ~[?:?]
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1169) ~[?:?]
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) ~[?:?]
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) ~[?:?]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) ~[?:?]
	at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:444) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:533) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:382) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:302) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:548) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.network.Selector.poll(Selector.java:486) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:645) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1560) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1491) ~[kafka-clients-4.0.0.jar:?]
	at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148) ~[?:?]
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129) ~[?:?]
	at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) ~[?:?]
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ~[?:?]
	at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
	at java.base/sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1329) ~[?:?]
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1226) ~[?:?]
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1169) ~[?:?]
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) ~[?:?]
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) ~[?:?]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) ~[?:?]
	at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:444) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:533) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:382) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:302) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:548) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.network.Selector.poll(Selector.java:486) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:645) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1560) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1491) ~[kafka-clients-4.0.0.jar:?]
	at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]

0 replies

dancb10 · 2025-05-30T07:55:06Z

dancb10
May 30, 2025
Author

I haven't seen how should we use these KafkaUsers to allow KafkaConnect to connect to the Kafka cluster.
Btw this is how i deploy the Kafka cluster:

---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: scylla-kafka-elasticsearch
  namespace: kafka
  labels:
    strimzi.io/cluster: scylla-kafka-elasticsearch
spec:
  replicas: 3
  roles:
    - broker
    - controller
  resources:
    requests:
      memory: "2Gi"
      cpu: "1"
    limits:
      memory: "4Gi"
      cpu: "2"
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        class: local-path
        kraftMetadata: shared
        deleteClaim: false

---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: scylla-kafka-elasticsearch
  namespace: kafka
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 4.0.0
    metadataVersion: 4.0-IV3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
  entityOperator:
    topicOperator: {}
    userOperator: {}

1 reply

scholzj May 30, 2025
Maintainer

Well, as I said ... this is wrong:

FROM quay.io/strimzi/kafka:latest-kafka-4.0.0

You need to use the image corresponding to your Strimzi version and ot latest. So assuing you use Stirmzi 0.46.0, it should be:

FROM quay.io/strimzi/kafka:0.46.0-kafka-4.0.0

As it stands in the documentation.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

[Bug]: KafkaConnect strimzi resource doesn't work in 0.46.0 #11489

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 3 comments 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Strimzi

[Bug]: KafkaConnect strimzi resource doesn't work in 0.46.0 #11489

Uh oh!

Uh oh!

dancb10 May 29, 2025

Bug Description

Bootstrap servers

REST Listeners

Plugins

Provided configuration

Additional configuration

Steps to reproduce

Expected behavior

Strimzi version

Kubernetes version

Installation method

Infrastructure

Configuration files and logs

Additional context

Replies: 3 comments · 1 reply

Uh oh!

Uh oh!

scholzj May 29, 2025 Maintainer

Uh oh!

dancb10 May 30, 2025 Author

Uh oh!

dancb10 May 30, 2025 Author

Uh oh!

scholzj May 30, 2025 Maintainer

dancb10
May 29, 2025

Replies: 3 comments 1 reply

scholzj
May 29, 2025
Maintainer

dancb10
May 30, 2025
Author

dancb10
May 30, 2025
Author

scholzj May 30, 2025
Maintainer