KafkaConnect fails to start after upgrade from 0.45.0 to 0.46.0 - Missing required configuration "key.converter"

[artempol-dev]

KafkaConnect fails to start after upgrade from 0.45.0 to 0.46.0 - Missing required configuration "key.converter"

Bug Report

Strimzi version: 0.46.0 (upgraded from 0.45.0)
Kafka version: 3.9.0

Description

After upgrading Strimzi from version 0.45.0 to 0.46.0, KafkaConnect fails to start with the following error:

org.apache.kafka.common.config.ConfigException: Missing required configuration "key.converter" which has no default value.

The configuration that was working in 0.45.0 now fails in 0.46.0, suggesting a breaking change in default configurations or validation logic.

Error Log

2025-06-16 13:07:48,444 ERROR Stopping due to error (org.apache.kafka.connect.cli.AbstractConnectCli) [main]
org.apache.kafka.common.config.ConfigException: Missing required configuration "key.converter" which has no default value.
	at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:534)
	at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:524)
	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)
	at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:439)
	at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:593)
	at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:587)
	at org.apache.kafka.connect.cli.ConnectDistributed.createConfig(ConnectDistributed.java:107)
	at org.apache.kafka.connect.cli.ConnectDistributed.createConfig(ConnectDistributed.java:54)
	at org.apache.kafka.connect.cli.AbstractConnectCli.startConnect(AbstractConnectCli.java:123)
	at org.apache.kafka.connect.cli.AbstractConnectCli.run(AbstractConnectCli.java:95)
	at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:112)

My configuration

As you can see I have key converter property in my kafka connect configmap

    ##########

    # Provided configurations

    ##########

    offset.storage.topic=kafka-connector-cluster-test-offsets

    value.converter=org.apache.kafka.connect.json.JsonConverter

    config.storage.topic=kafka-connector-cluster-test-configs

    key.converter=org.apache.kafka.connect.json.JsonConverter

[scholzj]

You should start by sharing the full configuration (custom resources) and full logs.

[artempol-dev]

@scholzj
Hey! Here it is

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  annotations:
    meta.helm.sh/release-name: cdc
    meta.helm.sh/release-namespace: kafka2
    strimzi.io/use-connector-resources: 'true'
  creationTimestamp: '2025-06-06T15:32:37Z'
  generation: 1
  labels:
    app.kubernetes.io/component: kafka-connector-cluster
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kafka-connector
    app.kubernetes.io/version: 0.1.0
    helm.sh/chart: kafka-connector-0.1.0
  name: kafka-connector-cluster-cdc
  namespace: kafka2
  resourceVersion: '1750078091095023017'
  uid: 928d4ef3-3bfc-4cf0-a1f9-05b8d5b0a2ba
  selfLink: >-
    /apis/kafka.strimzi.io/v1beta2/namespaces/kafka2/kafkaconnects/kafka-connector-cluster-cdc
status:
  conditions:
    - lastTransitionTime: '2025-06-16T12:48:11.058928040Z'
      message: >-
        Exceeded timeout of 300000ms while waiting for Pods resource
        kafka-connector-cluster-cdc-connect-0 in namespace kafka2 to be ready
      reason: TimeoutException
      status: 'True'
      type: NotReady
  labelSelector: >-
    strimzi.io/cluster=kafka-connector-cluster-cdc,strimzi.io/name=kafka-connector-cluster-cdc-connect,strimzi.io/kind=KafkaConnect
  observedGeneration: 1
  replicas: 1
  url: http://kafka-connector-cluster-cdc-connect-api.kafka2.svc:8083
spec:
  bootstrapServers: kafka-cluster-cdc-kafka-bootstrap:9092
  config:
    cleanup.policy: compact
    config.providers: secrets
    config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider
    config.storage.replication.factor: -1
    config.storage.topic: kafka-connector-cluster-test-configs
    group.id: kafka-connector-cluster-test
    offset.storage.replication.factor: -1
    offset.storage.topic: kafka-connector-cluster-test-offsets
    status.storage.replication.factor: -1
    status.storage.topic: kafka-connector-cluster-test-status
  image: >-
    .../infra/kafka-debezium-image:strimzi-0.45.0-kafka-3.9.0-debezium-2.7.1.Final
  jvmOptions:
    '-Xms': 1g
    '-Xmx': 1g
  livenessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
  logging:
    loggers:
      connect.root.logger.level: INFO
      log4j.logger.org.apache.kafka.connect.runtime.WorkerSinkTask: INFO
      log4j.logger.org.apache.kafka.connect.runtime.WorkerSourceTask: INFO
    type: inline
  metricsConfig:
    type: jmxPrometheusExporter
    valueFrom:
      configMapKeyRef:
        key: metrics-config.yml
        name: kafka-connector-cluster-cdc-connect-metrics
  readinessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
  replicas: 1
  resources:
    limits:
      cpu: 1000m
      ephemeral-storage: 5G
      memory: 2048Mi
    requests:
      cpu: 50m
      ephemeral-storage: 1G
      memory: 2048Mi
  template:
    connectContainer:
      env:
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: http://tempo-distributor.observability.svc.cluster.local:4317
        - name: OTEL_SERVICE_NAME
          value: kafka-connector-service
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        privileged: false
        readOnlyRootFilesystem: true
        runAsNonRoot: true
        runAsUser: 1001
    initContainer:
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        privileged: false
        readOnlyRootFilesystem: true
        runAsNonRoot: true
        runAsUser: 1001
    pod:
      metadata:
        annotations:
          checksum/config: a58d7f50e8aad938da69c3a8a2dfe83a2cb43812da390de8d3669b78ae5d534d
          cluster-autoscaler.kubernetes.io/safe-to-evict: 'true'
      securityContext:
        fsGroup: 100
        fsGroupChangePolicy: OnRootMismatch
        runAsGroup: 100
        runAsNonRoot: true
        runAsUser: 1001
        seccompProfile:
          type: RuntimeDefault
  tracing:
    type: opentelemetry
  version: 3.9.0

And logs:

kafka-connector-cluster-c1dc-connect-0.log

[scholzj]

Please share the new logs then. From both the Cluster Operator as well as the Connect cluster

[artempol-dev]

Well, I just rebstart operator and it helped to start a cluster connect. But problem stayed in another format. Now connectors just dont work with this upgrade

Operator and cluster logs:
0.46.0-strimzi-cluster-operator-5886f5974c-hbqqq.log

0.46.0-kafka-connector-cluster-cdc-connect-0.log

And connectors status:

  connectorStatus:
    connector:
      state: FAILED
      trace: "org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches io.debezium.connector.postgresql.PostgresConnector, available connectors are: PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorCheckpointConnector, name='org.apache.kafka.connect.mirror.MirrorCheckpointConnector', version='3.9.0', encodedVersion=3.9.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorHeartbeatConnector, name='org.apache.kafka.connect.mirror.MirrorHeartbeatConnector', version='3.9.0', encodedVersion=3.9.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorSourceConnector, name='org.apache.kafka.connect.mirror.MirrorSourceConnector', version='3.9.0', encodedVersion=3.9.0, type=source, typeName='source', location='classpath'}\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.connectorClass(Plugins.java:321)\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.newConnector(Plugins.java:292)\n\tat org.apache.kafka.connect.runtime.Worker.startConnector(Worker.java:314)\n\tat org.apache.kafka.connect.runtime.distributed.DistributedHerder.startConnector(DistributedHerder.java:2125)\n\tat org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$getConnectorStartingCallable$45(DistributedHerder.java:2131)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\n"
      worker_id: >-
        kafka-connector-cluster-cdc-connect-0.kafka-connector-cluster-cdc-connect.kafka2.svc:8083
    name: kafka-connector-cdc-fee-custody
    tasks:
      - id: 0
        state: FAILED
        trace: "org.apache.kafka.connect.errors.ConnectException: Failed to start task kafka-connector-cdc-fee-custody-0 since it is not a recognizable type (source or sink)\n\tat org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:2069)\n\tat org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$getTaskStartingCallable$40(DistributedHerder.java:2076)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\n"
        worker_id: >-
          kafka-connector-cluster-cdc-connect-0.kafka-connector-cluster-cdc-connect.kafka2.svc:8083

[artempol-dev]

Maybe 2.7.1 debezium version is too old for 0.46.0 strimzi version ? But I didnt see this in prerequisites

[scholzj]

Well, Debezium interfaces only with Kafka. Not with Strimzi. So as long as it worked with Kafka 3.9.0 in Strimzi 0.45.0 it should work with Kafka 3.9.0 in Strimzi 0.46.0. So I do not think that is the issue. From the log it seems like the plugin was not found in the image - are you sure it is there in the right place? Can you see the JARs when you exec into the Pod? How did you create the container image it is using?

[artempol-dev]

https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/2.7.1.Final/debezium-connector-postgres-2.7.1.Final-plugin.tar.gz

We just download this tar and build image with it
And basically nothing has changed , we used this image with 0.45.0 and 0.46.0

So I dont know what the difference