KafkaConnect not started Error Failed to create new KafkaAdminClient / Invalid PEM keystore configs

[totochichi]

Hello, while deploying KafkaConnect with a custom image using Strimzi Operator 0.47.0 and Kafka 3.9.0 (or newer), my Connect pods failed with some InvalidKeyexception errors.

In previous successful deployments (Strimzi 0.45.0 Kafka 3.8.0) we have always used secrets created along with the Connect Ingress object. I noticed there is some new secret objects created (connect-tls-trusted-certs) that is referenced in the connect config map, and separate kafka node secrets for each pods. However all Kafka pods are running fine, and we have not changed any secrets format nor any template, not sure if I am missing on any crucial config update here.

I have attached the config yaml, config map and logs below
connect_cm.txt
connect_config.txt
(connect-logs.txt)

Log extracted:

2025-07-29 20:20:27,897 ERROR Stopping due to error (org.apache.kafka.connect.cli.AbstractConnectCli) [main]
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
        at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:561)
        at org.apache.kafka.clients.admin.Admin.create(Admin.java:147)
        at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:282)
        at org.apache.kafka.connect.runtime.WorkerConfig.kafkaClusterId(WorkerConfig.java:413)
        at org.apache.kafka.connect.cli.AbstractConnectCli.startConnect(AbstractConnectCli.java:124)
        at org.apache.kafka.connect.cli.AbstractConnectCli.run(AbstractConnectCli.java:95)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:112)
Caused by: org.apache.kafka.common.KafkaException: Failed to create new NetworkClient
        at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:255)
        at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:190)
        at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:545)
        ... 6 more
Caused by: org.apache.kafka.common.errors.InvalidConfigurationException: Invalid PEM keystore configs
Caused by: org.apache.kafka.common.errors.InvalidConfigurationException: Private key could not be loaded
Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException : algid parse error, not a sequence
        at java.base/sun.security.rsa.RSAKeyFactory.engineGeneratePrivate(RSAKeyFactory.java:253)
        at java.base/java.security.KeyFactory.generatePrivate(KeyFactory.java:389)
        at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.privateKey(DefaultSslEngineFactory.java:527)
        at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.createKeyStoreFromPem(DefaultSslEngineFactory.java:465)
        at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.<init>(DefaultSslEngineFactory.java:438)
        at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:289)
        at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:162)
        at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:147)
        at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:100)
        at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:70)
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:193)
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:82)
        at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:120)
        at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:224)
        at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:190)
        at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:545)
        at org.apache.kafka.clients.admin.Admin.create(Admin.java:147)
        at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:282)
        at org.apache.kafka.connect.runtime.WorkerConfig.kafkaClusterId(WorkerConfig.java:413)
        at org.apache.kafka.connect.cli.AbstractConnectCli.startConnect(AbstractConnectCli.java:124)
        at org.apache.kafka.connect.cli.AbstractConnectCli.run(AbstractConnectCli.java:95)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:112)
Caused by: java.security.InvalidKeyException: IOException : algid parse error, not a sequence
        at java.base/sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:135)
        at java.base/sun.security.pkcs.PKCS8Key.<init>(PKCS8Key.java:95)
        at java.base/sun.security.rsa.RSAPrivateCrtKeyImpl.<init>(RSAPrivateCrtKeyImpl.java:162)
        at java.base/sun.security.rsa.RSAPrivateCrtKeyImpl.newKey(RSAPrivateCrtKeyImpl.java:92)
        at java.base/sun.security.rsa.RSAKeyFactory.generatePrivate(RSAKeyFactory.java:352)
        at java.base/sun.security.rsa.RSAKeyFactory.engineGeneratePrivate(RSAKeyFactory.java:249)
        ... 21 more

Thank you, I appreciate any inputs!

[scholzj]

Well, the errors seem pretty obvious and suggest issues with loading the private key. So that is probably where you need to look. Without you cluster, without knowing there there are no typos, without knowing what is in the secrets etc., it is hard to say much more than what the error tells you.

Also, from the log, you are clearly running Kafka 4.0. and not 3.9.0.

[totochichi]

Yes I wanted to highlight I run into the issue with Kafka 3.9.0 and newer, so including 4.0.0 in my case- my effort to isolate if the change happened with Kafka or Strimzi operator.

We did not make change to the Connect templates so the cert generated is consistent, and works well with earlier version of Strimzi operator (0.45.0).

I am not an expert in certs in general, and might have missed out on some principles, but the below step helps me resolve the issue:

The log seems to complain about InvalidKeySpecException, and is looking for PKCS8 format, which the header should look like:
-----BEGIN PRIVATE KEY----- while my cert header was -----BEGIN RSA PRIVATE KEY----- . So I converted the tls.key into PKCS8 format, and now the Connect pods rolled out successfully

Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException : algid parse error, not a sequence
	at java.base/sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:135) ~[?:?]
	at java.base/sun.security.pkcs.PKCS8Key.<init>(PKCS8Key.java:95) ~[?:?]

I see this PR on OpenSSL a little while ago that seems to explain the new errors thrown, which we did not see for Strimzi 0.45.0

[im-konge]

I guess that it's not issue in OpenSSL or anything, there was change in the way how the certs and other configuration loaded IIRC. So if you used the certs from the env variables, it's done in a different way.

[scholzj]

I'm not sure what do you mean with this ...

We did not make change to the Connect templates so the cert generated is consistent, and works well with earlier version of Strimzi operator (0.45.0).

There are no templates. You supply the certificates and it is up to you to supply them in the correct format. The #11224 PR has nothing to do with it. But it could be that the Strimzi 0.45 was able to process the incorrect certificate as it was configured slightly differently - now the certificates are handled directly by Kafka Connect. But to be honest that was just a coincidence and not something what worked by design. I don't think we were ever aware that this might have worked.

In any case, glad it works now.