Getting SSL handshake failed for zookeeper

[shreyasarani23]

I wanted to check the logs of my kafka broker so I did kubectl logs my-kafka-cluster-kafka-1 -n kafka .
Below is the log my kafka broker

2021-06-07 07:17:11,483 INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /10.244.1.60 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-1-ListenerName(REPLICATION-9091)-SSL-3]
2021-06-07 07:17:11,583 INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /10.244.1.60 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-1-ListenerName(REPLICATION-9091)-SSL-4]
2021-06-07 07:17:11,603 INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /10.244.1.60 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-1-ListenerName(REPLICATION-9091)-SSL-5]

Any idea on why I am getting SSL handshake failed?

Also attaching the logs of my zookeeper pods

2021-06-07 07:23:19,941 ERROR Unsuccessful handshake with session 0x0 (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-7-1]
2021-06-07 07:23:19,941 WARN Exception caught (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-7-1]
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: decrypt_error
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: decrypt_error
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:336)
        at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
        at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:185)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:171)
        at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681)
        at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433)
        at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637)
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:284)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1395)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1290)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1337)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
        ... 17 more
2021-06-07 07:23:20,253 INFO Received connection request from /10.244.1.58:53904 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [my-kafka-cluster-zookeeper-1.my-kafka-cluster-zookeeper-nodes.kafka.svc/10.244.5.34:3888]
2021-06-07 07:23:20,259 INFO Accepted TLS connection from /10.244.1.58:53904 - NONE - SSL_NULL_WITH_NULL_NULL (org.apache.zookeeper.server.quorum.UnifiedServerSocket) [my-kafka-cluster-zookeeper-1.my-kafka-cluster-zookeeper-nodes.kafka.svc/10.244.5.34:3888]
2021-06-07 07:23:20,259 WARN Exception reading or writing challenge: {} (org.apache.zookeeper.server.quorum.QuorumCnxManager) [my-kafka-cluster-zookeeper-1.my-kafka-cluster-zookeeper-nodes.kafka.svc/10.244.5.34:3888]
java.net.SocketException: Socket is closed
        at java.base/sun.security.ssl.SSLSocketImpl.getInputStream(SSLSocketImpl.java:797)
        at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.getRealInputStream(UnifiedServerSocket.java:700)
        at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.read(UnifiedServerSocket.java:694)
        at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
        at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
        at java.base/java.io.DataInputStream.readFully(DataInputStream.java:200)
        at java.base/java.io.DataInputStream.readLong(DataInputStream.java:421)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:571)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:522)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:945)
2021-06-07 07:23:20,792 INFO Processing ruok command from /127.0.0.1:46822 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2021-06-07 07:23:21,133 INFO Processing ruok command from /127.0.0.1:46824 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]

[scholzj]

TBH, I'm not sure I understand the errors. Do you have by any chance some stuff like Istio installed there injecting some sidecars etc.?

[shreyasarani23]

Sorry what do you mean by nobody should be trying to connect to these can u elaborate?. Actually I have spring boot service which is using kafka. In spring boot config I have given bootstrap server address my-kafka-cluster-kafka-bootstrap.kafka.svc:9092 to connect to kafka. And cluster is working fine I able to produce and consume messages by running producer and consumer docker image of kafka. I guess service uses some kind of ssl configuration

[scholzj]

Sorry what do you mean by nobody should be trying to connect to these can u elaborate?

The Kafka port 9091 is internal port used for replication. So only other Kafka brokers should be connecting there. The same applies to the 3888 port in ZooKeeper. Only other ZooKeepers should be connecting there. These ports are normally protected by Network Policies, so that other apps should not be able to connect to them. But it might depend a bit on your Kube cluster, whether it implements network policies etc.

Actually I have spring boot service which is using kafka. In spring boot config I have given bootstrap server address my-kafka-cluster-kafka-bootstrap.kafka.svc:9092 to connect to kafka. And cluster is working fine I able to produce and consume messages by running producer and consumer docker image of kafka. I guess service uses some kind of ssl configuration

I guess in that case - if the cluster seems to work fine otherwise - maybe these could be also some temporary network glitches or something like that? With TLS, a connection terminated at a wrong point can cause all kind of weird errors.

[shreyasarani23]

Ohk @scholzj got it. Any work around to remove these errors?

[scholzj]

No idea without knowing exactly what is causing them I'm afraid.

[nicolaelapteanu]

Hello,
In case anyone reaches this thread:

• I seem to see these logs in the zookeepers when I use zoo-entrance for my cluster.
  if I scale this deployment to 0 they are gone immediately.

Thank you!

later add:
I have found where the issue was on my side -> we were rebuilding Strimzi Kafka cluster and were not restarting deployment for zoo-entrance. so it was caching certificates of an older cluster while connecting to the new one.