Strimzi Kakfa pod is terminating and again gets active after certain period of time

[vaibhav0401]

Hi, i have 5 of the kafka pod out of which either of the pod is terminating and getting up after certain period of time, but causing the application to gets crashed. Even though 24 GB of JVM memory has been provided to it.
operatorLastSuccessfulVersion: 0.38.0

Error -
2025-06-24 02:18:10,052 INFO [ReplicaFetcher replicaId=2, leaderId=3, fetcherId=0] Error sending fetch request (sessionId=790042141, epoch=INITIAL) to node 3: (org.apache.kafka.clients.FetchSessionHandler) [ReplicaFetcherThread-0-3]
java.net.SocketTimeoutException: Failed to connect within 30000 ms
at kafka.server.BrokerBlockingSender.sendRequest(BrokerBlockingSender.scala:109)
at kafka.server.RemoteLeaderEndPoint.fetch(RemoteLeaderEndPoint.scala:79)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:316)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:130)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:129)
at scala.Option.foreach(Option.scala:437)
at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:112)
at kafka.server.ReplicaFetcherThread.doWork(ReplicaFetcherThread.scala:98)
at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:130)
2025-06-24 02:18:10,053 WARN [ReplicaFetcher replicaId=2, leaderId=3, fetcherId=0] Error in response for fetch request (type=FetchRequest, replicaId=2, maxWait=500, minBytes=1, maxBytes=10485760, fetchData={fm-ingestion-0=PartitionData(topicId=iILVc1EqSpW5sLdVu7toIw, fetchOffset=0, logStartOffset=0, maxBytes=1048576, currentLeaderEpoch=Optional[156], lastFetchedEpoch=Optional.empty)}, isolationLevel=READ_UNCOMMITTED, removed=, replaced=, metadata=(sessionId=790042141, epoch=INITIAL), rackId=) (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-3]
java.net.SocketTimeoutException: Failed to connect within 30000 ms
at kafka.server.BrokerBlockingSender.sendRequest(BrokerBlockingSender.scala:109)
at kafka.server.RemoteLeaderEndPoint.fetch(RemoteLeaderEndPoint.scala:79)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:316)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:130)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:129)
at scala.Option.foreach(Option.scala:437)
at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:112)
at kafka.server.ReplicaFetcherThread.doWork(ReplicaFetcherThread.scala:98)
at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:130)
2025-06-24 02:18:41,072 INFO [ReplicaFetcher replicaId=2, leaderId=3, fetcherId=0] Client requested connection close from node 3 (org.apache.kafka.clients.NetworkClient) [ReplicaFetcherThread-0-3]
2025-06-24 02:18:41,073 INFO [ReplicaFetcher replicaId=2, leaderId=3, fetcherId=0] Error sending fetch request (sessionId=790042141, epoch=INITIAL) to node 3: (org.apache.kafka.clients.FetchSessionHandler) [ReplicaFetcherThread-0-3]
java.net.SocketTimeoutException: Failed to connect within 30000 ms
at kafka.server.BrokerBlockingSender.sendRequest(BrokerBlockingSender.scala:109)
at kafka.server.RemoteLeaderEndPoint.fetch(RemoteLeaderEndPoint.scala:79)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:316)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:130)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:129)
at scala.Option.foreach(Option.scala:437)
at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:112)
at kafka.server.ReplicaFetcherThread.doWork(ReplicaFetcherThread.scala:98)
at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:130)
2025-06-24 02:18:41,073 WARN [ReplicaFetcher replicaId=2, leaderId=3, fetcherId=0] Error in response for fetch request (type=FetchRequest, replicaId=2, maxWait=500, minBytes=1, maxBytes=10485760, fetchData={fm-ingestion-0=PartitionData(topicId=iILVc1EqSpW5sLdVu7toIw, fetchOffset=0, logStartOffset=0, maxBytes=1048576, currentLeaderEpoch=Optional[156], lastFetchedEpoch=Optional.empty)}, isolationLevel=READ_UNCOMMITTED, removed=, replaced=, metadata=(sessionId=790042141, epoch=INITIAL), rackId=) (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-3]
java.net.SocketTimeoutException: Failed to connect within 30000 ms
at kafka.server.BrokerBlockingSender.sendRequest(BrokerBlockingSender.scala:109)
at kafka.server.RemoteLeaderEndPoint.fetch(RemoteLeaderEndPoint.scala:79)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:316)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:130)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:129)
at scala.Option.foreach(Option.scala:437)
at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:112)
at kafka.server.ReplicaFetcherThread.doWork(ReplicaFetcherThread.scala:98)
at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:130)
2025-06-24 02:19:12,104 INFO [ReplicaFetcher replicaId=2, leaderId=3, fetcherId=0] Client requested connection close from node 3 (org.apache.kafka.clients.NetworkClient) [ReplicaFetcherThread-0-3]
2025-06-24 02:19:12,104 INFO [ReplicaFetcher replicaId=2, leaderId=3, fetcherId=0] Error sending fetch request (sessionId=790042141, epoch=INITIAL) to node 3: (org.apache.kafka.clients.FetchSessionHandler) [ReplicaFetcherThread-0-3]
java.net.SocketTimeoutException: Failed to connect within 30000 ms
at kafka.server.BrokerBlockingSender.sendRequest(BrokerBlockingSender.scala:109)
at kafka.server.RemoteLeaderEndPoint.fetch(RemoteLeaderEndPoint.scala:79)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:316)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:130)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:129)
at scala.Option.foreach(Option.scala:437)
at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:112)
at kafka.server.ReplicaFetcherThread.doWork(ReplicaFetcherThread.scala:98)
at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:130)
2025-06-24 02:19:12,105 WARN [ReplicaFetcher replicaId=2, leaderId=3, fetcherId=0] Error in response for fetch request (type=FetchRequest, replicaId=2, maxWait=500, minBytes=1, maxBytes=10485760, fetchData={fm-ingestion-0=PartitionData(topicId=iILVc1EqSpW5sLdVu7toIw, fetchOffset=0, logStartOffset=0, maxBytes=1048576, currentLeaderEpoch=Optional[156], lastFetchedEpoch=Optional.empty)}, isolationLevel=READ_UNCOMMITTED, removed=, replaced=, metadata=(sessionId=790042141, epoch=INITIAL), rackId=) (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-3]
java.net.SocketTimeoutException: Failed to connect within 30000 ms
at kafka.server.BrokerBlockingSender.sendRequest(BrokerBlockingSender.scala:109)
at kafka.server.RemoteLeaderEndPoint.fetch(RemoteLeaderEndPoint.scala:79)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:316)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:130)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:129)
at scala.Option.foreach(Option.scala:437)
at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:112)
at kafka.server.ReplicaFetcherThread.doWork(ReplicaFetcherThread.scala:98)
at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:130)
2025-06-24 02:24:12,964 WARN Client session timed out, have not heard from server in 279157ms for session id 0x301e3a401100050 (org.apache.zookeeper.ClientCnxn) [main-SendThread(strimzi-cluster-zookeeper-client:2181)]
2025-06-24 02:24:12,964 INFO [ReplicaFetcher replicaId=2, leaderId=3, fetcherId=0] Client requested connection close from node 3 (org.apache.kafka.clients.NetworkClient) [ReplicaFetcherThread-0-3]
2025-06-24 02:24:13,077 WARN Session 0x301e3a401100050 for server strimzi-cluster-zookeeper-client/172.30.92.148:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn) [main-SendThread(strimzi-cluster-zookeeper-client:2181)]
org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 279157ms for session id 0x301e3a401100050
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1257)
2025-06-24 02:24:13,078 INFO [ReplicaFetcher replicaId=2, leaderId=3, fetcherId=0] Error sending fetch request (sessionId=790042141, epoch=INITIAL) to node 3: (org.apache.kafka.clients.FetchSessionHandler) [ReplicaFetcherThread-0-3]
java.net.SocketTimeoutException: Failed to connect within 30000 ms
at kafka.server.BrokerBlockingSender.sendRequest(BrokerBlockingSender.scala:109)
at kafka.server.RemoteLeaderEndPoint.fetch(RemoteLeaderEndPoint.scala:79)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:316)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:130)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:129)
at scala.Option.foreach(Option.scala:437)
at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:112)
at kafka.server.ReplicaFetcherThread.doWork(ReplicaFetcherThread.scala:98)
at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:130)
2025-06-24 02:24:13,079 INFO [ReplicaFetcher replicaId=2, leaderId=4, fetcherId=0] Disconnecting from node 4 due to request timeout. (org.apache.kafka.clients.NetworkClient) [ReplicaFetcherThread-0-4]
2025-06-24 02:24:13,080 WARN [ReplicaFetcher replicaId=2, leaderId=3, fetcherId=0] Error in response for fetch request (type=FetchRequest, replicaId=2, maxWait=500, minBytes=1, maxBytes=10485760, fetchData={fm-ingestion-0=PartitionData(topicId=iILVc1EqSpW5sLdVu7toIw, fetchOffset=0, logStartOffset=0, maxBytes=1048576, currentLeaderEpoch=Optional[156], lastFetchedEpoch=Optional.empty)}, isolationLevel=READ_UNCOMMITTED, removed=, replaced=, metadata=(sessionId=790042141, epoch=INITIAL), rackId=) (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-3]
java.net.SocketTimeoutException: Failed to connect within 30000 ms
at kafka.server.BrokerBlockingSender.sendRequest(BrokerBlockingSender.scala:109)
at kafka.server.RemoteLeaderEndPoint.fetch(RemoteLeaderEndPoint.scala:79)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:316)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:130)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:129)
at scala.Option.foreach(Option.scala:437)
at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:112)
at kafka.server.ReplicaFetcherThread.doWork(ReplicaFetcherThread.scala:98)
at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:130)
2025-06-24 02:24:13,081 INFO [ReplicaFetcher replicaId=2, leaderId=0, fetcherId=0] Partition unit-nims-inventory-feeder-0 has an older epoch (1495) than the current leader. Will await the new LeaderAndIsr state before resuming fetching. (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-0]
2025-06-24 02:24:13,081 INFO [ReplicaFetcher replicaId=2, leaderId=4, fetcherId=0] Cancelled in-flight FETCH request with correlation id 14624 due to node 4 being disconnected (elapsed time since creation: 273504ms, elapsed time since send: 273504ms, request timeout: 30000ms) (org.apache.kafka.clients.NetworkClient) [ReplicaFetcherThread-0-4]
2025-06-24 02:24:13,082 INFO [ReplicaFetcher replicaId=2, leaderId=4, fetcherId=0] Client requested connection close from node 4 (org.apache.kafka.clients.NetworkClient) [ReplicaFetcherThread-0-4]
2025-06-24 02:24:13,082 INFO [ReplicaFetcher replicaId=2, leaderId=4, fetcherId=0] Error sending fetch request (sessionId=1857628606, epoch=14624) to node 4: (org.apache.kafka.clients.FetchSessionHandler) [ReplicaFetcherThread-0-4]
java.io.IOException: Connection to 4 was disconnected before the response was read
at org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:99)
at kafka.server.BrokerBlockingSender.sendRequest(BrokerBlockingSender.scala:113)
at kafka.server.RemoteLeaderEndPoint.fetch(RemoteLeaderEndPoint.scala:79)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:316)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:130)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:129)
at scala.Option.foreach(Option.scala:437)
at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:112)
at kafka.server.ReplicaFetcherThread.doWork(ReplicaFetcherThread.scala:98)
at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:130)
2025-06-24 02:24:13,082 WARN [ReplicaFetcher replicaId=2, leaderId=4, fetcherId=0] Error in response for fetch request (type=FetchRequest, replicaId=2, maxWait=500, minBytes=1, maxBytes=10485760, fetchData={}, isolationLevel=READ_UNCOMMITTED, removed=, replaced=, metadata=(sessionId=1857628606, epoch=14624), rackId=) (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-4]
java.io.IOException: Connection to 4 was disconnected before the response was read
at org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:99)
at kafka.server.BrokerBlockingSender.sendRequest(BrokerBlockingSender.scala:113)
at kafka.server.RemoteLeaderEndPoint.fetch(RemoteLeaderEndPoint.scala:79)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:316)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:130)
at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:129)
at scala.Option.foreach(Option.scala:437)
at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:112)
at kafka.server.ReplicaFetcherThread.doWork(ReplicaFetcherThread.scala:98)
at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:130)
2025-06-24 02:24:13,083 INFO [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Partition __consumer_offsets-46 has an older epoch (1359) than the current leader. Will await the new LeaderAndIsr state before resuming fetching. (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-1]
2025-06-24 02:24:13,084 INFO [GroupCoordinator 2]: Member consumer-importing-granite-data-group-3-15650cb9-d185-44c7-823b-ef30b5dbf843 in group importing-granite-data-group has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [executor-Heartbeat]
2025-06-24 02:24:13,085 INFO [GroupCoordinator 2]: Preparing to rebalance group importing-granite-d

Pod description events
Events:
Type Reason Age From Message

---

Normal Killing 2m50s kubelet Stopping container kafka
Warning Unhealthy 2m16s kubelet Readiness probe errored: rpc error: code = Unknown desc = command error: time="2025-06-24T06:45:47Z" level=fatal msg="nsexec-1[149007]: failed to open /proc/1903671/ns/ipc: No such file or directory"
time="2025-06-24T06:45:47Z" level=fatal msg="nsexec-0[149000]: failed to sync with stage-1: next state: Invalid argument"
time="2025-06-24T06:45:47Z" level=error msg="exec failed: unable to start container process: error executing setns process: exit status 1"
, stdout: , stderr: , exit code -1
Warning Unhealthy 106s kubelet Readiness probe errored: rpc error: code = Unknown desc = command error: time="2025-06-24T06:46:17Z" level=fatal msg="nsexec-1[160990]: failed to open /proc/1903671/ns/ipc: No such file or directory"
time="2025-06-24T06:46:17Z" level=fatal msg="nsexec-0[160981]: failed to sync with stage-1: next state: Invalid argument"
time="2025-06-24T06:46:17Z" level=error msg="exec failed: unable to start container process: error executing setns process: exit status 1"
, stdout: , stderr: , exit code -1
Warning Unhealthy 76s kubelet Readiness probe errored: rpc error: code = Unknown desc = command error: time="2025-06-24T06:46:47Z" level=fatal msg="nsexec-1[174221]: failed to open /proc/1903671/ns/ipc: No such file or directory"
time="2025-06-24T06:46:47Z" level=fatal msg="nsexec-0[174214]: failed to sync with stage-1: next state: Invalid argument"
time="2025-06-24T06:46:47Z" level=error msg="exec failed: unable to start container process: error executing setns process: exit status 1"
, stdout: , stderr: , exit code -1
Warning Unhealthy 46s kubelet Readiness probe errored: rpc error: code = Unknown desc = command error: time="2025-06-24T06:47:17Z" level=fatal msg="nsexec-1[186454]: failed to open /proc/1903671/ns/ipc: No such file or directory"
time="2025-06-24T06:47:17Z" level=fatal msg="nsexec-0[186442]: failed to sync with stage-1: next state: Invalid argument"
time="2025-06-24T06:47:17Z" level=error msg="exec failed: unable to start container process: error executing setns process: exit status 1"
, stdout: , stderr: , exit code -1
Warning Unhealthy 16s kubelet Readiness probe errored: rpc error: code = Unknown desc = command error: time="2025-06-24T06:47:47Z" level=fatal msg="nsexec-1[198411]: failed to open /proc/1903671/ns/ipc: No such file or directory"
time="2025-06-24T06:47:47Z" level=fatal msg="nsexec-0[198402]: failed to sync with stage-1: next state: Invalid argument"
time="2025-06-24T06:47:47Z" level=error msg="exec failed: unable to start container process: error executing setns process: exit status 1"
, stdout: , stderr: , exit code -1

Thank you in Advance

[im-konge]

Please keep the discussion on one place.

As I mentioned on the Slack:

Hey, this is not much info about the problem. There is some connection issue it seems, but what was before this was happening? Did you do some upgrade? Did you change anything? Are you using ZK or KRaft? What is the operator version?
So I would start with sharing full logs and configuration

[showuon]

From the log, it looks the ZK and kafka are fine. The ZK connection is recovered after 2 secs and everything else works as expected.
2025-06-30 05:37:33,306 INFO Session establishment complete on server strimzi-cluster-zookeeper-client/172.30.92.148:2181, session id = 0x2020558e6cd00e2, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn) [nioEventLoopGroup-2-1]

we are observing that the Kafka pod is repeatedly terminating.

From the Kafka log, I see the kafka is not graceful shutdown:

2025-06-30 06:31:10,848 INFO [GroupCoordinator 0]: Preparing to rebalance group fm-summarization-response-group in state PreparingRebalance with old generation 4237 (__consumer_offsets-35) (reason: Adding new member consumer-fm-summarization-response-group-1-c407a54f-d4fb-4f2e-b324-9065c7ba8be5 with group instance id None; client reason: rebalance failed due to MemberIdRequiredException) (kafka.coordinator.group.GroupCoordinator) [data-plane-kafka-request-handler-4]
2025-06-30 06:31:55,725 INFO [GroupCoordinator 0]: Member consumer-fm-summarization-response-group-1-e364d4ab-bffe-4ed7-88d9-b2acc0a9d870 in group fm-summarization-response-group has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [executor-Heartbeat]
2025-06-30 06:31:55,726 INFO [GroupCoordinator 0]: Stabilized group fm-summarization-response-group generation 4238 (__consumer_offsets-35) with 2 members (kafka.coordinator.group.GroupCoordinator) [executor-Heartbeat]
2025-06-30 06:31:55,731 INFO [GroupCoordinator 0]: Assignment received from leader consumer-fm-summarization-response-group-1-c08f0c69-70ed-4f62-a162-93d66854d69b for group fm-summarization-response-group for generation 4238. The group has 2 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator) [data-plane-kafka-request-handler-0]
STRIMZI_BROKER_ID=0
Preparing truststore for replication listener
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/kafka/cluster.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore for replication listener is complete
Looking for the CA matching the server certificate
...

It means the kafka is not shutdown by kafka itself or strimzi operator. It could be terminated by kubernetes, or other reason, ex: resource not enough. Could you check kubectl events to see if there's anything happened in your kubernetes env?

[vaibhav0401]

Hi, We restarted the Kafka pod and monitored it from the beginning. Here are our observations:

1 -Pod becomes unresponsive after running fine for a few hours. It stays unresponsive for around 11 minutes, starts throwing errors, and then gets terminated. A new pod is automatically created after about 4 minutes.
During this time, we see multiple container-level errors like:

Warning  Unhealthy  4m26s  kubelet  Readiness probe errored: rpc error: code = Unknown desc = command error: time="2025-06-20T07:16:10Z" level=fatal msg="nsexec-1[3988679]: failed to open /proc/429935/ns/ipc: No such file or directory"
time="2025-06-20T07:16:10Z" level=fatal msg="nsexec-0[3988669]: failed to sync with stage-1: next state: Invalid argument"
time="2025-06-20T07:16:10Z" level=error msg="exec failed: unable to start container process: error executing setns process: exit status 1"
, stdout: , stderr: , exit code -1
  Warning  Unhealthy  3m56s  kubelet  Readiness probe errored: rpc error: code = Unknown desc = command error: time="2025-06-20T07:16:40Z" level=fatal msg="nsexec-1[4000192]: failed to open /proc/429935/ns/ipc: No such file or directory"
time="2025-06-20T07:16:40Z" level=fatal msg="nsexec-0[4000179]: failed to sync with stage-1: next state: Invalid argument"
time="2025-06-20T07:16:40Z" level=error msg="exec failed: unable to start container process: error executing setns process: exit status 1"
, stdout: , stderr: , exit code -1
  Warning  Unhealthy  3m26s  kubelet  Readiness probe errored: rpc error: code = Unknown desc = command error: time="2025-06-20T07:17:10Z" level=fatal msg="nsexec-1[4011452]: failed to open /proc/429935/ns/ipc: No such file or directory"
time="2025-06-20T07:17:10Z" level=fatal msg="nsexec-0[4011442]: failed to sync with stage-1: next state: Invalid argument"
time="2025-06-20T07:17:10Z" level=error msg="exec failed: unable to start container process: error executing setns process: exit status 1"
, stdout: , stderr: , exit code -1
  Warning  Unhealthy  2m56s  kubelet  Readiness probe errored: rpc error: code = Unknown desc = command error: time="2025-06-20T07:17:40Z" level=fatal msg="nsexec-1[4022147]: failed to open /proc/429935/ns/ipc: No such file or directory"
time="2025-06-20T07:17:40Z" level=fatal msg="nsexec-0[4022140]: failed to sync with stage-1: next state: Invalid argument"
time="2025-06-20T07:17:40Z" level=error msg="exec failed: unable to start container process: error executing setns process: exit status 1"
, stdout: , stderr: , exit code -1
  Warning  Unhealthy  2m26s  kubelet  Readiness probe errored: rpc error: code = Unknown desc = command error: time="2025-06-20T07:18:10Z" level=fatal msg="nsexec-1[4033436]: failed to open /proc/429935/ns/ipc: No such file or directory"
time="2025-06-20T07:18:10Z" level=fatal msg="nsexec-0[4033420]: failed to sync with stage-1: next state: Invalid argument"
time="2025-06-20T07:18:10Z" level=error msg="exec failed: unable to start container process: error executing setns process: exit status 1"
, stdout: , stderr: , exit code -1
  Warning  Unhealthy  116s  kubelet  Readiness probe errored: rpc error: code = Unknown desc = command error: time="2025-06-20T07:18:40Z" level=fatal msg="nsexec-1[4046158]: failed to open /proc/429935/ns/ipc: No such file or directory"
time="2025-06-20T07:18:40Z" level=fatal msg="nsexec-0[4046148]: failed to sync with stage-1: next state: Invalid argument"
time="2025-06-20T07:18:40Z" level=error msg="exec failed: unable to start container process: error executing setns process: exit status 1"
, stdout: , stderr: , exit code -1
  Warning  Unhealthy  86s  kubelet  Readiness probe errored: rpc error: code = Unknown desc = command error: time="2025-06-20T07:19:10Z" level=fatal msg="nsexec-1[4059584]: failed to open /proc/429935/ns/ipc: No such file or directory"
time="2025-06-20T07:19:10Z" level=fatal msg="nsexec-0[4059575]: failed to sync with stage-1: next state: Invalid argument"
time="2025-06-20T07:19:10Z" level=error msg="exec failed: unable to start container process: error executing setns process: exit status 1"
, stdout: , stderr: , exit code -1
  Warning  Unhealthy  56s  kubelet  Readiness probe errored: rpc error: code = Unknown desc = command error: time="2025-06-20T07:19:40Z" level=fatal msg="nsexec-1[4072541]: failed to open /proc/429935/ns/ipc: No such file or directory"
time="2025-06-20T07:19:40Z" level=fatal msg="nsexec-0[4072525]: failed to sync with stage-1: next state: Invalid argument"
time="2025-06-20T07:19:40Z" level=error msg="exec failed: unable to start container process: error executing setns process: exit status 1"
, stdout: , stderr: , exit code -1
  Normal   Killing        53s (x2 over 5m23s)  kubelet  Stopping container kafka
  Warning  FailedKillPod  53s                  kubelet  error killing pod: [failed to "KillContainer" for "kafka" with KillContainerError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded", failed to "KillPodSandbox" for "1944ebed-6608-485b-8df6-e810973204e1" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"]
  Warning  Unhealthy      26s                  kubelet  Readiness probe errored: rpc error: code = Unknown desc = command error: time="2025-06-20T07:20:10Z" level=fatal msg="nsexec-1[4085947]: failed to open /proc/429935/ns/ipc: No such file or directory"
time="2025-06-20T07:20:10Z" level=fatal msg="nsexec-0[4085927]: failed to sync with stage-1: next state: Invalid argument"
time="2025-06-20T07:20:10Z" level=error msg="exec failed: unable to start container process: error executing setns process: exit status 1"
, stdout: , stderr: , exit code -1

This seems to be related to the container runtime trying to access a missing process.

2 -We verified that the process ID mentioned in the error (429935) does not exist, and neither does the associated /proc/429935/ns/ipc directory. So the cleanup of the process seems proper.

3 -We also ran kubectl get events but found no related events in the namespace that point to any cluster-level issue.

4 -We couldn't find any specific issue with Kafka, Zookeeper, or the cluster logs. The errors are pointing towards container runtime or node-level issues (e.g., crio or kubelet).

Can you please help us confirm if this is a node-level problem, or suggest how we can collect further evidence to confirm this?

Thanks.

[showuon]

Yes, it looks like it's not the operator's issue. Usually you can find something in kubectl events, if no related events, I have no clue on this, sorry. Not sure if @im-konge has any suggestion?

[im-konge]

The only thing that comes to my mind would be if the containers have enough resources - just becase the fact that it runs for few hours without problem and then it fails. But based on the provided configuration, it should be fine. (plus the process seems to be not related to anything around resources etc.)
I don't have any other check in my mind now TBH.

As Luke mentioned, it's not operator's issue.

[im-konge]

One thing that I found is that it could be possibly OOM killed or something. But it's a bit shot in the dark.
Would it be possible to add some watcher to the Pod and the container to see if there is some starvation or something like that?

[vaibhav0401]

logs.zip