The kafka broker deployed on k8s is stuck in Terminal when it restarts or restarts abnormally

[lyd100822]

When the kafka broker deployed on k8s is restarted or restarted abnormally, it gets stuck in the terminal. How can I solve this problem? Are there any corresponding measures and suggestions on the kafka side? Is it related to the storage size?

[scholzj]

I guess first you have to figure out why is it stuck in terminal, whatever that means.

[lyd100822]

After the pod is restarted, I check the logs and describe pod k8s and find FailedKillPod. Later, I manually uninstall the storage, and then the pod can be automatically recycled. I don’t know why, so I ask the community. Although it is not often, it will be encountered once in a while. I want to ask if there is a good way to avoid it. Is it related to the storage size? For example, if the data is large, the recycling will be slower, and if the data is small, the recycling will be faster.

[lyd100822]

Do you need to split a single storage into multiple storages? For example, split 500G into 4 125G

[scholzj]

You are asking the community, but without providing any details. You have to provide the details, configurations, full logs, dumps of the Kubernetes events, explain your infrastructure etc. Without that, nobody knows what your problem is and what migth help.

[lyd100822]

Events:
  Type     Reason         Age                From     Message
  ----     ------         ----               ----     -------
  Warning  Unhealthy      38m (x3 over 39m)  kubelet  Liveness probe failed: OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "process_linux.go:86: executing setns process caused "exit status 21"": unknown
  Warning  FailedKillPod  28m (x6 over 38m)  kubelet  error killing pod: failed to "KillContainer" for "kafka" with KillContainerError: "rpc error: code = Unknown desc = operation timeout: context deadline exceeded"
  Normal   Killing        28m (x7 over 40m)  kubelet  Stopping container kafka
  Warning  Unhealthy      9s (x40 over 39m)  kubelet  Readiness probe failed: OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "process_linux.go:86: executing setns process caused "exit status 21"": unknown