Strimzi
Replies: 2 comments 16 replies
-
|
That the Pod is killed after the termination grace period is expected. This is how it works. Different nodes migth take different time to shutdown - that is at the end also why it is also configurable. But I think shutting down for 15 minutes indicates some kind of issue that you would need to investigate. One of the things that could cause so long shutdowns is when the shutdown is started during a recovery after an unclean shutdown (as Kafka will not shutdown before completing the recovery). This in a way can create a cycle when one unclean shutdown is causing more unclean shutdowns due to the recovery. Your logs do not suggest this is the case as they do not list any of the recovery logs. But they are also not complete. So I cannot say that this is not the issue here for sure. Given you say that you use KRaft and Kafka 3.7.0, I would suggest you to also consider upgrade as KRaft in 3.7 has still many missing features and issues. There is always chance this is some bug that migth be fixed in newer version. |
Beta Was this translation helpful? Give feedback.
-
|
It looks like the long shutdown has some issues communicating with some other nodes. But I don't see anything that would be the reason :-(. |
Beta Was this translation helpful? Give feedback.
-
|
Hey! Just a follow up on this. We managed to pinpoint that during the Kafka broker restarts (via strimzi.io/manual-rolling-update) and while the broker is in a terminating state, the network requests no longer pass the network policies that we have in place. Which explains the timeouts in the logs between the brokers and the controllers. Indeed, if we add an allow all network policy this problem doesn't happen. We are still trying to pinpoint why, e.g. we don't see the broker pod labels changing when the pod transitions to this state. |
Beta Was this translation helpful? Give feedback.
-
|
The network policies are "updated" shortsly before the manual rolling update: ... but I say "updated" because normally there is no change to them - especially to the internal ports used for replication and talking to controllers that have the same settings all the time. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @scholzj, The solution we will move forward with is to disable the option for Strimzi to automatically create this network policy, and create our own instead. |
Beta Was this translation helpful? Give feedback.
-
|
Sounds like some bug in Cilium. But yes, the option to disable them should be an option. When you disable it, the original network policies will remain in place. So you will only need to update them if you change your cluster - e.g. add a listener etc. |
Beta Was this translation helpful? Give feedback.
-
|
I have a similar scenario here on GKE (Google Cloud Platform). I started using strimzi 0.38 / Kafka 3.6 with ZK, and the brokers were taking a long time to restart (more than 30 minutes). This results in a very long restart time. terminationGracePeriodSeconds configured with 300 seconds. I also noticed that while one broker was still starting (not ready), another broker was already shutting down, which, in my case, can lead to topic creation failures. Any initiatives to resolve this scenario? Environment: |
Beta Was this translation helpful? Give feedback.
-
|
Hey @jairogubler
What scenario are you referring to? The scenario where Strimzi updates the network policies shortly before doing the manual rolling update? There is no plans to change that |
Beta Was this translation helpful? Give feedback.
-
|
I'm referring to the fact that the broker doesn't finish normally, because it can't communicate with the controllers during the termination. |
Beta Was this translation helpful? Give feedback.
-
|
If your brokers take more than 1 hour to restart, you likely have some major issues. It was not expected with Strimzi 0.38, neither with 0.42 and it is not expected with 0.47. There was no fix for that, because it is not a known issue, and it does not seem to be an issue affecting other users. So asking if someone resolved an isssue they ddi not know about does not make much sense. I'm also not sure why did you decided to attach your comment to this discussion. Did you tried if your problem is the same? Do you use Cilium? If you do not have the same problem, starting a new discussion and sharing your full configurations and logs would be probbaly the right approach. |
Beta Was this translation helpful? Give feedback.
-
|
I'm using GKE, and cilium is installed. Sorry about posting here, but the initial description has a similar scenario as mine. |
Beta Was this translation helpful? Give feedback.
-
|
Well, it is a discussion which has been more of less resolved. So if you use Cilium, it clearly might be applicable to you. But you can then test yourself whether the previous fix solves it for you or not. If it does not help, you might have a different issue after all. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi!
I found an unexpected behavior that is not clear to me whether it is Strimzi or Kafka related. I'm testing graceful shutdowns of Kafka brokers as they were taking too long to complete in my Kafka deployment (~15 minutes per broker). After some investigation I found that the brokers graceful shutdown was being forcefully aborted by K8s as it was taking longer than default
terminationGracePeriodSeconds. I increasedterminationGracePeriodSecondsto 5 minutes and saw that graceful broker restart reduced to 2-3 minutes (usingkubectl delete <broker>).Now, I did a rolling update of the Kafka brokers but I'm still seeing each broker taking ~15 minutes to restart. In this case, I've tested triggering rolling updates by changing some Kafka broker configurations and also by adding the strimzi.io/manual-rolling-update annotation.
Any idea why a Kafka broker restart via
kubectl deletetakes 2-3 minutes, whereas if the restart is managed by Strimzi it is still taking 15 minutes?Some details about my setup (let me know if more information is needed):
Kafka brokers have the following configuration overrides (everything else is the default from Strimzi):
Observations of what happens when Kafka brokers take 15 minutes to restart during a rolling restart managed by Strimzi:
terminationGracePeriodSecondsand get a SIGKILL, which explains why the longer recovery time during start.Thanks.
Beta Was this translation helpful? Give feedback.
All reactions