Update the deployment strategy for CSI deployments

[black-dragon74]

Describe the bug

When the deployments managed by the operator are updated and the pod placement in the same node,
it leads to the new pods always being in the pending state due to the way we have defined the updateStrategy and podAntiAffinity rules.

The strategy of maxSurge > 0 instructs the scheduler to create new pods before it evicts older ones. The new pods will never be in running state as the anti-affinity rules prevent that.

As a result, we reach a deadlock. The new pods can't start because of the old pods, and the old pods won't be removed until the new one starts.

To remedy this, we can have maxSurge: 0 and maxUnavailable: 1 which can enable us to do in-place rollouts, one by one.

[black-dragon74]

/assign

[github-actions]

Thanks for taking this issue! Let us know if you have any questions!

[Madhu-1]

How about setting the deploymentStrategy to Recreate which is good for ceph-csi RollingUpdate doesnt really make sense for ceph-csi deployment as we have leaderelection

[black-dragon74]

That would be quite disruptive (all pods will be deleted first) and we would have downtime (albeit a few seconds). In place rollouts make the most sense (delete one, recreate it, repeat for remaining replicas).

as we have leaderelection

We'll lose the lease holder so it will have no effect :(

[Madhu-1]

Recreate is the strategy we are using from many years(almost 5) by default in Rook

[Madhu-1]

leaderelection timeout is i think 2.5 minutes, if you loose the leader even in case of the RolingUpdate its the same no leader will be present to serve the request. the Recreate is better strategy as it will bring back the leader earlier than rolingupdate

[black-dragon74]

Recreate is the strategy we are using from many years(almost 5) by default in Rook

Intrigued by why we chose rolling updates for csi-op. Recreate is obviously much simpler and straightforward

[leelavg]

bring back

• follower becoming leader is a better bet than a newer pod to get scheduled and ready, in current context the statement may not hold much prominence but just saying.

[Madhu-1]

• follower becoming leader is a better bet than a newer pod to get scheduled and ready, in current context the statement may not hold much prominence but just saying.

The follower will become only after the leaderelection timeout which is i think more than a minute or 2 minutes not immediately.

[sebhoss]

for maxSurge: 0 you probably want to wait for kubernetes/kubernetes#107920

[Madhu-1]

for maxSurge: 0 you probably want to wait for kubernetes/kubernetes#107920

Thanks @sebhoss for the link, this is long way. we should look for alternative like Recreate which is opt for ceph-csi-operator