Can network isolation cause brain crack?

[Radiancebobo]

Suppose we k8s have 10 worker nodes, among which three worker nodes (node A, node B, node C) are running zookeeper. Suddenly, one day, the three nodes are isolated from each other. A and B can communicate, but C cannot establish communication with A and B. According to the principle of more than half, A and B are still a ZK cluster.

However, at this time, because k8s cannot communicate with the node A and node B, it will run zookeeper again on other nodes (assuming node E and node F), so C, E and F are also a ZK cluster.

Isn't that a brain crack? Or this assumption doesn't come out at all?

[scholzj]

I'm not sure it is that simple. You would for example normally use RWO storage for ZooKeeper. That should not allow a to start new brokers on E and F while A and B is still running and holds the storage. In the past, when I had a similar issue, the nodes A and B were either alive enough to shutdown the pods and unmount storage or they had to be stopped on the infrastructure level -> which freed the storage and killed the pods in any case.