Ensure all nodes in the cluster see all as UP as a prerequisite for a topology change and each step of a rolling restart

[rzetelskik]

Description

As a result of investigating issues found in multi-datacenter bootstrap, we concluded that Operator does not currently follow topology change / rolling restart procedures. It should always ensure that all nodes in the cluster see all nodes as UP before progressing. See scylladb/scylladb#25410 (comment).

Despite scylladb/scylladb@28c0a27 alleviating the issue on the happy path, it has become very prevalent in multi-datacenter E2E tests as the DCs are rolling restarted throughout the bootstrap of the entire cluster.

Acceptance criteria

• A mechanism ensuring all nodes in the cluster see all as UP as a prerequisite for the following operations exists
  • a topology change
  • each step of a rolling restart

Notes

For more context, see:

• Frequent failures in multi-datacenter bootstrap: the topology coordinator rejected request to join the cluster: request canceled because some required nodes are dead scylladb#25410
• RFE: Add /readyz and /healthz probes scylladb#8275
• Add endpoint to REST API to indicate whether this node is fully up scylladb#16763
• During rollout restart operator doesn't wait until previously restarted pod becomes part of a Scylla cluster #1077

A predicted implementational issue is that K8s uses the same readiness probe to determine if:
a) the node is ready to serve traffic,
b) the node can be brought down / added; the entire cluster is ready for a topology change / rolling restart if all nodes are ready.

It seems like the two concepts should be separate, and we should try to implement it somehow. We currently implement the readiness probe to only answer a) by checking if a node 1. considers itself UP (by GET /gossiper/endpoint/live/ ) and 2. has local CQL port open (by GET /storage_service/native_transport). The other point is not addressed by the readiness probe - addressing it there would result in availability issues.

There is nothing preventing the probe from talking to other nodes. Although, if we ask the entire cluster for the status of a node for every single node with a set frequency, it will cause a lot of noise. This may become a huge problem in clusters with high numbers of nodes or in multi-datacenter clusters where intercluster node-to-node communication may be expensive.

/kind bug
/priority important-soon

[mykaul]

See https://github.com/scylladb/scylla-ccm/blob/8529a7194ae88e34dbc0d17dddb2a76504dcc02e/ccmlib/scylla_node.py#L1473

[mflendrich]

Thanks @rzetelskik, this condenses a good chunk of research and multiple attempts at a solution into something actionable, which is a big deal.

So as not to lose it, let's speculate less and make it an actionable task (the scope of this issue is not clear enough from the issue description)
Please rephrase as a call for action ("Ensure all nodes in the cluster see all as UP as a prerequisite for $operation")

where I'm not sure that "rolling restart" is the right (part of) "operation". It is each step of the rolling restart, right?

Please also define the acceptance criteria.

Also we discussed that this possibly addresses what appears to be the root cause of #1077 (but so much happened in that issue that the clue of it may be lost).

[rzetelskik]

Please rephrase as a call for action ("Ensure all nodes in the cluster see all as UP as a prerequisite for $operation")

where I'm not sure that "rolling restart" is the right (part of) "operation". It is each step of the rolling restart, right?

Yes, clarified it.

Please also define the acceptance criteria.

Added and structured the description to be a bit more readable. Please let me know if you'd like to see more items there.

Also we discussed that this possibly addresses what appears to be the root cause of #1077 (but so much happened in that issue that the clue of it may be lost).

Yup, I mentioned it in #1077.