Skip to content

Add ability to dynamically change cluster membership #81

@btmorr

Description

@btmorr

See Section 6 of the extended Raft paper (https://raft.github.io/raft.pdf)

In order to add or replace nodes from a cluster, members have to be able to agree about the new roster and transition to it gracefully. Roughly, the tasks involved are:

  • Create ClusterConfig and JointConfig log record type (as differentiated from current log records)
  • Change the type of Log Entries to a union of LogRecord, ClusterConfig, and JointConfig
  • Add an endpoint or similar mechanism that an administrator can use to initiate a configuration change (such as POST /configure with a body that includes the new membership)
  • On start, a node relies on environment variables to determine valid cluster membership, but as soon as a leader is elected it should create a ClusterConfig that matches the configuration from the environment variable and commit it to the log
  • When a change in cluster membership is initiated, the leader uses the requested new membership to create a JointConfig with an "old" field describing the current configuration, and a "new" field describing the requested change. This is added to the log and sent out for append and then commit as with regular LogRecords, except that it must be appended by a majority of both the old and combined membership before being committed
  • As soon as the JointConfig is committed, the leader may return a success response to the admin endpoint, since at this point even if the leader fails the new configuration will succeed
  • During the JointConfig period, any LogAppend requests have the same acceptance requirement as the JointConfig (majority of both the old and combined membership)
  • As soon as possible after a JointConfig is appended, the leader should create a ClusterConfig entry with the new membership only, append and send it, and then commit it (this time only requiring the members of the new configuration). After this point nodes that are not members of the new configuration no longer participate.

Note:

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions