How should we go about merging major changes into the codebase

[alexpyattaev]

Reasoning

We have two relatively big changes coming to the validator:

• Multihoming changes
• Alpenglow

Both of these can be expected to be fairly intrusive and not obviously backwards compatible.

Question

Which transition policy do we adopt for solana crates?

• Prioritize API quality and break backwards compatibility (on a major version release)
• Prioritize backwards compatibility (provide ways to call into old functions with conventional deprecation warnings)

Scope of the problem: Multihoming

Non-exhaustive list of things that will have to be changed:

• Gossip: Sockets, NodeConfig
• TVU: new, TvuSockets, RetransmitStage::new,
• Repair: new + maybe other things?
• AncestorRepair: new + maybe other things?
• Streamer: APIs touching sockets to support rebinding
• TPU: new Streamers
• XDP: binding and re-binding
• Command-line args: bind-address, deprecate gossip-host
• AdminRpcRequestMetadataPostInit: access to Sockets

Detailed Changes

Gossip

NodeConfig

Struct Fields	Old Type	New Type	Description
gossip_addr	SocketAddr	N/A	Remove
advertised_ip	N/A	IpAddr	New field representing what IP is advertised to the cluster in gossip
gossip_port	N/A	u16	New field representing the gossip port advertised to the cluster
bind_ip_addr	IpAddr	N/A	Remove
bind_ip_addrs	N/A	BindIpAddrs	New field representing the IP addresses the node can bind to

pub struct BindIpAddrs {
    primary: IpAddr,
    others: Vec<IpAddr>,
}

• NodeConfig refactor including gossip_addr removal and advertised_ip, gossip_port addition was merged into master: Refactor NodeConfig to improve clarity #6553
• bind_ip_addr: IpAddr -> bind_ip_addrs: BindIpAddrs PR: Multihoming: add multiple bind addresses. #6522

Sockets

Struct Fields	Old Type	New Type	Description
gossip	UdpSocket	Multihomed<AtomicUdpSocket>	"Hotswappable socket" -> swap socket at runtime.
tvu	Vec<UdpSocket>	Multihomed<UdpSocket>	Tvu Ingress: New struct to hold all bindable sockets across all interfaces
retransmit_socket	Vec<UdpSocket>	Multihomed<AtomicUdpSocket>	Tvu Egress: New struct to hold all bindable, hotswappable sockets across all interfaces
repair	UdpSocket	Multihomed<AtomicUdpSocket>	Repair Client Socket: Hold all bindable, hotswappable sockets across all interfaces.
serve_repair	UdpSocket	Multihomed<UdpSocket>	Repair Server Socket: Hold all bindable sockets across all interfaces.

TVU

TvuSockets

Struct Fields	Old Type	New Type	Description
fetch	Vec<UdpSocket>	Multihomed<UdpSocket>	We'll bind to all sockets on all interfaces. Could be a Vec<UdpSocket>, but I want to make it clear we are binding on multiple interfaces
retransmit	Vec<UdpSocket>	Multihomed<AtomicUdpSocket>	New struct to hold all bindable, hotswappable sockets across all interfaces
repair	UdpSocket	Multihomed<AtomicUdpSocket>	Repair Client Socket: Hold all bindable, hotswappable sockets across all interfaces
ancestor_hashes_requests	UdpSocket	Multihomed<UdpSocket>	We'll bind to all sockets on all interfaces. But this socket will now only consume requests. Could be a Vec<UdpSocket>, but I want to make it clear we are binding on multiple interfaces
ancestor_hashes_requests_client	UdpSocket	Multihomed<AtomicUdpSocket>	For the client side, the socket needs to be hotswappable.

Tvu::new(...)

Function args	Old Type	New Type	Description
sockets	TvuSockets	TvuSockets	New version of TvuSockets as defined above

RetransmitStage::new(...)

Function args	Old Type	New Type	Description
retransmit_sockets	Arc<Vec<UdpSocket>>	Arc<Multihomed<AtomicUdpSocket>>	New struct to hold all bindable, hotswappable sockets across all interfaces

Repair Client

ShredFetchStage::new(...)

Function args	Old Type	New Type	Description
repair_socket	Arc<UdpSocket>	Arc<Multihomed<AtomicUdpSocket>>	all bindable, hotswappable sockets across all interfaces

WindowService::new(...)

Function args	Old Type	New Type	Description
repair_socket	Arc<UdpSocket>	Arc<Multihomed<AtomicUdpSocket>>	all bindable, hotswappable sockets across all interfaces

RepairService::new(...)

Function args	Old Type	New Type	Description
repair_socket	Arc<UdpSocket>	Arc<Multihomed<AtomicUdpSocket>>	all bindable, hotswappable sockets across all interfaces

Repair Server

ServeRepairService::(...)

Function args	Old Type	New Type	Description
repair_socket	UdpSocket	Multihomed<UdpSocket>	all bindable sockets across all interfaces

AncestorHashesService

Uses the same UDP socket for sending/receiving. This will get updated so that we bind to all interfaces on ingress but then use a hotswappable socket on egress.

WindowService::new(...)

Function args	Old Type	New Type	Description
ancestor_hashes_socket	Arc<UdpSocket>	Arc<Multihomed<UdpSocket>>	one bindable socket for each interface. Ingress only
ancestor_hashes_socket_client	N/A	Arc<Multihomed<AtomicUdpSocket>>	one bindable, hotswappable socket for each interface. Egress only

RepairService::new(...)

Function args	Old Type	New Type	Description
ancestor_hashes_socket	Arc<UdpSocket>	Arc<Multihomed<UdpSocket>>	one bindable socket for each interface. Ingress only
ancestor_hashes_socket_client	N/A	Arc<Multihomed<AtomicUdpSocket>>	one bindable, hotswappable socket for each interface. Egress only

AncestorHashesService::new(...)

Function args	Old Type	New Type	Description
ancestor_hashes_request_socket	Arc<UdpSocket>	Arc<Multihomed<UdpSocket>>	one bindable socket for each interface. Ingress only
ancestor_hashes_request_socket_client	N/A	Arc<Multihomed<AtomicUdpSocket>>	one bindable, hotswappable socket for each interface. Egress only

TPU

TpuSockets

Struct Fields	Old Type	New Type	Description
transactions	Vec<UdpSocket>	Multihomed<UdpSocket>	UDP, not QUIC. Bind to all interfaces on ingress. Could be a Vec<UdpSocket>, but I want to make it clear we are binding on multiple interfaces
transaction_forwards	Vec<UdpSocket>	Multihomed<UdpSocket>	UDP, not QUIC. Bind to all interfaces on ingress. Could be a Vec<UdpSocket>, but I want to make it clear we are binding on multiple interfaces
vote	Vec<UdpSocket>	Multihomed<UdpSocket>	UDP, not QUIC. Bind to all interfaces on ingress. Could be a Vec<UdpSocket>, but I want to make it clear we are binding on multiple interfaces
broadcast	Vec<UdpSocket>	Multihomed<AtomicUdpSocket>	UDP. Service to broadcast messages. These sockets must be bindable and hotswappable
transactions_quic	Vec<UdpSocket>	tbd	QUIC
transactions_forwards_quic	Vec<UdpSocket>	tbd	QUIC
vote_quic	Vec<UdpSocket>	tbd	QUIC
vote_forwarding_client	UdpClient	tbd	QUIC

TPU UDP Ingress

Bind to all interfaces on ingress. Besides TvuSockets update, all functions can remain the same

BroadcastStage

BroadcastStage::new(...)

Function args	Old Type	New Type	Description
sock	Arc<Vec<UdpSocket>>	Arc<Multihomed<AtomicUdpSocket>>	Bindable, hotswappable sockets

broadcast_shreds(...)

Function args	Old Type	New Type	Description
s	&UdpSocket	&AtomicUdpSocket	Bindable, hotswappable socket

TPU Streamer

Uses TpuSockets::transactions_quic
Have to take a different approach here. We cannot hotswap QUIC Endpoints like we can with UdpSockets

spawn_server_multi(...)

• In streamer/src/quic.rs
• TBD

spawn_server_multi(...)

• In streamer/src/nonblocking/quic.rs
• TBD

Streamer

Need to support hotswapping sockets. We can pretty easily avoid breaking the public api. We just need to define two new functions along side the current streamer::receiver() and streamer::responder()

streamer::receiver(...)

Create a new function called streamer::receiver_atomic(...) -> wrapper around recv_loop(...)

• Once we get GossipService and Repair switched over to new receiver_atomic(...), we can get rid of receiver(...)

streamer::responder(...)

Create a new function called streamer::responder_atomic(...) -> wrapper around responder_loop(...)

• Once we get GossipService and Repair switched over to new responder_atomic(...), we can get rid of responder(...)

CLI Args

--bind-address

Update --bind-address to accept multiple arguments. First --bind-address is primary. Following ones are backups

--gossip-host

Deprecate --gossip-host in favor of --bind-address. This removes a validator operator's ability to run a node behind a NAT, but will avoid confusion when starting a validator. We don't need the bind address to be different than the IP advertised to the cluster over gossip.
NOTE: already merged into master for v3.0 release

AdminRpc

AdminRpcRequestMetadataPostInit

Struct Fields	Old Type	New Type	Description
sockets	N/A	Option<Arc<Sockets>>	New field used for swapping sockets when switching from one IP to another. This is the same Sockets that stores all of the socket bindings and referenced in the validator as node.sockets

Scope of the problem: Alpenglow

Non-exhaustive list of things that will have to be changed:

• Everything that refers to tower BFT (replay, blockstore, TVU)
• Gossip: Sockets