Interaction and ordering of Kademlia, Identfiy and AutoNAT behaviors

[b-zee]

Assume these three behaviors: Kademlia, Identify and AutoNAT. Adding peers to the Kademlia routing table happens based on the Info we exchange (out-of-band) with Identify. However, we only want to add the proper address of a peer. For a peer to first determine its public address, it uses AutoNAT.

The addresses that get broadcast with Info include our local addresses, which can be ignored by our peers. But only after we have established confidently our external address the Info will also include that address. We can push the Info again when we see a change in NAT status. However, this status doesn't list the confidence. And there is no swarm event that I know of that we can use to see if external addresses were added or the scores have changed.

Am I going about this in the wrong way? In essence, what I am trying to prevent is a node behind NAT trying to join the Kademlia network. I am assuming that will be detrimental to the health of that network, as the node can not be dialed (assuming we are not yet using relays).

[thomaseizinger]

Kademlia will only ever add a node to the routing table if it successfully dialed that node:

rust-libp2p/protocols/kad/src/behaviour.rs

Lines 2050 to 2057 in 541eba3

	// The remote's address can only be put into the routing table,
	// and thus shared with other nodes, if the local node is the dialer,
	// since the remote address on an inbound connection may be specific
	// to that connection (e.g. typically the TCP port numbers).
	let address = match endpoint {
	ConnectedPoint::Dialer { address, .. } => Some(address),
	ConnectedPoint::Listener { .. } => None,
	};

The addresses we receive from identify will be use to attempt to connect to a node if we don't have a connection yet:

rust-libp2p/protocols/identify/src/behaviour.rs

Lines 330 to 343 in 541eba3

	fn handle_pending_outbound_connection(
	&mut self,
	_connection_id: ConnectionId,
	maybe_peer: Option<PeerId>,
	_addresses: &[Multiaddr],
	_effective_role: Endpoint,
	) -> Result<Vec<Multiaddr>, ConnectionDenied> {
	let peer = match maybe_peer {
	None => return Ok(vec![]),
	Some(peer) => peer,
	};
	Ok(self.discovered_peers.get(&peer))
	}

which gets populated by:

rust-libp2p/protocols/identify/src/behaviour.rs

Lines 286 to 293 in 541eba3

	handler::Event::Identified(mut info) => {
	// Remove invalid multiaddrs.
	info.listen_addrs
	.retain(|addr| multiaddr_matches_peer_id(addr, &peer_id));
	// Replace existing addresses to prevent other peer from filling up our memory.
	self.discovered_peers
	.put(peer_id, info.listen_addrs.iter().cloned());

Does this help?

[b-zee]

Amazing, thanks for the pointers.

Kademlia will only ever add a node to the routing table if it successfully dialed that node

Does this mean that clients are attempted to be dialed? (In case of using QUIC or TCP with port reuse, that might mean that those clients are actually successfully dialed. (Even assuming NAT, as the hole will be punched because the port is reused.))

[thomaseizinger]

Kademlia will only ever add a node to the routing table if it successfully dialed that node

Does this mean that clients are attempted to be dialed?

If you look for usages of ToSwarm::Dial in the kademlia protocol, you'll find the following three:

1. We are working on a query and need to send a message to a peer we are not yet connected to:

rust-libp2p/protocols/kad/src/behaviour.rs

Lines 2402 to 2404 in 3fde132

	self.queued_events.push_back(ToSwarm::Dial {
	opts: DialOpts::peer_id(peer_id).build(),
	});

2. We evicted an entry from the routing table because a bucket was full, thus we test connectivity to that node:

rust-libp2p/protocols/kad/src/behaviour.rs

Lines 574 to 576 in 3fde132

	self.queued_events.push_back(ToSwarm::Dial {
	opts: DialOpts::peer_id(disconnected.into_preimage()).build(),
	});

rust-libp2p/protocols/kad/src/behaviour.rs

Lines 1212 to 1216 in 3fde132

	self.queued_events.push_back(ToSwarm::Dial {
	opts: DialOpts::peer_id(disconnected.into_preimage())
	.build(),
	})
	}

[b-zee]

Thanks again, sorry if that was a bit low effort. I am reading through the code myself now, and getting the hang of tracing these things.

I think that mostly clears it up for now. I will try and see if I can get these basic behaviors interacting with each other.

[mxinden]

Thanks @thomaseizinger 👍

I will try and see if I can get these basic behaviors interacting with each other.

I am sorry for the trouble @b-zee you are facing with some of this. Note that the libp2p-autonat implementation is relatively new and not much tested in the wild.

[thomaseizinger]

Thanks again, sorry if that was a bit low effort. I am reading through the code myself now, and getting the hang of tracing these things.

No worries at all :)

It is a big codebase, feel free to ask if more questions arise!

[b-zee]

@thomaseizinger

You mentioned that peers only get added when they are dialed by us:

Kademlia will only ever add a node to the routing table if it successfully dialed that node

Though, this seems to be overwritten of course if we manually call add_address. This was something I overlooked in our codebase.

Now, how would we approach leaving this to the Kademlia protocol. Does Kademlia add all nodes that support the kad protocol and are dialed by us? Or is this only in case this dial was part of a query? E.g. we called add_address manually based on the identify protocol to see if it's a peer with a certain user agent string, but we'd still only want to add them if we can successfully dial them.

[thomaseizinger]

Now, how would we approach leaving this to the Kademlia protocol. Does Kademlia add all nodes that support the kad protocol and are dialed by us? Or is this only in case this dial was part of a query? E.g. we called add_address manually based on the identify protocol to see if it's a peer with a certain user agent string, but we'd still only want to add them if we can successfully dial them.

I am working on this being automatic, yes: #3877

Should be done soon (hopefully) :)

[b-zee]

Cool, thanks for replying!

Does this mean that a successful dial to a peer will add the peer to the routing table (assuming the right protocol etc)? That is, no add_server needed?

Is there a workaround for now to achieve this behavior? I don't think we can see whether we are the Dialer, so it's only possible to keep track of this ourselves using ConnectionEstablished events?

[thomaseizinger]

Cool, thanks for replying!

Does this mean that a successful dial to a peer will add the peer to the routing table (assuming the right protocol etc)? That is, no add_server needed?

Yes!

Is there a workaround for now to achieve this behavior? I don't think we can see whether we are the Dialer, so it's only possible to keep track of this ourselves using ConnectionEstablished events?

The ConnectedPoint exposes dialer/listener :)

[b-zee]

Yeah, so it's something we'd track in a list manually. I think that's a solid workaround for now unless you know of something simpler.

Thanks a lot for the help! Great work you guys are doing!