Multiple DCUTR connections between two peers

[geotro]

I'm a beginner libp2p user, specifically using the rust version.

Is it possible to start multiple DCUTR between two peers to achieve more than one direct connection behind NAT?

Thx

[thomaseizinger]

In principle, yes. You might need to open the connections sequentially though, not sure if it will work in parallel. @mxinden can likely answer this question in more detail.

Have you tried it already? Are you running in an error / some kind of problem?

[dariusc93]

if youre doing it with 2 instances of libp2p behind both nat that should work fine (ive done it with testing a few projects), although I never tried it with multiple connections to the same peer (assuming this is what youre implying) though more information on your use case might be helpful.

[mxinden]

@geotro would you mind expanding on why you want more than one connection between the same two peers?

[geotro]

@thomaseizinger Must admit I didn't try it yet, will do soon and revert.

@mxinden I want to do this in order to be sure I can utilize all the available bandwidth between two nodes

[geotro]

The use case is transferring large amounts of data, as quickly as possible, between two peers. Experience shows that a single TCP connection cannot usually saturate the available bandwidth over the internet, probably due to congestion control algorithms, on local networks it's a different story of course.

We did achieve pretty good single connection data rates with libp2p between two peers in different countries, but still far from using most of the bandwidth available to both peers.

The next step is to create as many parallel connections as necessary to achieve that.

[rkuhn]

This report is interesting in that it matches my experiments done in 2005 between research sites (TUM and CERN), where routing was across the GEANT network, so should arguably have been optimised for this use-case. The endpoints had GigE fiber connection more or less directly to the backbone.

One thing I also learnt back then is that TCP over such an ethernet/STM setup with a half dozen routers yields an undetected data corruption rate of the order of 1 per TB, which is “handled” by dropping the connection on the higher level. The lesson is that higher level segmentation should take this into account — it seems difficult to imagine that libp2p could grow the ability to recover from data corruption while keeping the same connection (it would be much easier when using UDP→QUIC).

[mxinden]

yields an undetected data corruption rate of the order of 1 per TB

😮 is this still the case today? What was the root cause?

[mxinden]

The use case is transferring large amounts of data, as quickly as possible, between two peers. Experience shows that a single TCP connection cannot usually saturate the available bandwidth over the internet, probably due to congestion control algorithms, on local networks it's a different story of course.

We did achieve pretty good single connection data rates with libp2p between two peers in different countries, but still far from using most of the bandwidth available to both peers.

The next step is to create as many parallel connections as necessary to achieve that.

Cool. I am curious what your experiments show. Would you mind posting the results here once you gathered the data? Likely useful for others as well. @geotro.

[rkuhn]

😮 is this still the case today? What was the root cause?

Well, ethernet frames have a bit error rate, TCP checksum is only 16bit, so I’d say something like this is to be expected.

[mxinden]

Is it possible to start multiple DCUTR between two peers to achieve more than one direct connection behind NAT?

This is not possible today. libp2p-dcutr will not attempt a second hole punch in case a direct connection already exists. Forcing libp2p-dcutr to do so manually would probably be a small patch-set only. Though, by the contract of the protocol, this needs to be done by the listener side of the relayed connection.

[geotro]

@mxinden sure, would be happy to test and report back. Could you point me in the right direction of what needs to be modified to allow this?

[mxinden]

You would want to add a method to the dcutr Behaviour which does the same as when an incoming relayed connection is handled:

rust-libp2p/protocols/dcutr/src/behaviour.rs

Lines 108 to 133 in 5906140

	// TODO: Try dialing the remote peer directly. Specification:
	//
	// > The protocol starts with the completion of a relay connection from A to B. Upon
	// observing the new connection, the inbound peer (here B) checks the addresses
	// advertised by A via identify. If that set includes public addresses, then A may
	// be reachable by a direct connection, in which case B attempts a unilateral
	// connection upgrade by initiating a direct connection to A.
	//
	// https://github.com/libp2p/specs/blob/master/relay/DCUtR.md#the-protocol
	self.queued_actions.extend([
	ActionBuilder::Connect {
	peer_id: *peer_id,
	attempt: 1,
	handler: NotifyHandler::One(*connection_id),
	},
	NetworkBehaviourAction::GenerateEvent(
	Event::InitiatedDirectConnectionUpgrade {
	remote_peer_id: *peer_id,
	local_relayed_addr: match connected_point {
	ConnectedPoint::Listener { local_addr, .. } => local_addr.clone(),
	ConnectedPoint::Dialer { .. } => unreachable!("Due to outer if."),
	},
	},
	)
	.into(),
	]);

This method you can then call from the Swarm::behaviour_mut() to trigger the next DCUtR process.

[geotro]

Thank you. Hoping to be able to test this soon. Will report back with the results.