Unexpected connection closed

[tthebst]

Hey,

I'm trying to implement a file syncing protocol with the request-response behavior. The file is split into chunks (of size 100kb-1Mb) and each chunk is fetched with an RPC. Similar to the file-sharing example I have a client that sends fetch chunk requests to the swarm. The number of concurrent requests is limited with a semaphore to 50 per peer. Now the issue is that when doing the fetching I see some spontaneous disconnects: On the side that fetches the chunk I get a timeout error:

Some(Handler(Right(Timeout))) (which is the request-response handler)

and on the other side, presumably the consequence:

Connection closed to peer QmeJaswmFURBu7wexRcYuvXgp3JLqNqRXP8L1eroSHaNR8. Connections left 1. Cause: Some(IO(Custom { kind: Other, error: YamuxError(Closed) }))"

From reading the libp2p code it seems to me that the stream negotiation for the request-response times out but I find this a little surprising since I would assume 10s is enough.

Things I tried:

• Use V1Lazy: I think this improved things a bit but it did not resolve it.
• Try to not immediately initiate 50 requests and do a slow start. This helps.

Is there anything I can do here besides reducing the number of requests?

I use yamux with a TCP transport.

[mxinden]

👋 sorry for the late reply.

I am guessing that you are hitting the max_negotiating_inbound_streams limit. This is a DOS defense mechanism. The remote node needs to allocate memory for each inbound stream, thus we need to limit the number of in-flight streams on the inbound side.

rust-libp2p/swarm/src/lib.rs

Lines 1692 to 1704 in 7c85f92

	/// The maximum number of inbound streams concurrently negotiating on a
	/// connection. New inbound streams exceeding the limit are dropped and thus
	/// reset.
	///
	/// Note: This only enforces a limit on the number of concurrently
	/// negotiating inbound streams. The total number of inbound streams on a
	/// connection is the sum of negotiating and negotiated streams. A limit on
	/// the total number of streams can be enforced at the
	/// [`StreamMuxerBox`](libp2p_core::muxing::StreamMuxerBox) level.
	pub fn max_negotiating_inbound_streams(mut self, v: usize) -> Self {
	self.pool_config = self.pool_config.with_max_negotiating_inbound_streams(v);
	self
	}

For the sake of debugging only, would you mind setting that number to usize::MAX?

libp2p-request-response does not yet enforce backpressure between its NetworkBehaviour implementation and its ConnectionHandler. Note that this is on the outbound side only and thus not a DOS attack. For now, you would need to pace your requests on the outbound side. We can fix this properly similar to how we plan to fix it for libp2p-kad, see #3710. That would remove all need for pacing on the outbound side.

In case this is indeed your issue, i.e. usize::MAX resolves it, I am happy to prioritize the proper fix within libp2p-request-response.

In case my guess is wrong, would you mind providing more logs, a pointer to the code that you are using?

[tthebst]

I tried setting max_negotiating_inbound_streams to usize::MAX and it didn't help. I will try to collect some logs to see what is happening.