Stream receive window setting in quinn is changed by libp2p from 1.25MiB to 10MiB

[b-zee]

With our Autonomi test networks we're receiving some feedback by community members that are suggesting their routers get overwhelmed/saturated in terms of UDP traffic. This might partly be due to a current problem we're having with too many open connections, but it's plausible another configuration plays a role here: the max_stream_data setting for libp2p-quic.

The doc for this setting is:

Max unacknowledged data in bytes that may be sent on a single stream.

Also explained by quinn (stream_receive_window) as:

Maximum number of bytes the peer may transmit without acknowledgement on any one stream before becoming blocked.

This should be set to at least the expected connection latency multiplied by the maximum desired throughput. Setting this smaller than receive_window helps ensure that a single stream doesn’t monopolize receive buffers, which may otherwise occur if the application chooses not to read from a large stream for a time while still requiring data on other streams.

In quinn this value is by default set to 1.250.000 (1.25 MiB):

const EXPECTED_RTT: u32 = 100; // ms
const MAX_STREAM_BANDWIDTH: u32 = 12500 * 1000; // bytes/s
                                                // Window size needed to avoid pipeline
                                                // stalls
const STREAM_RWND: u32 = MAX_STREAM_BANDWIDTH / 1000 * EXPECTED_RTT;

But libp2p-quic sets this to 10.000.000 (10MiB). I'm curious if there's a rationale behind increasing the value from 1.25MiB to 10MiB. This seems like a significant change.

I'm also curious if anyone perhaps knows if this can play a role in our network.

Thanks in advance!

[elenaf9]

@kpp as the author of #3454, do you remember why the decision was made to override the the quinn default here?

[b-zee]

Maybe it's been taken from this thread? libp2p/go-libp2p-quic-transport#79

[jxs]

Hi @b-zee thanks for your report! @nazar-pc probably this also relates to your problem?
@b-zee if you use the default quinn value does your reported situation improve? And what impact does it have on network performance?
@nazar-pc would you be willing to test quic connections with quinn's default value?
Thanks!

[nazar-pc]

I don't have much interest in QUIC for DSN purposes, it is unlikely to work with our scenario of opening/closing large number of connections rapidly. Please prove me wrong, but I'm reluctant to do any large-scale testing at this point, very low on my priority list.

[Voim4]

For a context, here is a link to a post in Autonomi forum detailing a view of the situation by one expert.

It is a very long post though, so be warned! :)

https://forum.autonomi.community/t/discussion-on-low-level-data-flow-and-home-networks-found-solution-to-allow-4mb-max-chunk-size-as-smooth-as-1-2-mb-its-a-setting-in-quic/40540?

[elenaf9]

Wow, impressive blogpost and long discussion!

I read through the initial post. Two things that I would like to note (sorry if that has already been discussed - I didn't read through the whole thread):

1. max_stream_data doesn't apply to a connection, but instead to a stream on the connection. One connection may have multiple streams, in which case the max allowed unacknowledged data on the connection is the # streams X max_stream_data. max_connection_data applies an additional maximum to the whole connection.
   But this detail is probably not super important for you if you have a smaller number of streams.

Now with QUIC there is the possibility to do flow control on the data being uploaded to another node. Since the value is 10MB currently there would be up to 10MB sent in one go.

2. max_stream_data is not the maximum outgoing data that we can sent unacknowledged on a stream. Instead it is the max unacknowledged data that we allow the remote peer to sent to us, before they must stop sending (else it's a protocol violation).
   See also the QUIC RFC about Flow Control.
   If I understood the post correctly, it mostly talks about outgoing packets and the rate at which they are sent. They are limited by what the remote has set as their max_stream_data not by our own max_stream_data.

   Quinn has an additional config option send_window (not part of the official QUIC RFC) that sets a max for outgoing unacknowledged packages. The default for it is 8*1.25MB, which we don't change in rust-libp2p.

[Voim4]

Thanks Elena, for taking time to look into this! It seems it has already brought some more clarity to the issue. Much appreciated.