Really asymmetric upload and download speeds using QUIC

[SergioLuis]

Hi all!

We write some software that does Remote Procedure Calling using a protocol of our own - no gRPC nor any other known frameworks. This software is written in pure C# and the newest versions of dotnet.

This RPC protocol is transport-agnostic. We default to TCP, but we also support UPD & UDT (in fact, we maintain a completely managed implementation of UDT).

So in order to test QUIC, which we all are really excited about, I rewrote the transport part of the RPC protocol, plugging in the new QUIC APIs in .NET where necessary.

But because the QUIC APIs are still not public in .NET 7 Preview, I built myself from scratch System.Net.Quic.dll from the dotnet/runtime repository and msquic from this one.

These are the arguments I used to build msquic:

./scripts/build.ps1 -Tls openssl -Arch x64 -Config Release

I built msquic both for macOS x64 and ARM64, and for Linux x64 - and got it working perfectly (meaning that the RPC protocol works as expected using QUIC instead of TCP for both Linux to Linux, macOS to macOS, and macOS to Linux, both in x64 environments, ARM64, and mixed ARM64 - I have yet to test this on Windows, but as you can see, I had quite some fun testing this 😄 ...!)

However, the speed test results are not what I expected. I ran some synthetic upload and download tests. Instead of manually tampering with the network QoS, I created a remote VM on Google Cloud (in Sydney) and connected to it from Madrid (Spain).

The software we write has some built-in network speed tests, and that's what I used to compare TCP to QUIC - so that the comparison is as fair as possible.

The tests consist of uploading and downloading a chunk of 128MB of random bytes in packets of 1MB (so, 128 uploads and 128 downloads). Each upload and download test is an iteration, and the test performs four iterations in a row, reusing the same connection for all of them, and using the same (bidirectional) stream for uploading and downloading. These are the results:

QUIC:

iostats quic://XXX.XXX.XXX.XXX:443 --nettotalmb=128 --serveruploadtest --serverdownloadtest --packetdatasize=1 --networkiterations=4
Performing network tests with server: quic://XXX.XXX.XXX.XXX:443. Please wait...

Upload speed   = 21.51 Mbps.   Time uploading 128MB = 47595ms.    Server: quic://XXX.XXX.XXX.XXX:443
Download speed = 6.70 Mbps.  Time downloading 128MB = 152853ms.   Server: quic://XXX.XXX.XXX.XXX:443

Upload speed   = 22.55 Mbps.   Time uploading 128MB = 45401ms.    Server: quic://XXX.XXX.XXX.XXX:443
Download speed = 7.05 Mbps.  Time downloading 128MB = 145291ms.   Server: quic://XXX.XXX.XXX.XXX:443

Upload speed   = 22.67 Mbps.   Time uploading 128MB = 45161ms.    Server: quic://XXX.XXX.XXX.XXX:443
Download speed = 6.28 Mbps.  Time downloading 128MB = 163068ms.   Server: quic://XXX.XXX.XXX.XXX:443

Upload speed   = 22.69 Mbps.   Time uploading 128MB = 45130ms.    Server: quic://XXX.XXX.XXX.XXX:443
Download speed = 6.37 Mbps.  Time downloading 128MB = 160831ms.   Server: quic://XXX.XXX.XXX.XXX:443

SSL over TCP:

iostats ssl://XXX.XXX.XXX.XXX:8787 --nettotalmb=128 --serveruploadtest --serverdownloadtest --packetdatasize=1 --networkiterations=4
Performing network tests with server: ssl://XXX.XXX.XXX.XXX:8787. Please wait...

Upload speed   = 11.89 Mbps.   Time uploading 128MB = 86105ms.    Server: ssl://XXX.XXX.XXX.XXX:8787
Download speed = 18.14 Mbps. Time downloading 128MB = 56442ms.    Server: ssl://XXX.XXX.XXX.XXX:8787

Upload speed   = 12.39 Mbps.   Time uploading 128MB = 82676ms.    Server: ssl://XXX.XXX.XXX.XXX:8787
Download speed = 18.09 Mbps. Time downloading 128MB = 56593ms.    Server: ssl://XXX.XXX.XXX.XXX:8787

Upload speed   = 12.41 Mbps.   Time uploading 128MB = 82539ms.    Server: ssl://XXX.XXX.XXX.XXX:8787
Download speed = 18.08 Mbps. Time downloading 128MB = 56639ms.    Server: ssl://XXX.XXX.XXX.XXX:8787

Upload speed   = 12.37 Mbps.   Time uploading 128MB = 82778ms.    Server: ssl://XXX.XXX.XXX.XXX:8787
Download speed = 18.35 Mbps. Time downloading 128MB = 55803ms.    Server: ssl://XXX.XXX.XXX.XXX:8787

As you can see, I have QUIC running on port 443, in case someone was doing some shenanigans in the routing by favoring UDP traffic over HTTP/3's default port.

Some things to unpack here:

• QUIC's upload speed (client to server) outperforms TCP.
• QUIC's download speed is really, really low compared to the upload speed, and even compared to its TCP counterpart.
• Despite reusing the same connection over the entire test, I can't see QUIC speed ramping up over time. This is something we can see when doing the same tests over UDT. Did it already reach maximum network bandwidth usage in the first iteration...?

I tested this over two different networks in case there was some firewall in the middle throttling UDP packets in some way. The next thing I plan to do is repeat these tests within Google infrastructure (both client and server, in different regions) and check if there are any differences.

Any light you can shed on this will be more than welcome :)

[nibanks]

First off, I want to say thanks for trying MsQuic and thanks for all the detailed info. Unfortunately, our most detailed diagnostics are currently only available when running on Windows. So if you could try that out and collect Basic.Full traces and share them, I can take a look.

[SergioLuis]

Thank you for your prompt reply and your work with MsQuic!

If that's the case, I'll prepare a Windows-only scenario and get back here with some more results :)