Only ~100 kbps throughput in secnetperf test

[furrio]

I want to evaluate the maximum throughput of peer-to-peer transmission. The NIC I'm using is from Mellanox CX-6 series. The port rate is 100Gbps, but the throughput of quic is only around 100kps, for both local and peer-to-peer testing.

I run the local test as follow:

In section 1:

PS /home/ceni-a1/msquic/artifacts/bin/linux/x64_Debug_openssl> ./secnetperf -exec:maxtput
Started!

In section 2:

PS /home/ceni-a1/msquic/artifacts/bin/linux/x64_Debug_openssl> ./secnetperf -target:localhost -exec:maxtput -down:5s -ptput:1
Started!

Result: Download 131072 bytes @ 104 kbps (10005.009 ms).
App Main returning status 0

I monitored the CPU usage during the test and the usage is almost zero. In the same scenario, TCP throughput can reach 13G and CPU usage can reach 100%.

I build the repository on Ubuntu18.04.

[nibanks]

Can you try a release build?

[furrio]

Changing to the release build improves performance for a little bit, but still well below the the benchmark.

PS /home/ceni-a1/msquic/artifacts/bin/linux/x64_Release_openssl> ./secnetperf -target:localhost -exec:maxtput -down:10s -ptput:1
Started!

Result: Download 24817913 bytes @ 19825 kbps (10014.540 ms).
App Main returning status 0
PS /home/ceni-a1/msquic/artifacts/bin/linux/x64_Release_openssl> ./secnetperf -target:localhost -exec:maxtput -up:10mb -streams:10 -ptput:1
Started!

Result: Upload 10000000 bytes @ 8408 kbps (9514.238 ms).
Result: Upload 10000000 bytes @ 6183 kbps (12938.695 ms).
Result: Upload 10000000 bytes @ 4805 kbps (16645.940 ms).
Result: Upload 10000000 bytes @ 3945 kbps (20277.119 ms).
Result: Upload 10000000 bytes @ 3163 kbps (25291.560 ms).
Result: Upload 10000000 bytes @ 2989 kbps (26758.182 ms).
Result: Upload 10000000 bytes @ 2948 kbps (27132.617 ms).
Result: Upload 10000000 bytes @ 2457 kbps (32548.429 ms).
Result: Upload 10000000 bytes @ 2308 kbps (34655.189 ms).
Result: Upload 10000000 bytes @ 2232 kbps (35831.396 ms).
App Main returning status 0

[nibanks]

Try setting -pconn:1 -pstream:1 on both sides. It will print some statistics that helps diagnose performance issues.

[furrio]

PS /home/ceni-a1/msquic/artifacts/bin/linux/x64_Release_openssl> ./secnetperf -target:localhost -exec:maxtput -down:10s -ptput:1 -pconn:1 -pstream:1
Started!

Stream Timings (flow blocked):
  SCHEDULING:               0 us
  PACING:                   2809 us
  AMPLIFICATION_PROT:       0 us
  CONGESTION_CONTROL:       234 us
  CONN_FLOW_CONTROL:        0 us
  STREAM_ID_FLOW_CONTROL:   0 us
  STREAM_FLOW_CONTROL:      0 us
  APP:                      0 us
Result: Download 24765933 bytes @ 19778 kbps (10017.328 ms).
Connection Statistics:
  RTT                       218 us
  MinRTT                    88 us
  EcnCapable                0
  SendTotalPackets          632
  SendSuspectedLostPackets  0
  SendSpuriousLostPackets   0
  SendCongestionCount       0
  SendEcnCongestionCount    0
  RecvTotalPackets          17211
  RecvReorderedPackets      0
  RecvDroppedPackets        0
  RecvDuplicatePackets      0
  RecvDecryptionFailures    0
App Main returning status 0

Disable pacing:

PS /home/ceni-a1/msquic/artifacts/bin/linux/x64_Release_openssl> ./secnetperf -target:localhost -exec:maxtput -down:10s -ptput:1 -pconn:1 -pstream:1 -pacing:0
Started!

Stream Timings (flow blocked):
  SCHEDULING:               0 us
  PACING:                   0 us
  AMPLIFICATION_PROT:       0 us
  CONGESTION_CONTROL:       493 us
  CONN_FLOW_CONTROL:        0 us
  STREAM_ID_FLOW_CONTROL:   0 us
  STREAM_FLOW_CONTROL:      0 us
  APP:                      0 us
Result: Download 25039479 bytes @ 20002 kbps (10014.437 ms).
Connection Statistics:
  RTT                       202 us
  MinRTT                    137 us
  EcnCapable                0
  SendTotalPackets          645
  SendSuspectedLostPackets  0
  SendSpuriousLostPackets   0
  SendCongestionCount       0
  SendEcnCongestionCount    0
  RecvTotalPackets          17407
  RecvReorderedPackets      0
  RecvDroppedPackets        0
  RecvDuplicatePackets      0
  RecvDecryptionFailures    0
App Main returning status 0

[nibanks]

Can you do an upload and share the numbers? I need the output on the sender side.

[furrio]

Looks like congestion control took a lot of time.

PS /home/ceni-a1/msquic/artifacts/bin/linux/x64_Release_openssl> ./secnetperf -target:localhost -exec:maxtput -up:5s -ptput:1 -pconn:1 -pstream:1 -cc:cubic
Started!

Stream Timings (flow blocked):
  SCHEDULING:               16 us
  PACING:                   2116 us
  AMPLIFICATION_PROT:       0 us
  CONGESTION_CONTROL:       4962747 us
  CONN_FLOW_CONTROL:        0 us
  STREAM_ID_FLOW_CONTROL:   0 us
  STREAM_FLOW_CONTROL:      0 us
  APP:                      55179 us
Result: Upload 13893632 bytes @ 22006 kbps (5050.752 ms).
Connection Statistics:
  RTT                       309 us
  MinRTT                    87 us
  EcnCapable                0
  SendTotalPackets          10201
  SendSuspectedLostPackets  495
  SendSpuriousLostPackets   0
  SendCongestionCount       11
  SendEcnCongestionCount    0
  RecvTotalPackets          424
  RecvReorderedPackets      0
  RecvDroppedPackets        0
  RecvDuplicatePackets      0
  RecvDecryptionFailures    0
App Main returning status 0
PS /home/ceni-a1/msquic/artifacts/bin/linux/x64_Release_openssl> ./secnetperf -target:localhost -exec:maxtput -up:5s -ptput:1 -pconn:1 -pstream:1 -cc:bbr
Started!

Stream Timings (flow blocked):
  SCHEDULING:               181 us
  PACING:                   0 us
  AMPLIFICATION_PROT:       0 us
  CONGESTION_CONTROL:       4938207 us
  CONN_FLOW_CONTROL:        0 us
  STREAM_ID_FLOW_CONTROL:   0 us
  STREAM_FLOW_CONTROL:      0 us
  APP:                      434412 us
Result: Upload 13500416 bytes @ 21326 kbps (5064.329 ms).
Connection Statistics:
  RTT                       230 us
  MinRTT                    78 us
  EcnCapable                0
  SendTotalPackets          20993
  SendSuspectedLostPackets  11520
  SendSpuriousLostPackets   0
  SendCongestionCount       196
  SendEcnCongestionCount    0
  RecvTotalPackets          514
  RecvReorderedPackets      0
  RecvDroppedPackets        0
  RecvDuplicatePackets      0
  RecvDecryptionFailures    0
App Main returning status 0

[imorlxs]

I am having the same problem... When testing QUIC against TCP, I got 100Mbps (encryption enabled) or 300Mbps (encryption disabled) vs 936Mbps on a 1Gbps network (Two Raspberry Pi 4 and a Dell 2048 -- 10Gbps switch (10Gbps and 1Gbps Interfaces)). I understand that with encryption the CPU becomes a bottleneck, and without it, CONGESTION_CONTROL uses a lot of time blocking the flow (we can send a lot more packets now). Do we know any solution to that? Can we increase the throughput?

[nibanks]

What do the connection statistics look like for the connection?

[imorlxs]

The command I'm using is: ./secnetperf -target:<ip> -up:100s -ptput:1 -pstream:1 -sendbuf:0 -pconn:1 -encrypt:0

And the connection and stream statistics are:

Stream Timings (flow blocked):
  SCHEDULING:               218037 us
  PACING:                   21485685 us
  AMPLIFICATION_PROT:       0 us
  CONGESTION_CONTROL:       50064751 us
  CONN_FLOW_CONTROL:        0 us
  STREAM_ID_FLOW_CONTROL:   0 us
  STREAM_FLOW_CONTROL:      1608068 us
  APP:                      2614 us
Connection Statistics:
  RTT                       332461 us
  MinRTT                    554 us
  EcnCapable                0
  SendTotalPackets          2792732
  SendSuspectedLostPackets  1952
  SendSpuriousLostPackets   0
  SendCongestionCount       1
  SendEcnCongestionCount    0
  RecvTotalPackets          922
  RecvReorderedPackets      0
  RecvDroppedPackets        0
  RecvDuplicatePackets      0
  RecvDecryptionFailures    0
Result: Upload 320520 kbps.

Thank you for your fast response ^^

[anrossi]

Are the two test devices (client and server) directly connected to the switch over ethernet? 332ms RTT seems very high to me for being directly connected to a switch. Is there other traffic going on the switch as well?

[imorlxs]

Yes, they are connected directly to the switch, but I don't know if in the instant of the test there is other traffic. I repeated the test a couple times and the RTT remains almost the same.

However, with two x64 machines, the throughput is as expected. So I guess that the problem is the Raspberry Pi 4 devices and their limited CPU

[anrossi]

Sadly, yeah I think the raspberry pi might be the limiting factor here. FWIW, I tried using a raspberry pi 4 as a router on a 1gbps network and it only moved about 300mbps, so that might be the limit of the hardware.