Questionable Application Performance in Low Bandwidth - High Packet Loss scenarios

[Johan511]

I am currently working on an application where achieving low latency is the main aim and to test it's performance setup a network emulation using netem.

When the netem rules are set at <4mbps, 0% packet loss> we observe a latency of around 5 seconds with very high variance compared to a latency of around 300ms when the packet loss is set at 5-10%.

When the bandwidth is set at 50mbps, we observe the expected behaviour of high packet loss causing higher latencies and lower throughput

Is this the expected behaviour of MsQuic?
Are there any logs I can observe/attach to understand this behaviour?

I could monitor QUIC_STATISTICS_V2 but was curious if there were any live logs

[nibanks]

The stream statistics will give you the high level view of what is blocking progress. If it's congestion control then that usually means there is packet loss somewhere and we'll need logs to diagnose. Ideally, if you can reproduce on Windows, we have the best tooling for analyzing the output etl file.

[Johan511]

Thank You @nibanks
We diagnosed the issue by injecting some logs into MsQuic

The problem was with our (badly designed) application protocol

Fwiw,

We abort streams which have not finished sending before a certain deadline, but these aborts (which are done due to inadequacies in the network) are never properly communicated to the Congestion Control algorithm which overflows the wire with packets.

[nibanks]

Can you elaborate more on the part about not updating connection control?

[anrossi]

I'm assuming the receiver is aborting the stream, but due to loss the stream close frame isn't making it back to the sender, so they keep retransmitting?

[Johan511]

We are working on a live streaming application
If the publisher does not send a segment within 250ms (say it is 64KB), publisher aborts the send of that segment (by aborting the stream)

But what is happening is, some part of the 64KB (say 32KB) is sent and is on the wire when the stream is aborted

This 32KB is preventing the send of the next segment and adding delays

Until now we had tested with high packet drop conditions where the congestion window was usually low enough that the number of bytes on wire was not significantly hurting the delay, but when loss is 0, the congestion window explodes soon leading to a lot of bytes in the wire which end up doing nothing but hurt the application performance

[nibanks]

I'm still not clear if there is an issue in MsQuic or just at the app layer. If the app aborts the stream, then it shouldn't be sending any more data on the wire, right?

[Johan511]

The issue is in the bad design of my application layer protocol (not in MsQuic, I apologise for starting this discussion)

We try to send 1MB of data within a 250ms window (every 250ms in a loop), and put in 5 different streams which sends (32KB, 64KB, 128KB, 256KB, 512KB) with increasing priority

Consider a 64KBps connection, ideally we want to send the 32KB chunk and not any of the other chunks as there is no way they will be received in the 250ms window

But if the Congestion Control Window is 100KB, MsQuic puts the 32KB and 64KB chunk on wire

Aborting the stream carrying the 64KB chunk does not serve any purpose as the bytes are now on wire where they will block the send of the future 32KB chunks