Low bandwidth compatible pacer

[vitocln]

We are experimenting with using Quinn over slow internet connections, such as 400 Kbps. We've observed unexpected and continuous packet losses in this context. After some investigation, we've identified the pacer as the primary cause.

The current pacer is designed for typical internet connections and enforces a hard lower bound of 10 packets per 2 ms. This rate is excessive for low-bandwidth networks.

To address this, we've implemented a "dirty fix" that allows for reduced pacing capacity and slower token refill rates. With this patch, the unexpected packet losses were eliminated:

diff --git a/quinn-proto/src/connection/pacing.rs b/quinn-proto/src/connection/pacing.rs
index 2e469948..da2dbcf0 100644
--- a/quinn-proto/src/connection/pacing.rs
+++ b/quinn-proto/src/connection/pacing.rs
@@ -28,7 +28,7 @@ impl Pacer {
             capacity,
             last_window: window,
             last_mtu: mtu,
-            tokens: capacity,
+            tokens: capacity.max(mtu as u64), // build up the first token directly
             prev: now,
         }
     }
@@ -89,11 +89,11 @@ impl Pacer {
         }
 
         let elapsed_rtts = time_elapsed.as_secs_f64() / smoothed_rtt.as_secs_f64();
-        let new_tokens = window as f64 * 1.25 * elapsed_rtts;
+        let new_tokens = (window as f64 * 1.25 * elapsed_rtts).min(self.capacity as f64);
+
         self.tokens = self
             .tokens
-            .saturating_add(new_tokens as _)
-            .min(self.capacity);
+            .saturating_add(new_tokens as _);
 
         self.prev = now;
 
@@ -103,7 +103,7 @@ impl Pacer {
         }
 
         let unscaled_delay = smoothed_rtt
-            .checked_mul((bytes_to_send.max(self.capacity) - self.tokens) as _)
+            .checked_mul((bytes_to_send.max(self.capacity) - self.tokens).min(self.capacity) as _)
             .unwrap_or(Duration::MAX)
             / window;
 
@@ -131,9 +131,7 @@ fn optimal_capacity(smoothed_rtt: Duration, window: u64, mtu: u16) -> u64 {
 
     let capacity = ((window as u128 * BURST_INTERVAL_NANOS) / rtt) as u64;
 
-    // Small bursts are less efficient (no GSO), could increase latency and don't effectively
-    // use the channel's buffer capacity. Large bursts might block the connection on sending.
-    capacity.clamp(MIN_BURST_SIZE * mtu as u64, MAX_BURST_SIZE * mtu as u64)
+    capacity.min(MAX_BURST_SIZE * mtu as u64)
 }
 
 /// The burst interval
@@ -144,8 +142,7 @@ fn optimal_capacity(smoothed_rtt: Duration, window: u64, mtu: u16) -> u64 {
 /// more applicable.
 const BURST_INTERVAL_NANOS: u128 = 2_000_000; // 2ms
 
-/// Allows some usage of GSO, and doesn't slow down the handshake.
-const MIN_BURST_SIZE: u64 = 10;
+const MIN_BURST_SIZE: u64 = 1;
 
 /// Creating 256 packets took 1ms in a benchmark, so larger bursts don't make sense.
 const MAX_BURST_SIZE: u64 = 256;

However, this approach has a trade-off: Quinn's transmit loop is now triggered and blocked by the pacer more frequently, leading to increased CPU usage.

We understand that this patch is not ready to be merged upstream in its current form. However, if there's interest, we're willing to invest additional time to rework it into a cleaner solution — one that supports both regular and low-bandwidth networks. For example, we could introduce a TransportConfig option to enable reduced pacer capacity where appropriate.

Would the Quinn project be interested in supporting a pacer compatible with slow network conditions?

[djc]

Would the Quinn project be interested in supporting a pacer compatible with slow network conditions?

Definitely!

From my perspective it would be optimal if we can make this work by responding to observable metrics rather than depending only on configuration.

Who's we?

[Ralith]

Strong agree that Quinn should Just Work in this context. However, I wonder if there's more going on than a limited bandwidth. 400 Kbps isn't even that low!

The congestion controller is the component normally responsible for adapting to the available bandwidth, and it starts with conservative assumptions. The purpose of pacing is to avoid flooding out buffers by trying to send a whole congestion window worth of data instantaneously, which could otherwise occur even when the average bandwidth consumed is within the link's capacity. Most connections should spend far more time blocked on congestion control than on pacing.

[Ralith]

Note that 1400 bytes * 10 packets / 2 ms is only 7MBps. If your NIC -- not the whole network path, just your physical local network adapter -- can transmit at that rate, then the pacer should be fine as-is. Sustained packet loss under those conditions suggests something is malfunctioning in the congestion controller or with the network path itself.

[vitocln]

Who's we?

We are a company working in communication systems.

The problem can occur even with a correctly adjusted congestion window. For testing purposes, we created a congestion control algorithm that varies the CWND linearly up and down, forming a sawtooth pattern. We observed that packet losses can still occur even when the CWND is set appropriately.

The core issue is that some networks prefer packets to be spaced out by a few milliseconds, rather than sent in bursts (within a single millisecond) over longer intervals.

For example:

400 Kbps ≈ 35 packets of 1400 bytes per second
  → about 1 packet every 28 ms
  or
  → ~24 packets sent in a burst (under 1 ms) every 700 ms

Even though both methods yield the same bandwidth, the first case represents "ideal" pacing. The second can cause issues on certain networks.

The problem is that the current pacer behaves like the second case—burst sending—which probably works fine in 99% of scenarios. However, we've encountered a few setups where it leads to problems.

[Ralith]

Thanks, that analysis makes sense. I don't immediately see a way Quinn could auto-detect this condition, so a config option might be justified.

One other approach to consider would be leveraging Linux's SO_TXTIME feature, which could be used to delegate pacing to the kernel, or even hardware, achieving the best of both worlds. I think this would be technically straightforward, with the main drawback of being Linux-specific and that IIRC it requires (one-time) use of elevated permissions to configure a tc queueing discipline which implements it. For example, unprivileged applications running in random end user environments generally cannot do this, but if you fully control your environment it should be trivial. See e.g. man 8 tc-etf.

I think a key question is whether application developers can be expected to know in advance whether they'll be deployed on such networks. Can you share some details on what networks you've found to have this behavior? That will help motivate choice of default behavior.

[Ralith]

Actually, I do have one idea for a portable configuration-independent approach: maybe we should dynamically adjust burst size based on congestion window? Large bursts should only be important when sending at high average rates. One wakeup per 28ms, as in the example above, seems pretty unobjectionable. Perhaps we should define burst intervals rather than sizes -- e.g. try to burst at most once per 10ms by default, sending less often only if we have nothing to send, and more often only if the maximum burst size at that rate still won't fill the congestion window.

[djc]

Perhaps we should define burst intervals rather than sizes -- e.g. try to burst at most once per 10ms by default, sending less often only if we have nothing to send, and more often only if the maximum burst size at that rate still won't fill the congestion window.

Sounds good to me!

[Ralith]

Musing on this some more:

→ ~24 packets sent in a burst (under 1 ms) every 700 ms

Wow, that's a really huge delay. I'm guessing not all of that is artificial (if you're consistently saturating the link you should see batches of exactly 10), but even batches of 10 are spaced by more than 200ms in this situation, meaning we're inducing an average of 100ms extra latency, with 100ms of variance. Not great.

Formalizing my idea above a little more precisely, I think the correct behavior here should for the pacer to yield after the lesser of a maximum interval (say, 10ms), and the interval required to permit a minimum batch size (which can still be 10). That will preserve the current behavior for congestion windows that exceed a minimum batch size per maximum pacing interval, and provide both smoother pacing and improved latency below that, while guaranteeing that we don't needlessly churn the CPU regardless.

[vitocln]

Can you share some details on what networks you've found to have this behavior?

It's a network over a GEO satellite, using a VSAT modem.

Wow, that's a really huge delay.

Yes, that was just a random example — I’m not even sure why I said 700ms, to be honest.

I think the correct behavior here should for the pacer to yield after the lesser of a maximum interval (say, 10ms), and the interval required to permit a minimum batch size (which can still be 10). That will preserve the current behavior for congestion windows that exceed a minimum batch size per maximum pacing interval, and provide both smoother pacing and improved latency below that.

Your suggestion sounds good. We'll aim to propose an initial implementation in the coming weeks. We just have a few urgent matters to handle first. Thanks a lot.