BBR has Extreme Asymmetric Sensitivity to Packet Loss Direction

[mmport80]

Unsure whether this behaviour is expected or whether this is the best place to ask about it.

It seems odd to me however.

Summary

When using Quinn (with BBR congestion control) for HTTP/3 log data transport, I've observed extreme asymmetric sensitivity to packet loss direction. Loss in the server→client direction causes disproportionate performance degradation compared to client→server loss, even though the primary data flow is client→server.

Test Setup

Client: Rust application using Reqwest with Quinn and BBR congestion control
Network: Simulated using tc/netem with controlled packet loss in each direction
Test case: Client sending log data to server

Observed Behavior

Direction of Loss:

• Client → Server. Increase in packet loss has minimal impact in test completion time (even at 5% etc.)
• Server → Client. Exponential loss.. For every 1% of packet loss i see about 100% more time taken to complete my test. E.g. 1% packet loss will double test time, etc.

Environment Details

• Quinn version: 0.11.7 (as configured by Reqwest)
• Caddy version: 2.7.6
• Reqwest version: 0.12.16 (default configuration)
• Network simulation: Linux tc/netem

I can provide more details if required - maybe this is well known..

[djc]

I don't think this is well-known, it seems pretty surprising. Providing a ready-made reproducer would probably be useful.

[mmport80]

OK, I switched to cubic and I see no obvious asymmetry. I will try to give you a small reproducer.

[Ralith]

I actually don't think this is surprising. The direction in the reverse of most application data flow will consist mostly of ACKs. When an ACK is lost, recovery of the congestion and flow control window budget consumed by the packets it acknowledges is delayed. Further, detection of lost ACKs is slower than detection of application-data-bearing packets because ACKs are not, themselves, ACK-eliciting. This probably compounds with the fact that our BBR implementation is both experimental and obsolete. We should consider dropping it entirely since we're not really maintaining it.

I would be surprised if cubic exhibited no asymmetry at all, since the information being lost, and the logic involved, is extremely asymmetrical, but perhaps the effect is just small. Overall be careful simulating packet loss synthetically -- it's easy to end up with really unrealistic scenarios.

[mmport80]

I adjusted the example client

Instead of my usecase where the client uploads, the client is now downloading. Not the best for reproduction.

I think I'll go back to the drawing board.

    ///////////////////////////////////

    // let client_config =
    //     quinn::ClientConfig::new(Arc::new(QuicClientConfig::try_from(client_crypto)?));
    // let mut endpoint = quinn::Endpoint::client(options.bind)?;
    // endpoint.set_default_client_config(client_config);

    let mut client_config =
        quinn::ClientConfig::new(Arc::new(QuicClientConfig::try_from(client_crypto)?));

    // Configure transport with BBR
    let mut transport_config = quinn::TransportConfig::default();
    transport_config
        .congestion_controller_factory(Arc::new(quinn::congestion::BbrConfig::default()));

    // Apply transport config
    client_config.transport_config(Arc::new(transport_config));

    let mut endpoint = quinn::Endpoint::client(options.bind)?;
    endpoint.set_default_client_config(client_config);

    ///////////////////////////////////

Completely different to my original extreme asymmetry result. The below jibes with my a prior expectations..

I will take a closer look at how Reqwest configures Quinn next and try to do a more faithful reproduction..

$ ./test_transfers.sh
QUINN File Transfer Benchmark
=============================
Each scenario will run 5 times

Setting up scenario: baseline
Running 5 tests...
Test 1/5: 22 seconds
Test 2/5: 20 seconds
Test 3/5: 22 seconds
Test 4/5: 22 seconds
Test 5/5: 21 seconds
------------------------------------------
Scenario: baseline. 0% loss, 15ms latency, 2ms jitter in each direction
Average time: 21.40 seconds
------------------------------------------

Setting up scenario: server_to_client_loss
Running 5 tests...
Test 1/5: 24 seconds
Test 2/5: 25 seconds
Test 3/5: 26 seconds
Test 4/5: 27 seconds
Test 5/5: 25 seconds
------------------------------------------
Scenario: server_to_client_loss. +1% loss
Average time: 25.40 seconds
------------------------------------------

Setting up scenario: client_to_server_loss
Running 5 tests...
Test 1/5: 20 seconds
Test 2/5: 22 seconds
Test 3/5: 24 seconds
Test 4/5: 26 seconds
Test 5/5: 21 seconds
------------------------------------------
Scenario: client_to_server_loss. +1% loss
Average time: 22.60 seconds
------------------------------------------

SUMMARY OF RESULTS
=============================
Baseline: 21.40 seconds
Server to client packet loss: 25.40 seconds
Client to server packet loss: 22.60 seconds
=============================

[Ralith]

To be clear, Quinn's BBR congestion controller isn't really maintained, and I'm not sure anyone has the resources to investigate its behavior. If Cubic works for you I'd encourage you to stick with it.

[mmport80]

This probably compounds with the fact that our BBR implementation is both experimental and obsolete. We should consider dropping it entirely since we're not really maintaining it.

This would be a pity, apart from this problem I stumbled upon, BBR seems almost magical (at least to a neophyte like myself)

[Ralith]

It's a very cool concept, and we'd love to support it properly, but there are presently no contributors with the time to drive it forwards. Bear in mind that even a fully maintained, up-to-date BBR implementation is still a fairly experimental thing; most of the internet runs just fine on Cubic.