Async handling of Onion Messages with BackpressureQueue
#10219
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @gijswijs, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request significantly enhances the Lightning Network node's capability to handle onion messages by introducing an asynchronous processing model. By integrating a BackpressureQueue with Random Early Drop, the system can now gracefully manage high volumes of onion messages, preventing performance degradation and ensuring the stability of peer communication. This makes the node a more robust and efficient router for onion message traffic, laying the groundwork for future privacy-preserving communication features.
Highlights
- Asynchronous Onion Message Handling: The core onion message processing has been refactored to operate asynchronously, utilizing a new
BackpressureQueueto prevent synchronous processing from blocking critical peer communication. This ensures that incoming onion messages are enqueued and processed in a background goroutine, improving node stability and throughput. - Introduction of
BackpressureQueue: A new generic, bounded queue namedBackpressureQueuehas been implemented. This queue incorporates a 'Random Early Drop' (RED) predicate, which intelligently discards incoming messages as the queue approaches its capacity. This mechanism provides backpressure to the network under heavy load, preventing resource exhaustion without completely blocking new messages. - Onion Message Protocol Support: New
lnwire.OnionMessagetypes, corresponding feature bits (OnionMessagesOptional,OnionMessagesRequired), and RPC endpoints (SendOnionMessage,SubscribeOnionMessages) have been added. This enables nodes to advertise their capability to handle onion messages and provides an interface for sending and receiving them. - Blinded Path Enhancements for Onion Messages: The
htlcswitch/hoppackage has been updated to specifically handle onion messages within blinded paths. This includes adding aNextNodeIDfield toForwardingInfofor routing messages that don't rely on channel IDs, and adapting payload parsing and validation logic to differentiate between regular payments and onion messages. - Integration Tests for Onion Messages: New integration tests have been introduced to validate the functionality of onion messages, covering both basic sending/receiving between two nodes and multi-hop forwarding through a network of three nodes. These tests ensure the robustness and correctness of the new asynchronous handling and routing logic.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request introduces a significant and well-designed refactoring to handle onion messages asynchronously. The introduction of a generic BackpressureQueue with Random Early Drop (RED) is a great approach to improve node stability and throughput under heavy message load. The code is well-structured, and the inclusion of new integration tests is commendable. My review focuses on a couple of minor areas for code cleanup and simplification to enhance maintainability.
gijswijs
force-pushed
the
onion-messaging-2
branch
from
7f780b2 to
bfdf6ff
Compare
This message type is a message that carries an onion-encrypted payload used for BOLT12 messages.
This commit creates the necessary endpoints for onion messages. Specifically, it adds the following: - `SendOnionMessage` endpoint to send onion messages. - `SubscribeOnionMessages` endpoint to subscribe to incoming onion messages. It uses the `msgmux` package to handle the onion messages.
gijswijs
force-pushed
the
onion-messaging-2
branch
from
bfdf6ff to
551c7dd
Compare
The new wire message defines the OnionMessagePayload, FinalHopPayload, ReplyPath, and related TLV encoding/decoding logic.
Use fork github.com/gijswijs/lightning-onion at commit fac332540872 to include onion-messaging support before the upstream PR is merged. This temporary replace in go.mod ensures compatibility with the current module path while allowing local development and CI to build with the new functionality.
With this commit we implement the logic to parse, decrypt, and forward onion messages. It contains a refactor to its constructor to accept dependencies like the onionProcessor and a message sender function. In brontide.go and server.go it adds the plumbing to for passing through the onionProcessor from the hop iterator and the SendOnionMessage function to the OnionEndpoint's constructor.
Adds the NewNonFinalBlindedRouteDataOnionMessage function to create blinded route data specifically for onion messages.
With this Update we change the SubscribeOnionMessages RPC to return a stream of OnionMessageUpdate messages instead of OnionMessage. This way we also send back the decrypted payload if any, so we can inspect that in itests.
Adds the new integration test file to test forwarding of onion messages through a multi-hop path.
In this commit, we add a new type of queue: the back pressure queue. This is a bounded queue based on a simple channel, that will consult a predicate to decide if we should preemptively drop a message or not. We then provide a sample predicate for this use case, based on random early dropping. Given a min and max threshold, we'll start to drop message randomly once we get past the min threshold, ramping up to the max threshold where we'll start to always drop the message.
This commit implements the BackpressureQueue in the onion message endpoint.
gijswijs
force-pushed
the
onion-messaging-2
branch
from
551c7dd to
27d4f31
Compare
| }() | ||
|
|
||
| for { | ||
| result := o.onionMsgQueue.Dequeue(ctx) |
There was a problem hiding this comment.
Can be worked into the actor system as mentioned in other PRs.
|
@gijswijs, remember to re-request review from reviewers when ready |
4 participants
This pull request refactors the onion message handling to be asynchronous, addressing potential performance bottlenecks in the synchronous design from #10089. This PR also depends on #9838 (commit 3511604 was cherry-picked into this PR). This PR is meant to be merged after those two PRs have been merged.
The main changes are:
Introduction of
BackpressureQueue: A new generic, bounded queue is added. This queue uses a "Random Early Drop" (RED) predicate to start randomly dropping incoming messages as it approaches capacity. This prevents the queue from blocking when full and provides backpressure to the network under heavy load.Asynchronous Message Processing: The
onion_message.OnionEndpointis modified to use this newBackpressureQueue.In summary, where the previous PRs introduced the ability to forward onion messages, this PR makes the implementation robust. It ensures that a high volume of onion messages won't block critical peer communication, improving the node's stability and throughput as a router in the onion message network.