From 99a715b7bfdf829768af454170981aa20d678580 Mon Sep 17 00:00:00 2001 From: Daniel Cason Date: Tue, 4 Mar 2025 09:22:31 +0100 Subject: [PATCH 01/52] docs: added file to ADR 003 on values propopagation --- .../adr-003-values-propagation.md | 38 +++++++++++++++++++ docs/architecture/readme.md | 9 +++-- 2 files changed, 43 insertions(+), 4 deletions(-) create mode 100644 docs/architecture/adr-003-values-propagation.md diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md new file mode 100644 index 000000000..357ed7ce9 --- /dev/null +++ b/docs/architecture/adr-003-values-propagation.md @@ -0,0 +1,38 @@ +# ADR 003: Propagation of Proposed Values + +## Changelog + +* 2025-03-04: First draft + +## Context + +> This section contains all the context one needs to understand the current state, and why there is a problem. It should be as succinct as possible and introduce the high level idea behind the solution. + +## Decision + +> This section explains all of the details of the proposed solution, including implementation details. +It should also describe affects / corollary items that may need to be changed as a part of this. +If the proposed change will be large, please also indicate a way to do the change to maximize ease of review. +(e.g. the optimal split of things to do between separate PR's) + +## Status + +> A decision may be "proposed" if it hasn't been agreed upon yet, or "accepted" once it is agreed upon. If a later ADR changes or reverses a decision, it may be marked as "deprecated" or "superseded" with a reference to its replacement. + +Proposed + +## Consequences + +> This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. + +### Positive + +### Negative + +### Neutral + +## References + +> Are there any relevant PR comments, issues that led up to this, or articles referenced for why we made the given design choice? If so link them here! + +* {reference link} diff --git a/docs/architecture/readme.md b/docs/architecture/readme.md index 5f10f20d1..3de2afa5e 100644 --- a/docs/architecture/readme.md +++ b/docs/architecture/readme.md @@ -25,7 +25,8 @@ To suggest an ADR, please make use of the [ADR template](./adr-template.md) prov ## Table of Contents -| ADR \# | Description | Status | -|----------------------------------|-----------------------------------------|----------| -| [001](./adr-001-architecture.md) | High Level Architecture | Accepted | -| [002](./adr-002-node-actor.md) | Node Architecture using the Actor Model | Proposed | +| ADR \# | Description | Status | +|----------------------------------------|-----------------------------------------|----------| +| [001](./adr-001-architecture.md) | High Level Architecture | Accepted | +| [002](./adr-002-node-actor.md) | Node Architecture using the Actor Model | Proposed | +| [003](./adr-003-values-propagation.md) | Propagation of Proposed Values | Proposed | From c462f6fcca8ca528ced63c52795055e80a5d372e Mon Sep 17 00:00:00 2001 From: Daniel Cason Date: Tue, 4 Mar 2025 11:12:51 +0100 Subject: [PATCH 02/52] docs: ADR 003 general Context --- .../adr-003-values-propagation.md | 70 ++++++++++++++++++- 1 file changed, 69 insertions(+), 1 deletion(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 357ed7ce9..64b1b6fe8 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -8,6 +8,69 @@ > This section contains all the context one needs to understand the current state, and why there is a problem. It should be as succinct as possible and introduce the high level idea behind the solution. +Malachite implements a consensus algorithm, [Tendermint][consensus-spec], +that receives as input, for each instance or height of consensus, +a number of proposed values and should output a single decided value, +among the proposed ones. + +There is no assumptions regarding what a **value** is or represents: +its semantics is defined by whatever software uses consensus to propose and +decide values, which from now on we generically refer to as the *application*. +For example, blockchain applications provide as input to consensus blocks to be +appended to the blockchain. + +This ADR is motivated by the fact that a consensus algorithm should actually +implement two main roles: + +- **Value Propagation**: proposed values should be transmitted to all consensus + processes; +- **Value Decision**: a single value, among the possibly multiple proposed + values, must be decided. + +As most abstract consensus algorithms, Tendermint implements both the Value +Propagation and Value Decision roles. +But the analysis of its [pseudo-code][consensus-code] reveals that there is +already an abstract distinction between the two roles. +More specifically, Tendermint is organized into heights and rounds; each round +has three round steps (`propose`, `prevote`, and `precommit`), each of which +having an associated message type: + +- `PROPOSAL` messages carry a proposed value `v` and its main role is of + **Value Propagation**. + They are referred to as [**proposals**][consensus-proposals] and are + broadcast by the `proposer` of a round in the `propose` round step; +- `PREVOTE` and `PRECOMMIT` messages carry an identifier `id(v)` of a proposed + value `v`, or the special value `nil` meaning "no value", and play the role + of **Value Decision**. + They are referred to as [**votes**][consensus-votes], and are broadcast by + all processes, respectively, in the `prevote` and `precommit` round steps. + +Note that `PROPOSAL` messages have also a role on **Value Decision**, as they +carry consensus-related fields that are validated by processes and have +implications in the algorithm. +The point is that they are the only messages with **Value Dissemination** role, +so that the `propose` round step is where the dissemination of values takes +place. + +A second remark is that if a vote issued by a process carries `id(v)`, then the +process must have received `v` as part of a `PROPOSAL` message. +In other words, the **Value Decision** stage of a round can only succeed in +deciding a value if the associated **Value Propagation** stage has also been +successful in delivering the proposed value `v`. + +In fact, every state-transition predicate in the [pseudo-code][consensus-code] +that may lead to a successful round of consensus, i.e. to the decision of a +value `v`, requires the **Value Propagation** stage to be successful, that is, +has the condition: + +``` +XX: upon ⟨PROPOSAL, h_p, r, v, vr⟩ from proposer(h_p, r) +``` + +where `h_p` is current height of consensus process `p`, `r` is a round +(typically `p`'s current round `round_p`), and `vr` is a previous valid round +number `vr < r`, which is only relevant during the `propose` round step. + ## Decision > This section explains all of the details of the proposed solution, including implementation details. @@ -35,4 +98,9 @@ Proposed > Are there any relevant PR comments, issues that led up to this, or articles referenced for why we made the given design choice? If so link them here! -* {reference link} +* [Tendermint consensus specification][consensus-spec] and [pseudo-code][consensus-code] + +[consensus-spec]: ../../specs/consensus/README.md +[consensus-code]: ../../specs/consensus/pseudo-code.md +[consensus-proposals]: ../../specs/consensus/overview.md#proposals +[consensus-votes]: ../../specs/consensus/overview.md#votes From 0900b60f15faedbe63b8c39dc2c3e2c5defdb318 Mon Sep 17 00:00:00 2001 From: Daniel Cason Date: Tue, 4 Mar 2025 13:26:14 +0100 Subject: [PATCH 03/52] docs: ADR 003, Tendermint context vs general one --- .../adr-003-values-propagation.md | 32 ++++++++++++------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 64b1b6fe8..d9149c016 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -27,12 +27,19 @@ implement two main roles: - **Value Decision**: a single value, among the possibly multiple proposed values, must be decided. -As most abstract consensus algorithms, Tendermint implements both the Value -Propagation and Value Decision roles. -But the analysis of its [pseudo-code][consensus-code] reveals that there is -already an abstract distinction between the two roles. +Usually, the most expensive, in terms of latency and bandwidth consumption, +part of the algorithm is the **Value Propagation** one, as discussed in this +[section](#value-propagation). +The next [section](#tendermint) discusses how these roles are implemented by +Tendermint; readers familiar with the algorithm may skip it. + +### Tendermint + +Tendermint implements both the **Value Propagation** and **Value Decision** roles. +Its [pseudo-code][consensus-code], however, reveals that there is already an +abstract distinction between the two roles. More specifically, Tendermint is organized into heights and rounds; each round -has three round steps (`propose`, `prevote`, and `precommit`), each of which +has three round steps (`propose`, `prevote`, and `precommit`), each round step having an associated message type: - `PROPOSAL` messages carry a proposed value `v` and its main role is of @@ -53,15 +60,15 @@ so that the `propose` round step is where the dissemination of values takes place. A second remark is that if a vote issued by a process carries `id(v)`, then the -process must have received `v` as part of a `PROPOSAL` message. +process must have received a `PROPOSAL` message carrying value `v`. In other words, the **Value Decision** stage of a round can only succeed in -deciding a value if the associated **Value Propagation** stage has also been -successful in delivering the proposed value `v`. +deciding a value `v` if the associated **Value Propagation** stage has also +been successful in delivering the proposed value `v` to all (correct) processes. In fact, every state-transition predicate in the [pseudo-code][consensus-code] that may lead to a successful round of consensus, i.e. to the decision of a -value `v`, requires the **Value Propagation** stage to be successful, that is, -has the condition: +value `v`, requires the **Value Propagation** stage to have been successful, +that is, includes the condition: ``` XX: upon ⟨PROPOSAL, h_p, r, v, vr⟩ from proposer(h_p, r) @@ -71,6 +78,9 @@ where `h_p` is current height of consensus process `p`, `r` is a round (typically `p`'s current round `round_p`), and `vr` is a previous valid round number `vr < r`, which is only relevant during the `propose` round step. +### Value Propagation + + ## Decision > This section explains all of the details of the proposed solution, including implementation details. @@ -98,7 +108,7 @@ Proposed > Are there any relevant PR comments, issues that led up to this, or articles referenced for why we made the given design choice? If so link them here! -* [Tendermint consensus specification][consensus-spec] and [pseudo-code][consensus-code] +* [Tendermint consensus specification][consensus-spec] [consensus-spec]: ../../specs/consensus/README.md [consensus-code]: ../../specs/consensus/pseudo-code.md From 1f564d71f0233760b9b0c5dc5d2128ce1cd01861 Mon Sep 17 00:00:00 2001 From: Daniel Cason Date: Tue, 4 Mar 2025 14:25:06 +0100 Subject: [PATCH 04/52] docs: ADR 003, Value Propagation context --- .../adr-003-values-propagation.md | 54 +++++++++++++++++-- 1 file changed, 49 insertions(+), 5 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index d9149c016..ef48d99a2 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -27,11 +27,11 @@ implement two main roles: - **Value Decision**: a single value, among the possibly multiple proposed values, must be decided. -Usually, the most expensive, in terms of latency and bandwidth consumption, -part of the algorithm is the **Value Propagation** one, as discussed in this -[section](#value-propagation). -The next [section](#tendermint) discusses how these roles are implemented by -Tendermint; readers familiar with the algorithm may skip it. +As discussed in the [Value Propagation](#value-propagation) section, +this role tends to be the most expensive, in terms of latency and bandwidth +consumption, of the algorithm. +The [Tendermint](#tendermint) section discusses how these roles are implemented +by Tendermint; readers familiar with the algorithm may skip it. ### Tendermint @@ -80,6 +80,49 @@ number `vr < r`, which is only relevant during the `propose` round step. ### Value Propagation +The propagation of proposed values in Tendermint happens in the first step, the +`propose` step, of a consensus round. +The `proposer` of the round selects a value `v` to propose and includes it in a +`PROPOSAL` message that is send to every process. +In the ordinary case, the proposed value is the input for the process, +retrieved via the `getValue()` function ([pseudo-code][consensus-code] line 18). + +The propagation latency for the `PROPOSAL` message is directly associated to +the byte size of the proposed value `v`. +This is in contrast with the other consensus messages, `PREVOTE` and +`PRECOMMIT`, that carry an `id(v)`, which is an identifier of a proposed value +`v`, that is expected to be smaller than `v` and essentially of fixed byte +size. As a result, the latency of the `prevote` and `precommit` round steps +should be fairly constant. + +Thus, as proposed values `v` get larger, the `propose` step becomes the most +expensive of a consensus round. +It is therefore natural to devise strategies to render the implementation of +the **Value Propagation** role more efficient. + +Here it is important to notice that the +[High Level Architecture for Tendermint Consensus Implementation (ADR 001)][adr001] +already enables distinct implementations for the broadcast of `PROPOSAL` messages, +that propagate potentially _large_ (variable-size) proposed values `v`, +and for vote messages (`PREVOTE` and `PRECOMMIT`), +carrying _small_ (fixed-size) value identifiers `id(v)`: + +```rust +/// Output of the round state machine. +pub enum Output + where Ctx: Context, +{ + [...] + + /// Broadcast the proposal. + Proposal(Ctx::Proposal), + + /// Broadcast the vote. + Vote(Ctx::Vote), + + [...] +} +``` ## Decision @@ -114,3 +157,4 @@ Proposed [consensus-code]: ../../specs/consensus/pseudo-code.md [consensus-proposals]: ../../specs/consensus/overview.md#proposals [consensus-votes]: ../../specs/consensus/overview.md#votes +[adr001]: ./adr-001-architecture.md From ead61e63394eaef77c0cbac939ab81fa87fdd796 Mon Sep 17 00:00:00 2001 From: Daniel Cason Date: Tue, 4 Mar 2025 16:37:15 +0100 Subject: [PATCH 05/52] docs: ADR 103 Alternatives overview --- .../adr-003-values-propagation.md | 72 ++++++++++++++++--- 1 file changed, 63 insertions(+), 9 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index ef48d99a2..0c79af5e5 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -2,7 +2,7 @@ ## Changelog -* 2025-03-04: First draft +* 2025-03-04: Context and proposed values propagation Alternatives ## Context @@ -90,10 +90,10 @@ retrieved via the `getValue()` function ([pseudo-code][consensus-code] line 18). The propagation latency for the `PROPOSAL` message is directly associated to the byte size of the proposed value `v`. This is in contrast with the other consensus messages, `PREVOTE` and -`PRECOMMIT`, that carry an `id(v)`, which is an identifier of a proposed value -`v`, that is expected to be smaller than `v` and essentially of fixed byte -size. As a result, the latency of the `prevote` and `precommit` round steps -should be fairly constant. +`PRECOMMIT`, that carry an identifier `id(v)` of a proposed value `v`, +that is expected to be smaller than `v` and essentially fixed size. +As a result, the latency of the `prevote` and `precommit` round steps should be +fairly constant. Thus, as proposed values `v` get larger, the `propose` step becomes the most expensive of a consensus round. @@ -102,9 +102,11 @@ the **Value Propagation** role more efficient. Here it is important to notice that the [High Level Architecture for Tendermint Consensus Implementation (ADR 001)][adr001] -already enables distinct implementations for the broadcast of `PROPOSAL` messages, +already enables the use of distinct protocols +for the broadcast of [proposals][consensus-proposals] or `PROPOSAL` messages, that propagate potentially _large_ (variable-size) proposed values `v`, -and for vote messages (`PREVOTE` and `PRECOMMIT`), +and for the broadcast of [votes][consensus-votes], +the generic name for `PREVOTE` and `PRECOMMIT` messages, carrying _small_ (fixed-size) value identifiers `id(v)`: ```rust @@ -112,18 +114,70 @@ carrying _small_ (fixed-size) value identifiers `id(v)`: pub enum Output where Ctx: Context, { - [...] + // Several fields ommitted /// Broadcast the proposal. Proposal(Ctx::Proposal), /// Broadcast the vote. Vote(Ctx::Vote), +} +``` + +In the same [ADR 001][adr001], are defined the corresponding inputs to the +consensus state-machine implementation. +The `ProposeValue` input provides the proposed value `v` for a round, +namely the return of the `getValue()` function. +The `Proposal` input represents the reception of a proposal, +carrying a proposed value `v`. +And the `Vote` input represents the reception of a vote, +carrying an identifier `id(v)` of a proposed value `v` +(or the special value `nil`, meaning "no value"): + +```rust +pub enum Input + where Ctx: Context, +{ + // Several fields ommitted + + /// Propose a value for the given round + ProposeValue(Round, Ctx::Value), - [...] + /// Receive a proposal, of the given validity + Proposal(Ctx::Proposal, Validity), + + /// Receive a vote + Vote(Vote), } ``` +The issues that this ADR is meant to address are mainly two: + +1. The fact that [ADR 001][adr001] is not any longer in line with the implementation; +2. The ways that applications, given Malachite consensus' interface, handle the + propagation of proposed values. + +The [Alternatives](#alternatives) section below overviews and discusses some +approaches to handle **Value Propagation** efficiently. + +## Alternatives + +This section presents a (possibly not comprehensive) list of approaches to +handle **Value Propagation** for consensus protocols in general, and for +Tendermint in particular, discussing the pros and cons of each of them. + +### Consensus by Value + +In this approach, the consensus implementation play both the +**Value Propagation** and **Value Decision** roles. + + +### Consensus by Reference + +In this approach, the consensus implementation plays only the +**Value Decision** role. +The application is responsible for implementing the **Value Propagation** role. + ## Decision > This section explains all of the details of the proposed solution, including implementation details. From f0ee5edf6ad8e7ca8ca8257361f9c6d8ac6bb852 Mon Sep 17 00:00:00 2001 From: Daniel Cason Date: Thu, 6 Mar 2025 18:55:13 +0100 Subject: [PATCH 06/52] docs: ADR 003, Context simplified and rewritten --- .../adr-003-values-propagation.md | 184 +++--------------- 1 file changed, 28 insertions(+), 156 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 0c79af5e5..44283be7a 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -15,168 +15,40 @@ among the proposed ones. There is no assumptions regarding what a **value** is or represents: its semantics is defined by whatever software uses consensus to propose and -decide values, which from now on we generically refer to as the *application*. +decide values - which from now it is generically referred to as the *application*. For example, blockchain applications provide as input to consensus blocks to be appended to the blockchain. -This ADR is motivated by the fact that a consensus algorithm should actually -implement two main roles: +There is also no assumption, at consensus level, regarding the **size** of +proposed values: a priori, they can have an arbitrary byte size. +The application, however, is expected to define a maximum byte size for +proposed values and to configure consensus accordingly - of particular +relevance here is the configured duration for timeouts. + +In particular when the size of proposed values is a factor, +it is important to highlight that the implementation of a consensus algorithm +actually comprises two stages: - **Value Propagation**: proposed values should be transmitted to all consensus processes; -- **Value Decision**: a single value, among the possibly multiple proposed - values, must be decided. - -As discussed in the [Value Propagation](#value-propagation) section, -this role tends to be the most expensive, in terms of latency and bandwidth -consumption, of the algorithm. -The [Tendermint](#tendermint) section discusses how these roles are implemented -by Tendermint; readers familiar with the algorithm may skip it. - -### Tendermint - -Tendermint implements both the **Value Propagation** and **Value Decision** roles. -Its [pseudo-code][consensus-code], however, reveals that there is already an -abstract distinction between the two roles. -More specifically, Tendermint is organized into heights and rounds; each round -has three round steps (`propose`, `prevote`, and `precommit`), each round step -having an associated message type: - -- `PROPOSAL` messages carry a proposed value `v` and its main role is of - **Value Propagation**. - They are referred to as [**proposals**][consensus-proposals] and are - broadcast by the `proposer` of a round in the `propose` round step; -- `PREVOTE` and `PRECOMMIT` messages carry an identifier `id(v)` of a proposed - value `v`, or the special value `nil` meaning "no value", and play the role - of **Value Decision**. - They are referred to as [**votes**][consensus-votes], and are broadcast by - all processes, respectively, in the `prevote` and `precommit` round steps. - -Note that `PROPOSAL` messages have also a role on **Value Decision**, as they -carry consensus-related fields that are validated by processes and have -implications in the algorithm. -The point is that they are the only messages with **Value Dissemination** role, -so that the `propose` round step is where the dissemination of values takes -place. - -A second remark is that if a vote issued by a process carries `id(v)`, then the -process must have received a `PROPOSAL` message carrying value `v`. -In other words, the **Value Decision** stage of a round can only succeed in -deciding a value `v` if the associated **Value Propagation** stage has also -been successful in delivering the proposed value `v` to all (correct) processes. - -In fact, every state-transition predicate in the [pseudo-code][consensus-code] -that may lead to a successful round of consensus, i.e. to the decision of a -value `v`, requires the **Value Propagation** stage to have been successful, -that is, includes the condition: - -``` -XX: upon ⟨PROPOSAL, h_p, r, v, vr⟩ from proposer(h_p, r) -``` - -where `h_p` is current height of consensus process `p`, `r` is a round -(typically `p`'s current round `round_p`), and `vr` is a previous valid round -number `vr < r`, which is only relevant during the `propose` round step. - -### Value Propagation - -The propagation of proposed values in Tendermint happens in the first step, the -`propose` step, of a consensus round. -The `proposer` of the round selects a value `v` to propose and includes it in a -`PROPOSAL` message that is send to every process. -In the ordinary case, the proposed value is the input for the process, -retrieved via the `getValue()` function ([pseudo-code][consensus-code] line 18). - -The propagation latency for the `PROPOSAL` message is directly associated to -the byte size of the proposed value `v`. -This is in contrast with the other consensus messages, `PREVOTE` and -`PRECOMMIT`, that carry an identifier `id(v)` of a proposed value `v`, -that is expected to be smaller than `v` and essentially fixed size. -As a result, the latency of the `prevote` and `precommit` round steps should be -fairly constant. - -Thus, as proposed values `v` get larger, the `propose` step becomes the most -expensive of a consensus round. -It is therefore natural to devise strategies to render the implementation of -the **Value Propagation** role more efficient. - -Here it is important to notice that the -[High Level Architecture for Tendermint Consensus Implementation (ADR 001)][adr001] -already enables the use of distinct protocols -for the broadcast of [proposals][consensus-proposals] or `PROPOSAL` messages, -that propagate potentially _large_ (variable-size) proposed values `v`, -and for the broadcast of [votes][consensus-votes], -the generic name for `PREVOTE` and `PRECOMMIT` messages, -carrying _small_ (fixed-size) value identifiers `id(v)`: - -```rust -/// Output of the round state machine. -pub enum Output - where Ctx: Context, -{ - // Several fields ommitted - - /// Broadcast the proposal. - Proposal(Ctx::Proposal), - - /// Broadcast the vote. - Vote(Ctx::Vote), -} -``` - -In the same [ADR 001][adr001], are defined the corresponding inputs to the -consensus state-machine implementation. -The `ProposeValue` input provides the proposed value `v` for a round, -namely the return of the `getValue()` function. -The `Proposal` input represents the reception of a proposal, -carrying a proposed value `v`. -And the `Vote` input represents the reception of a vote, -carrying an identifier `id(v)` of a proposed value `v` -(or the special value `nil`, meaning "no value"): - -```rust -pub enum Input - where Ctx: Context, -{ - // Several fields ommitted - - /// Propose a value for the given round - ProposeValue(Round, Ctx::Value), - - /// Receive a proposal, of the given validity - Proposal(Ctx::Proposal, Validity), - - /// Receive a vote - Vote(Vote), -} -``` - -The issues that this ADR is meant to address are mainly two: - -1. The fact that [ADR 001][adr001] is not any longer in line with the implementation; -2. The ways that applications, given Malachite consensus' interface, handle the - propagation of proposed values. - -The [Alternatives](#alternatives) section below overviews and discusses some -approaches to handle **Value Propagation** efficiently. - -## Alternatives - -This section presents a (possibly not comprehensive) list of approaches to -handle **Value Propagation** for consensus protocols in general, and for -Tendermint in particular, discussing the pros and cons of each of them. - -### Consensus by Value - -In this approach, the consensus implementation play both the -**Value Propagation** and **Value Decision** roles. - - -### Consensus by Reference - -In this approach, the consensus implementation plays only the -**Value Decision** role. -The application is responsible for implementing the **Value Propagation** role. +- **Value Decision**: a single value, among the possibly multiple proposed and + successfully propagated values, must be decided. + +The cost of the **Value Propagation** state, +in terms of latency and bandwidth consumption, +evidently depends on the size of the proposed values. +While the cost of the **Value Decision** stage should be independent from the +size of the proposed values. + +In Tendermint, the message that plays the role of **Value Propagation** is the +`PROPOSAL` message, as it carries the proposed value `v` for a round of consensus. +While the `PREVOTE` and `PRECOMMIT` messages are dedicated to the +**Value Decision** role and carry an identifier `id(v)` of a proposed value +`v`, or the special `nil` value - meaning "no value". +The function `id(v)` can be implemented in multiple ways, the most common of +which is by returning a hash of `v`. +This means that the size of the messages used in the **Value Decision** stage +is usually fixed and independent from the size of the proposed value. ## Decision From 99a288c2939154eba9b2c6d690ae8e55c48ee196 Mon Sep 17 00:00:00 2001 From: Daniel Cason Date: Tue, 18 Mar 2025 14:24:37 +0100 Subject: [PATCH 07/52] docs: ADR 003, sketched Alternatives sections --- .../adr-003-values-propagation.md | 21 +++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 44283be7a..39677310f 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -2,7 +2,7 @@ ## Changelog -* 2025-03-04: Context and proposed values propagation Alternatives +* 2025-03-18: Context and description of the problem ## Context @@ -34,7 +34,7 @@ actually comprises two stages: - **Value Decision**: a single value, among the possibly multiple proposed and successfully propagated values, must be decided. -The cost of the **Value Propagation** state, +The cost of the **Value Propagation** stage, in terms of latency and bandwidth consumption, evidently depends on the size of the proposed values. While the cost of the **Value Decision** stage should be independent from the @@ -50,6 +50,23 @@ which is by returning a hash of `v`. This means that the size of the messages used in the **Value Decision** stage is usually fixed and independent from the size of the proposed value. +## Alternatives + +This section presents a (possibly not comprehensive) list of approaches to +handle **Value Propagation** for consensus protocols in general, and for +Tendermint in particular, discussing the pros and cons of each of them. + +### Consensus by Value + +In this approach, the consensus implementation plays both the +**Value Propagation** and **Value Decision** roles. + +### Consensus by Reference + +In this approach, the consensus implementation plays only the +**Value Decision** role, while the application is fully responsible for +implementing the **Value Propagation** role. + ## Decision > This section explains all of the details of the proposed solution, including implementation details. From 0a52f8a10a1572e06ada524e4d53e4cb6bef1f25 Mon Sep 17 00:00:00 2001 From: Daniel Cason Date: Tue, 18 Mar 2025 19:51:44 +0100 Subject: [PATCH 08/52] docs: ADR 003, overivew of dissemination approaches --- .../adr-003-values-propagation.md | 113 +++++++++++++++--- 1 file changed, 94 insertions(+), 19 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 39677310f..f40c50ec6 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -38,17 +38,20 @@ The cost of the **Value Propagation** stage, in terms of latency and bandwidth consumption, evidently depends on the size of the proposed values. While the cost of the **Value Decision** stage should be independent from the -size of the proposed values. +size of the proposed values, and essentially constant. In Tendermint, the message that plays the role of **Value Propagation** is the -`PROPOSAL` message, as it carries the proposed value `v` for a round of consensus. -While the `PREVOTE` and `PRECOMMIT` messages are dedicated to the -**Value Decision** role and carry an identifier `id(v)` of a proposed value -`v`, or the special `nil` value - meaning "no value". -The function `id(v)` can be implemented in multiple ways, the most common of -which is by returning a hash of `v`. -This means that the size of the messages used in the **Value Decision** stage -is usually fixed and independent from the size of the proposed value. +`PROPOSAL` message. +It is broadcast by the proposer of a round of consensus and carries the +proposed value `v`, defined by the proposer process. + +The **Value Decision** role is played by `PREVOTE` and `PRECOMMIT`, generically +called votes. +A vote carries either an identifier `id(v)` of a proposed value `v`, or the +special `nil` value. +The function `id(v)` should provide a short representation of a value `v`. +It can be implemented in multiple ways, the most common of which is by +returning a fixed byte-size hash of `v`. ## Alternatives @@ -61,28 +64,99 @@ Tendermint in particular, discussing the pros and cons of each of them. In this approach, the consensus implementation plays both the **Value Propagation** and **Value Decision** roles. +This means that a `PROPOSAL(h, r, v, vr)` consensus message broadcast by a +process carries its proposed value `v`. +Other processes learn the proposed value `v` when they receive the associated +`PROPOSAL` message. + +As previously discussed, the vote messages (`PREVOTE` and `PRECOMMIT`) do not +carry the proposed value `v`, but a short representation `id(v)` of it. +A process cannot sign a vote for `id(v)` if it does not know the value `v`. +So receiving `PROPOSAL` message carrying the value `v` is a requirement for +signing a vote for `id(v)`. + +If the round of consensus is successful, the value `v` carried by the round's +`PROPOSAL` message is the value delivered to the application as the decision +for that height of consensus. + +**TODO**: how Malachite supports this approach. +In particular, how the application can disseminate proposed values in an +efficient way. + ### Consensus by Reference -In this approach, the consensus implementation plays only the -**Value Decision** role, while the application is fully responsible for -implementing the **Value Propagation** role. +In this approach, the application is responsible for implementing the +**Value Propagation** stage, +while the consensus algorithm implements **Value Decision** stage. + +This means that a `PROPOSAL(h, r, v, vr)` consensus message broadcast by a +process does not carry the value proposed by this process. +The value `v` ordered by the consensus algorithm is instead a reference, a +description, or an identifier of the value actually proposed by the process, +whose propagation is a responsibility of the application. + +So if a process wants to propose a value `V` using this approach, it has: +(i) to propagate `V` to all processes, then (ii) produce a reference `v` of the +proposed value `V` and provide `v` to the consensus implementation. +On the receive side, a process that receives a `PROPOSAL` carrying `v` should +ensure that the referenced value `V` has been received as well. +Only in this case, the process can deliver the `PROPOSAL` for `v` to the +consensus implementation. + +Since the values that are proposed and decided by consensus are references to +actually proposed values, `v` is expected to be a short representation of `V`. +For this reason, the optimization of having vote messages carrying `id(v)` +instead of `v` becomes pretty much irrelevant. + +If the round of consensus is successful, the reference `v` carried by the +round's `PROPOSAL` message is the value delivered by the consensus +implementation as the decision for that height. +But the actual decision value for that height of consensus is `V`, the value +referenced by `v`. +It is `V` and not `v` that should be delivered to the application. + +Notice, however, that a value can only be decided by the consensus +implementation if a `PROPOSAL` message carrying that value was previously +decided. +As already mentioned, in this approach, a `PROPOSAL` message carrying a +reference `v` can only be delivered to the consensus implementation if the +referenced value `V` is known by the process. +Therefore, a process where `v` is decided by the consensus implementation +should be able to deliver the actual proposed value `V` to the application. + +**TODO**: under the hood, Malachite implements this approach. +But we probably need to offer an example of this approach, for instance, when +values/payloads are disseminated independently and consensus is used to order +the disseminated values, by receiving as `v` identifiers of disseminated values. + +## Current Design + +**TODO**: the three modes of execution. + +The Proposals Streaming protocol. ## Decision -> This section explains all of the details of the proposed solution, including implementation details. -It should also describe affects / corollary items that may need to be changed as a part of this. -If the proposed change will be large, please also indicate a way to do the change to maximize ease of review. -(e.g. the optimal split of things to do between separate PR's) +> This section explains all of the details of the proposed solution, including +> implementation details. +It should also describe affects / corollary items that may need to be changed +as a part of this. If the proposed change will be large, please also indicate +a way to do the change to maximize ease of review. (e.g. the optimal split of +things to do between separate PR's) ## Status -> A decision may be "proposed" if it hasn't been agreed upon yet, or "accepted" once it is agreed upon. If a later ADR changes or reverses a decision, it may be marked as "deprecated" or "superseded" with a reference to its replacement. +> A decision may be "proposed" if it hasn't been agreed upon yet, or "accepted" +> once it is agreed upon. If a later ADR changes or reverses a decision, it may +> be marked as "deprecated" or "superseded" with a reference to its +> replacement. Proposed ## Consequences -> This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. +> This section describes the consequences, after applying the decision. All +> consequences should be summarized here, not just the "positive" ones. ### Positive @@ -92,7 +166,8 @@ Proposed ## References -> Are there any relevant PR comments, issues that led up to this, or articles referenced for why we made the given design choice? If so link them here! +> Are there any relevant PR comments, issues that led up to this, or articles +> referenced for why we made the given design choice? If so link them here! * [Tendermint consensus specification][consensus-spec] From ebd5da706de356998e89bfe050917021bc3ab950 Mon Sep 17 00:00:00 2001 From: Nenad Milosevic Date: Fri, 21 Mar 2025 16:44:28 +0100 Subject: [PATCH 09/52] first take on describing current designs --- .../adr-003-values-propagation.md | 206 ++++++++++++------ 1 file changed, 139 insertions(+), 67 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index f40c50ec6..4ab479e11 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -3,55 +3,148 @@ ## Changelog * 2025-03-18: Context and description of the problem +* 2025-03-21: Current design description ## Context -> This section contains all the context one needs to understand the current state, and why there is a problem. It should be as succinct as possible and introduce the high level idea behind the solution. - -Malachite implements a consensus algorithm, [Tendermint][consensus-spec], -that receives as input, for each instance or height of consensus, -a number of proposed values and should output a single decided value, -among the proposed ones. - -There is no assumptions regarding what a **value** is or represents: -its semantics is defined by whatever software uses consensus to propose and -decide values - which from now it is generically referred to as the *application*. -For example, blockchain applications provide as input to consensus blocks to be -appended to the blockchain. - -There is also no assumption, at consensus level, regarding the **size** of -proposed values: a priori, they can have an arbitrary byte size. -The application, however, is expected to define a maximum byte size for -proposed values and to configure consensus accordingly - of particular -relevance here is the configured duration for timeouts. - -In particular when the size of proposed values is a factor, -it is important to highlight that the implementation of a consensus algorithm -actually comprises two stages: - -- **Value Propagation**: proposed values should be transmitted to all consensus - processes; -- **Value Decision**: a single value, among the possibly multiple proposed and - successfully propagated values, must be decided. - -The cost of the **Value Propagation** stage, -in terms of latency and bandwidth consumption, -evidently depends on the size of the proposed values. -While the cost of the **Value Decision** stage should be independent from the -size of the proposed values, and essentially constant. - -In Tendermint, the message that plays the role of **Value Propagation** is the -`PROPOSAL` message. -It is broadcast by the proposer of a round of consensus and carries the -proposed value `v`, defined by the proposer process. +Malachite implements a consensus algorithm, [Tendermint][consensus-spec], +which allow processes to agree on a single decided value for each +consensus instance or *height*. + +Possible decision values are provided by the software using the consensus +mechanism, which we refer to generically as the *application*. There are +no assumptions about what a **value** represents—its semantics are defined +entirely by the application. For example, in blockchain applications, input +values are blocks proposed to be appended to the blockchain. + +Similarly, the consensus algorithm makes no assumptions about the **size** +of the proposed values; they may be of arbitrary byte size. However, +the application is expected to define a maximum size for proposed values +and to configure the consensus parameters accordingly—most notably, +the durations of timeouts. + +When value size becomes a relevant factor, it is important to recognize +that the consensus process comprises two distinct stages: + +- **Value Propagation**: Proposed values must be transmitted to all consensus participants. +- **Value Decision**: One of the successfully propagated values is selected and decided. + +The cost of **Value Propagation**, in terms of latency and bandwidth, +clearly depends on the size of the proposed values. In contrast, the +**Value Decision** stage should be independent of value size and +incur a roughly constant cost. + +In Tendermint, **Value Propagation** is performed via the `PROPOSAL` +message, which is broadcast by the designated proposer of the round +and includes the proposed value `v`. + +The **Value Decision** phase involves `PREVOTE` and `PRECOMMIT` +messages—collectively referred to as *votes*. Each vote includes +either an identifier `id(v)` of the proposed value `v` or the +special value `nil`. The function `id(v)` provides a compact +representation of `v`, typically implemented as a fixed-size hash. + +From this, we can see that **value propagation** is more challenging, +as it involves disseminating potentially large amounts of data through +the network. In contrast, the **value decision** phase requires only +the transmission of vote messages, which are relatively small and of +constant size. As a result, Malachite’s low-level API provides greater +flexibility in the value propagation stage to enable more optimized +implementations. + +In this document, we focus on the core parts of the consensus +implementation responsible for **value propagation**. + +## Current design + +At the moment, Malachite consensus implementation is supporting three +different methods to handle value propagation: +1) **ProposalOnly** +2) **PartsOnly** +3) **ProposalAndParts** + +A specific mode can be set as a cosnensus parameter (TODO add a reference to the code here): +``` +/// Consensus parameters. +#[derive_where(Clone, Debug)] +pub struct Params { + ... + + /// The messages required to deliver proposals + pub value_payload: ValuePayload, + + ... +} + +``` + +In the following, we examine each approach to understand how the consensus +interacts with the environment, depending on the mode of operation adopted. +Specifically, we present the inputs related to value propagation that +must be provided to the consensus. + +| **Input** | **Description** | +|-----------|-----------------| +| `Proposal(SignedProposal)` | This input is generated and passed to the consensus when a `PROPOSAL` message is received from the network. | +| `Propose(LocallyProposedValue)` | This input is produced by the application in response to a consensus request (`getValue`) for a value to propose. | +| `ProposedValue(ProposedValue, ValueOrigin)` | This input is also generated by the application when the application is responsible for disseminating values through the network. It informs the consensus that a value has been received. | + +When processing each of these inputs, the consensus may produce different effects that must be handled by the environment to fully process the input. These interactions, along with the effects associated with each input, are explained in a separate ADR. Here, our goal is to describe how these inputs are used in relation to the chosen mode of value propagation. + + +**ProposalOnly** – This approach most closely follows the original Tendermint +algorithm. In this mode, the consensus generates the `getValue` effect to +request a value from the application. The application responds by sending the +`Propose(LocallyProposedValue)` input to the consensus, containing the +value `v` that should be ordered. + +The consensus then generates a `PROPOSAL` message that includes the actual +value `v` received via `LocallyProposedValue`, and generate a `Publish` +effect. This effect will be handled by the networking module, and the proposal, +with the full value embedded in it, will be disseminated through the network. + +Upon receiving a `PROPOSAL` message from the network, the networking module +generates a `Proposal(SignedProposal)` input, which is passed directly to +the consensus for processing. + +In this setup, the application only needs to provide a value to the consensus through `Propose(LocallyProposedValue)`, and value propagation is entirely handled by the networking module. The consensus only processes `PROPOSAL` messages that already contain the full value. As a result, the `ProposedValue(ProposedValue, ValueOrigin)` input is never generated by the application in this mode. + +This setup is the simplest, as the application is only responsible for providing the value to be ordered. However, if the value is large, the resulting `PROPOSAL` messages will also be large and must be propagated as such through the network. Any optimizations for value propagation, such as chunking the value into smaller parts and reassembling it on the receiving side, must be implemented at the networking level. This is because both the consensus and application remain unaware of how the value is transmitted. + +The other two modes of operation are designed to support such optimizations at the application level rather than at the network level. We will explore how this is achieved in the following sections. + +**ProposalAndParts** – In this mode of operation, the application, as a response to `getValue`, +generates `Propose(LocallyProposedValue)` input that does not contain the full value `v`, +but instead carries a value identifier. As a result, when processing this input, the consensus generates a +`PROPOSAL` message that includes only the value identifier, not the full value. + +When a node receives a `PROPOSAL` message in this mode, the networking module generates the `Proposal(SignedProposal)` input and passes it to the consensus. However, in this mode, receiving a `Proposal(SignedProposal)` alone is not sufficient for the consensus to consider the proposal complete. +The consensus must also receive confirmation from the application that the full value corresponding +to the identifier in the `PROPOSAL` has been obtained. + +This confirmation is delivered via the `ProposedValue(ProposedValue, ValueOrigin)` input, which the application generates upon receiving the full value. Only when the consensus has both inputs `Proposal(SignedProposal)` with a value identifier and `ProposedValue(ProposedValue, ValueOrigin)` with the matching full value, it can consider the proposal complete. At that point, it proceeds as it would in the `ProposalOnly` mode upon receiving a `Proposal(SignedProposal)` input. + +In this approach, dissemination of the full value is delegated to the application. Specifically, the application is responsible for distributing the full value across the network. One way to implement this is by chunking the value into parts, broadcasting these parts using the networking module, and then informing the consensus once all parts have been received and reassembled. + +PartsOnly - This mode is very similar to Proposal and Parts but the difference is that when receiving +`Propose(LocallyProposedValue)` input from the application the consensus don't do anything. Namely, +the `PROPOSAL` message is not sent through the network. Instead, it waits to receive `ProposedValue(ProposedValue, ValueOrigin)` input that is generated when the application +receive the full value and then it consider the proposal complete and proceed. As a result, in this case +value propagation is totally delegated to the application. + +To sum up, in different modes, different inputs are required to achieve the same effect as receiving the `PROPOSAL` message in the original Tendermint algorithm. + +- In `ProposalOnly`, a single `Proposal(SignedProposal)` input is sufficient. +- In `ProposalAndParts`, both `Proposal(SignedProposal)` and `ProposedValue(ProposedValue, ValueOrigin)` are needed. +- In `PartsOnly`, only `ProposedValue(ProposedValue, ValueOrigin)` is used to trigger the same effect, as no `PROPOSAL` message is sent over the network. + +Each mode shifts responsibility for value propagation between the network and the application. + +### `TimeoutPropose` +An important consideration is that, regardless of the mode of operation, all inputs required to complete a +proposal must be received by the consensus before `timeoutPropose` expires. This timeout must be configured to accommodate for the time needed for a complete value propagation, especially +in cases where the value is large and requires longer to propagate through the network. -The **Value Decision** role is played by `PREVOTE` and `PRECOMMIT`, generically -called votes. -A vote carries either an identifier `id(v)` of a proposed value `v`, or the -special `nil` value. -The function `id(v)` should provide a short representation of a value `v`. -It can be implemented in multiple ways, the most common of which is by -returning a fixed byte-size hash of `v`. ## Alternatives @@ -129,34 +222,16 @@ But we probably need to offer an example of this approach, for instance, when values/payloads are disseminated independently and consensus is used to order the disseminated values, by receiving as `v` identifiers of disseminated values. -## Current Design - -**TODO**: the three modes of execution. - -The Proposals Streaming protocol. - ## Decision -> This section explains all of the details of the proposed solution, including -> implementation details. -It should also describe affects / corollary items that may need to be changed -as a part of this. If the proposed change will be large, please also indicate -a way to do the change to maximize ease of review. (e.g. the optimal split of -things to do between separate PR's) ## Status -> A decision may be "proposed" if it hasn't been agreed upon yet, or "accepted" -> once it is agreed upon. If a later ADR changes or reverses a decision, it may -> be marked as "deprecated" or "superseded" with a reference to its -> replacement. - Proposed ## Consequences -> This section describes the consequences, after applying the decision. All -> consequences should be summarized here, not just the "positive" ones. + ### Positive @@ -166,9 +241,6 @@ Proposed ## References -> Are there any relevant PR comments, issues that led up to this, or articles -> referenced for why we made the given design choice? If so link them here! - * [Tendermint consensus specification][consensus-spec] [consensus-spec]: ../../specs/consensus/README.md From 82a0d086c6e813ad2f153b49f14ddee2c35d7ae5 Mon Sep 17 00:00:00 2001 From: Nenad Milosevic Date: Fri, 21 Mar 2025 17:46:13 +0100 Subject: [PATCH 10/52] small improvments on the text --- .../adr-003-values-propagation.md | 95 ++++++++++++------- 1 file changed, 60 insertions(+), 35 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 4ab479e11..9a1711b7b 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -11,8 +11,8 @@ Malachite implements a consensus algorithm, [Tendermint][consensus-spec], which allow processes to agree on a single decided value for each consensus instance or *height*. -Possible decision values are provided by the software using the consensus -mechanism, which we refer to generically as the *application*. There are +Possible decision values are provided by the software that uses the consensus, +which we refer to generically as the *application*. There are no assumptions about what a **value** represents—its semantics are defined entirely by the application. For example, in blockchain applications, input values are blocks proposed to be appended to the blockchain. @@ -44,27 +44,28 @@ either an identifier `id(v)` of the proposed value `v` or the special value `nil`. The function `id(v)` provides a compact representation of `v`, typically implemented as a fixed-size hash. -From this, we can see that **value propagation** is more challenging, +From this, we can see that **Value Propagation** is more challenging, as it involves disseminating potentially large amounts of data through -the network. In contrast, the **value decision** phase requires only +the network. In contrast, the **Value Decision** phase requires only the transmission of vote messages, which are relatively small and of constant size. As a result, Malachite’s low-level API provides greater flexibility in the value propagation stage to enable more optimized implementations. In this document, we focus on the core parts of the consensus -implementation responsible for **value propagation**. +implementation responsible for **Value Propagation**. ## Current design At the moment, Malachite consensus implementation is supporting three -different methods to handle value propagation: +different modes of operation to handle value propagation: 1) **ProposalOnly** 2) **PartsOnly** 3) **ProposalAndParts** -A specific mode can be set as a cosnensus parameter (TODO add a reference to the code here): -``` +A specific mode can be set as a `value_payload` consensus parameter (TODO: not sure if this is right code reference): + +```rust /// Consensus parameters. #[derive_where(Clone, Debug)] pub struct Params { @@ -78,10 +79,15 @@ pub struct Params { ``` + In the following, we examine each approach to understand how the consensus interacts with the environment, depending on the mode of operation adopted. -Specifically, we present the inputs related to value propagation that -must be provided to the consensus. +Specifically, we focus on the consensus inputs related to value propagation. +In general consensus inputs are how the consensus reacts to the events from +the environment. A complete overview of all inputs processes by the consensus +can be found in [ADR-004 Coroutine-Based Effect System for Consensus](./adr-004-coroutine-effect-system.md). + + | **Input** | **Description** | |-----------|-----------------| @@ -89,10 +95,16 @@ must be provided to the consensus. | `Propose(LocallyProposedValue)` | This input is produced by the application in response to a consensus request (`getValue`) for a value to propose. | | `ProposedValue(ProposedValue, ValueOrigin)` | This input is also generated by the application when the application is responsible for disseminating values through the network. It informs the consensus that a value has been received. | -When processing each of these inputs, the consensus may produce different effects that must be handled by the environment to fully process the input. These interactions, along with the effects associated with each input, are explained in a separate ADR. Here, our goal is to describe how these inputs are used in relation to the chosen mode of value propagation. + +When processing each of these inputs, the consensus may produce various effects +that must be handled by the environment to fully process the input. All such +effects are described in more detail in [ADR-004](./adr-004-coroutine-effect-system.md). +Here, our focus is limited to interactions related to value propagation and how they +differ depending on the selected mode of operation. -**ProposalOnly** – This approach most closely follows the original Tendermint + +**ProposalOnly** - This approach most closely follows the original Tendermint algorithm. In this mode, the consensus generates the `getValue` effect to request a value from the application. The application responds by sending the `Propose(LocallyProposedValue)` input to the consensus, containing the @@ -113,37 +125,50 @@ This setup is the simplest, as the application is only responsible for providing The other two modes of operation are designed to support such optimizations at the application level rather than at the network level. We will explore how this is achieved in the following sections. -**ProposalAndParts** – In this mode of operation, the application, as a response to `getValue`, -generates `Propose(LocallyProposedValue)` input that does not contain the full value `v`, -but instead carries a value identifier. As a result, when processing this input, the consensus generates a -`PROPOSAL` message that includes only the value identifier, not the full value. - -When a node receives a `PROPOSAL` message in this mode, the networking module generates the `Proposal(SignedProposal)` input and passes it to the consensus. However, in this mode, receiving a `Proposal(SignedProposal)` alone is not sufficient for the consensus to consider the proposal complete. -The consensus must also receive confirmation from the application that the full value corresponding -to the identifier in the `PROPOSAL` has been obtained. - -This confirmation is delivered via the `ProposedValue(ProposedValue, ValueOrigin)` input, which the application generates upon receiving the full value. Only when the consensus has both inputs `Proposal(SignedProposal)` with a value identifier and `ProposedValue(ProposedValue, ValueOrigin)` with the matching full value, it can consider the proposal complete. At that point, it proceeds as it would in the `ProposalOnly` mode upon receiving a `Proposal(SignedProposal)` input. - -In this approach, dissemination of the full value is delegated to the application. Specifically, the application is responsible for distributing the full value across the network. One way to implement this is by chunking the value into parts, broadcasting these parts using the networking module, and then informing the consensus once all parts have been received and reassembled. - -PartsOnly - This mode is very similar to Proposal and Parts but the difference is that when receiving -`Propose(LocallyProposedValue)` input from the application the consensus don't do anything. Namely, -the `PROPOSAL` message is not sent through the network. Instead, it waits to receive `ProposedValue(ProposedValue, ValueOrigin)` input that is generated when the application -receive the full value and then it consider the proposal complete and proceed. As a result, in this case +**ProposalAndParts** - In this mode of operation, the application is responsible to propagate the full value `v`. +Consequently, as a response to `getValue`, the application generates `Propose(LocallyProposedValue)` input +that does not contain the full value `v`, but instead carries a value identifier. As a result, the `PROPOSAL` +message generated by consensus upon processing this input includes only the value identifier, +not the full value. Nevertheless, in this mode, the consensus generates the `Publish` effect and this `PROPOSAL` +message is disseminated through the network via networking module. + +When a node receives a `PROPOSAL` message in this mode, the networking module generates the `Proposal(SignedProposal)` +input and passes it to the consensus. However, in this mode, receiving a `Proposal(SignedProposal)` alone is not +sufficient for the consensus to consider the proposal complete. The consensus must also receive confirmation from +the application that the full value corresponding to the identifier in the `PROPOSAL` has been obtained. + +This confirmation is delivered via the `ProposedValue(ProposedValue, ValueOrigin)` input, +which the application generates upon receiving the full value through the network. Only when the consensus has both +inputs `Proposal(SignedProposal)` with a value identifier and `ProposedValue(ProposedValue, ValueOrigin)` +with the matching full value, it can consider the proposal complete. At that point, it proceeds as it would +in the `ProposalOnly` mode upon receiving a `Proposal(SignedProposal)` input. + +In this approach, dissemination of the full value is delegated to the application. Specifically, the application +is responsible for distributing the full value across the network. So the application is albe to chunk +the value into parts, broadcasting these parts using the networking module, and then informing the consensus +once all parts have been received and reassembled. + +**PartsOnly** - This mode is very similar to `ProposalandParts` but the difference is that when receiving +`Propose(LocallyProposedValue)` input from the application the consensus doesn't do anything. Namely, +the `Publish` effect is not created and the `PROPOSAL` message is not sent through the network. Instead, +it waits to receive `ProposedValue(ProposedValue, ValueOrigin)` input from the application and when +this happens the consensus considers the proposal as complete and proceed. The application generates this +input upon receiving the full value from the network. As a result, in this case value propagation is totally delegated to the application. -To sum up, in different modes, different inputs are required to achieve the same effect as receiving the `PROPOSAL` message in the original Tendermint algorithm. +To sum up, in different modes, different inputs are required to achieve the same effect as receiving +the `PROPOSAL` message in the original Tendermint algorithm. - In `ProposalOnly`, a single `Proposal(SignedProposal)` input is sufficient. -- In `ProposalAndParts`, both `Proposal(SignedProposal)` and `ProposedValue(ProposedValue, ValueOrigin)` are needed. -- In `PartsOnly`, only `ProposedValue(ProposedValue, ValueOrigin)` is used to trigger the same effect, as no `PROPOSAL` message is sent over the network. +- In `ProposalAndParts`, both `Proposal(SignedProposal)` and `ProposedValue(ProposedValue, ValueOrigin)` inputs are needed. +- In `PartsOnly`, only `ProposedValue(ProposedValue, ValueOrigin)` input is enough, as no `PROPOSAL` message is sent over the network. -Each mode shifts responsibility for value propagation between the network and the application. ### `TimeoutPropose` An important consideration is that, regardless of the mode of operation, all inputs required to complete a -proposal must be received by the consensus before `timeoutPropose` expires. This timeout must be configured to accommodate for the time needed for a complete value propagation, especially -in cases where the value is large and requires longer to propagate through the network. +proposal must be received by the consensus before `timeoutPropose` expires. This timeout must be configured +to accommodate for the time needed for a complete value propagation. This is especially important +in cases where the value is large and requires longer to be propagated through the network. ## Alternatives From f5d27687dec097ee68a6a628164b5f34f6739185 Mon Sep 17 00:00:00 2001 From: Bastien Faivre Date: Mon, 24 Mar 2025 15:19:43 +0100 Subject: [PATCH 11/52] feat: mermaid diagrams of value propagation modes --- .../adr-003-values-propagation.md | 84 ++++++++++++++++++- 1 file changed, 80 insertions(+), 4 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 9a1711b7b..36b5a02dc 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -102,9 +102,30 @@ effects are described in more detail in [ADR-004](./adr-004-coroutine-effect-sys Here, our focus is limited to interactions related to value propagation and how they differ depending on the selected mode of operation. +### ProposalOnly + +```mermaid +sequenceDiagram + box Proposer + participant C1 as Consensus + participant A1 as Application + end + + participant N as Network + + box Other nodes + participant A2 as Application + participant C2 as Consensus + end + + C1->>A1: getValue + A1->>C1: Propose(LocallyProposedValue) (full value v) + C1->>N: Publish(PROPOSAL with v) + N->>C2: Proposal(SignedProposal) (with v) + Note over C2: Has full value → can proceed +``` - -**ProposalOnly** - This approach most closely follows the original Tendermint +This approach most closely follows the original Tendermint algorithm. In this mode, the consensus generates the `getValue` effect to request a value from the application. The application responds by sending the `Propose(LocallyProposedValue)` input to the consensus, containing the @@ -125,7 +146,35 @@ This setup is the simplest, as the application is only responsible for providing The other two modes of operation are designed to support such optimizations at the application level rather than at the network level. We will explore how this is achieved in the following sections. -**ProposalAndParts** - In this mode of operation, the application is responsible to propagate the full value `v`. +### ProposalAndParts + +```mermaid +sequenceDiagram + box Proposer + participant C1 as Consensus + participant A1 as Application + end + + participant N as Network + + box Other nodes + participant A2 as Application + participant C2 as Consensus + end + + C1->>A1: getValue + A1->>C1: Propose(LocallyProposedValue) (value ID only) + C1->>N: Publish(PROPOSAL with value ID) + N->>C2: Proposal(SignedProposal) (ID only) + + A1->>N: Full value v + N->>A2: Full value v + A2->>C2: ProposedValue(ProposedValue, ValueOrigin) + + Note over C2: Has ID + full value → can proceed +``` + +In this mode of operation, the application is responsible to propagate the full value `v`. Consequently, as a response to `getValue`, the application generates `Propose(LocallyProposedValue)` input that does not contain the full value `v`, but instead carries a value identifier. As a result, the `PROPOSAL` message generated by consensus upon processing this input includes only the value identifier, @@ -148,7 +197,34 @@ is responsible for distributing the full value across the network. So the applic the value into parts, broadcasting these parts using the networking module, and then informing the consensus once all parts have been received and reassembled. -**PartsOnly** - This mode is very similar to `ProposalandParts` but the difference is that when receiving +### PartsOnly + +```mermaid +sequenceDiagram + box Proposer + participant C1 as Consensus + participant A1 as Application + end + + participant N as Network + + box Other nodes + participant A2 as Application + participant C2 as Consensus + end + + C1->>A1: getValue + A1->>C1: Propose(LocallyProposedValue) (value ID) + + A1->>N: Full value v + N->>A2: Full value v + + A2->>C2: ProposedValue(ProposedValue, ValueOrigin) + + Note over C2: Has ProposedValue → can proceed +``` + +This mode is very similar to `ProposalandParts` but the difference is that when receiving `Propose(LocallyProposedValue)` input from the application the consensus doesn't do anything. Namely, the `Publish` effect is not created and the `PROPOSAL` message is not sent through the network. Instead, it waits to receive `ProposedValue(ProposedValue, ValueOrigin)` input from the application and when From 879dc27ba82572e14b9bd4018bfce2f22bcaa792 Mon Sep 17 00:00:00 2001 From: nenadmilosevic95 <50905385+nenadmilosevic95@users.noreply.github.com> Date: Mon, 24 Mar 2025 15:42:28 +0100 Subject: [PATCH 12/52] Minor changes Signed-off-by: nenadmilosevic95 <50905385+nenadmilosevic95@users.noreply.github.com> --- docs/architecture/adr-003-values-propagation.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 36b5a02dc..b259e42d2 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -169,7 +169,7 @@ sequenceDiagram A1->>N: Full value v N->>A2: Full value v - A2->>C2: ProposedValue(ProposedValue, ValueOrigin) + A2->>C2: ProposedValue(ProposedValue, ValueOrigin) (with v) Note over C2: Has ID + full value → can proceed ``` @@ -219,9 +219,9 @@ sequenceDiagram A1->>N: Full value v N->>A2: Full value v - A2->>C2: ProposedValue(ProposedValue, ValueOrigin) + A2->>C2: ProposedValue(ProposedValue, ValueOrigin) (with v) - Note over C2: Has ProposedValue → can proceed + Note over C2: Has full value → can proceed ``` This mode is very similar to `ProposalandParts` but the difference is that when receiving From f830402102064e364844920299c9e94a357aeb0d Mon Sep 17 00:00:00 2001 From: nenadmilosevic95 <50905385+nenadmilosevic95@users.noreply.github.com> Date: Mon, 24 Mar 2025 16:03:05 +0100 Subject: [PATCH 13/52] Small text changes Signed-off-by: nenadmilosevic95 <50905385+nenadmilosevic95@users.noreply.github.com> --- docs/architecture/adr-003-values-propagation.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index b259e42d2..633c34338 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -273,9 +273,9 @@ If the round of consensus is successful, the value `v` carried by the round's `PROPOSAL` message is the value delivered to the application as the decision for that height of consensus. -**TODO**: how Malachite supports this approach. -In particular, how the application can disseminate proposed values in an -efficient way. +Malachite, in `ProposalOnly` mode, when the application returns a full value in +`Propose(LocallyProposedValue)`, follows this approach. + ### Consensus by Reference @@ -318,10 +318,10 @@ referenced value `V` is known by the process. Therefore, a process where `v` is decided by the consensus implementation should be able to deliver the actual proposed value `V` to the application. -**TODO**: under the hood, Malachite implements this approach. -But we probably need to offer an example of this approach, for instance, when -values/payloads are disseminated independently and consensus is used to order -the disseminated values, by receiving as `v` identifiers of disseminated values. +Malachite, when used in `ProposalAndParts` and `PartsOnly` modes, represents +a variant of this approach in which the dissemination of full values is +entirely delegated to the application. + ## Decision From f2450456cf6acc6355c770401632c0640269d6b7 Mon Sep 17 00:00:00 2001 From: Bastien Faivre Date: Mon, 24 Mar 2025 18:29:55 +0100 Subject: [PATCH 14/52] fix: diagrams --- docs/architecture/adr-003-values-propagation.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 633c34338..bb4865ce2 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -107,20 +107,20 @@ differ depending on the selected mode of operation. ```mermaid sequenceDiagram box Proposer - participant C1 as Consensus participant A1 as Application + participant C1 as Consensus end participant N as Network box Other nodes - participant A2 as Application participant C2 as Consensus + participant A2 as Application end C1->>A1: getValue A1->>C1: Propose(LocallyProposedValue) (full value v) - C1->>N: Publish(PROPOSAL with v) + C1->>N: Proposal(SignedProposal) (with v) N->>C2: Proposal(SignedProposal) (with v) Note over C2: Has full value → can proceed ``` @@ -151,20 +151,20 @@ The other two modes of operation are designed to support such optimizations at t ```mermaid sequenceDiagram box Proposer - participant C1 as Consensus participant A1 as Application + participant C1 as Consensus end participant N as Network box Other nodes - participant A2 as Application participant C2 as Consensus + participant A2 as Application end C1->>A1: getValue A1->>C1: Propose(LocallyProposedValue) (value ID only) - C1->>N: Publish(PROPOSAL with value ID) + C1->>N: Proposal(SignedProposal) (ID only) N->>C2: Proposal(SignedProposal) (ID only) A1->>N: Full value v @@ -202,15 +202,15 @@ once all parts have been received and reassembled. ```mermaid sequenceDiagram box Proposer - participant C1 as Consensus participant A1 as Application + participant C1 as Consensus end participant N as Network box Other nodes - participant A2 as Application participant C2 as Consensus + participant A2 as Application end C1->>A1: getValue From 001a1bed542dc6bf0d430c302fe1987816369db7 Mon Sep 17 00:00:00 2001 From: nenadmilosevic95 <50905385+nenadmilosevic95@users.noreply.github.com> Date: Tue, 25 Mar 2025 08:22:32 +0100 Subject: [PATCH 15/52] Update docs/architecture/adr-003-values-propagation.md Co-authored-by: Romain Ruetschi Signed-off-by: nenadmilosevic95 <50905385+nenadmilosevic95@users.noreply.github.com> --- docs/architecture/adr-003-values-propagation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index bb4865ce2..5fe7e3ef0 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -63,7 +63,7 @@ different modes of operation to handle value propagation: 2) **PartsOnly** 3) **ProposalAndParts** -A specific mode can be set as a `value_payload` consensus parameter (TODO: not sure if this is right code reference): +A specific mode can be set as a `value_payload` consensus parameter: ```rust /// Consensus parameters. From 9c98fea83cd04fa0da64371aecc178944d2ac049 Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Tue, 25 Mar 2025 13:23:41 +0100 Subject: [PATCH 16/52] Use V and v, fix diagrams, add some app requirements --- .../adr-003-values-propagation.md | 145 ++++++++++-------- 1 file changed, 80 insertions(+), 65 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 5fe7e3ef0..a01d423c9 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -55,6 +55,10 @@ implementations. In this document, we focus on the core parts of the consensus implementation responsible for **Value Propagation**. +Note: Not sure where to mention this but seems relevant. There have been some slight changes to the Tendermint consensus: +- `getValue()` is asynchronous and the propose timeout is passed as a parameter in order to let the builder know how long it has to build a value. +- `valid(v)` is checked when a `PROPOSAL` is available but before the algorithm runs. This is equivalent in a way with `PROPOSAL(h, r, v, vr, valid)` where valid = {Valid | Invalid}. + ## Current design At the moment, Malachite consensus implementation is supporting three @@ -89,20 +93,36 @@ can be found in [ADR-004 Coroutine-Based Effect System for Consensus](./adr-004- -| **Input** | **Description** | -|-----------|-----------------| -| `Proposal(SignedProposal)` | This input is generated and passed to the consensus when a `PROPOSAL` message is received from the network. | -| `Propose(LocallyProposedValue)` | This input is produced by the application in response to a consensus request (`getValue`) for a value to propose. | -| `ProposedValue(ProposedValue, ValueOrigin)` | This input is also generated by the application when the application is responsible for disseminating values through the network. It informs the consensus that a value has been received. | +| **Input** | **Fields** | **Description** | +|-----------|------------|-----------------| +| `Proposal (SignedProposal)` | `height`, `round`, `value`, `value_origin` | This input is generated and passed to the consensus when a proposal message is received from the network. | +| `Propose (LocallyProposedValue)` | `height`, `round`, `value` | This input is produced by the application in response to a consensus request (`getValue`) for a value to propose. | +| `ProposedValue (ProposedValue, ValueOrigin)` | `height`, `round`, `valid_round`, `proposer`, `value`, `validity` | This input is also generated by the application when the application is responsible for disseminating values through the network. It informs the consensus that a value has been received and validated. | When processing each of these inputs, the consensus may produce various effects that must be handled by the environment to fully process the input. All such effects are described in more detail in [ADR-004](./adr-004-coroutine-effect-system.md). Here, our focus is limited to interactions related to value propagation and how they -differ depending on the selected mode of operation. +differ depending on the selected mode of operation. Towards this goal, in the following sections, the `height` , `round`, `valid_round`, and `value_origin` fields are omitted from the inputs. + +### Possible value types for Consensus Core? + +The concrete value type is defined by the context `Ctx` and is passed to consensus as a type parameter. Each application using the consensus will define its own concrete value type. +The value type must implement the `Value` trait, including the `id()` method. Consensus core uses the `id()` when generating the votes. + +The following notations are used in the following sections: +- `V` denotes the full application value, this for example can be a block in a blockchain application +- `v` is a short representation of the value `V`, it may be for example the hash of the value. ### ProposalOnly +Note: This mode is under development and not yet fully supported in Malachite (see the dotted lines in the diagram). + +This approach most closely follows the original Tendermint algorithm. + +It is expected to be used by applications that have small values to propose. The maximum value size is defined by the p2p network configuration. For example, if libp2p gossipsub is used, this will be the `max_transmit_size` configured for the gossipsub protocol. + +To reduce the size of the vote messages it is recommended that `id(V)` method implementation should return a short representation of the value. ```mermaid sequenceDiagram @@ -118,36 +138,43 @@ sequenceDiagram participant A2 as Application end - C1->>A1: getValue - A1->>C1: Propose(LocallyProposedValue) (full value v) - C1->>N: Proposal(SignedProposal) (with v) - N->>C2: Proposal(SignedProposal) (with v) - Note over C2: Has full value → can proceed + C1->>A1: getValue() + A1->>C1: Propose(LocallyProposedValue(V)) + C1->>N: Proposal(SignedProposal(V)) + N->>C2: Proposal(SignedProposal(V)) + C2-->>A2: Proposal(V) + A2-->>C2: ProposedValue(V, validity) + + Note over C2: Has V _and its validity_ → can proceed + ``` -This approach most closely follows the original Tendermint -algorithm. In this mode, the consensus generates the `getValue` effect to -request a value from the application. The application responds by sending the -`Propose(LocallyProposedValue)` input to the consensus, containing the -value `v` that should be ordered. +In this mode, the consensus generates the `getValue` effect to request a value from the application. The application responds by sending the `Propose(LocallyProposedValue(V))` input to the consensus. -The consensus then generates a `PROPOSAL` message that includes the actual -value `v` received via `LocallyProposedValue`, and generate a `Publish` +The consensus then generates a `Proposal(SignedProposal(V))` message that includes the actual +value `V` received via `LocallyProposedValue(V)`, and generates a `Publish` effect. This effect will be handled by the networking module, and the proposal, with the full value embedded in it, will be disseminated through the network. -Upon receiving a `PROPOSAL` message from the network, the networking module -generates a `Proposal(SignedProposal)` input, which is passed directly to -the consensus for processing. +Upon receiving a `Proposal(SignedProposal(V))` message from the network, the networking module passes it directly to the consensus for processing. +Consensus verifies the proposal is properly signed by the Proposer for the current height and round. + +_(Implementation in progress) The consensus will then pass the unsigned `Proposal(V)` message to the application for validation. Once validation is performed the application generates the `ProposedValue(V, validity)` input and pass it to the consensus._ -In this setup, the application only needs to provide a value to the consensus through `Propose(LocallyProposedValue)`, and value propagation is entirely handled by the networking module. The consensus only processes `PROPOSAL` messages that already contain the full value. As a result, the `ProposedValue(ProposedValue, ValueOrigin)` input is never generated by the application in this mode. +In this setup, the application only needs to provide a value to the consensus through `Propose(LocallyProposedValue(V))`, and value propagation is entirely handled by the networking module. The consensus only processes proposal messages that already contain the full value. -This setup is the simplest, as the application is only responsible for providing the value to be ordered. However, if the value is large, the resulting `PROPOSAL` messages will also be large and must be propagated as such through the network. Any optimizations for value propagation, such as chunking the value into smaller parts and reassembling it on the receiving side, must be implemented at the networking level. This is because both the consensus and application remain unaware of how the value is transmitted. +This setup is the simplest, as the application is only responsible for providing the value to be ordered. However, if the value is large, the resulting proposal messages will also be large and must be propagated as such through the network. Any optimizations for value propagation, such as chunking the value into smaller parts and reassembling it on the receiving side, must be implemented at the networking level. This is because both the consensus and application remain unaware of how the value is transmitted. The other two modes of operation are designed to support such optimizations at the application level rather than at the network level. We will explore how this is achieved in the following sections. ### ProposalAndParts +In this mode of operation, the application is responsible to define and implement the dissemination protocol for the full value `V` and its metadata (`height`, `round` and possibly `valid_round`). It is expected that the application splits the value `V` into parts, signs each part individually, and disseminates them throughout the network. At the receiving end, the application should verify that the parts are properly signed by the Proposer for the `height` and `round` as derived from the parts and reassemble the full value `V`. +The application communicates to consensus that a value is available using a reference `v` to `V`. `v` is expected to be short representation of `V` and a possible (but not mandatory) implementation for `id` is `id(v) = v`. + +In order to handle restarts the application should persist the undecided values `V`. + + ```mermaid sequenceDiagram box Proposer @@ -162,43 +189,31 @@ sequenceDiagram participant A2 as Application end - C1->>A1: getValue - A1->>C1: Propose(LocallyProposedValue) (value ID only) - C1->>N: Proposal(SignedProposal) (ID only) - N->>C2: Proposal(SignedProposal) (ID only) + C1->>A1: getValue() + A1->>C1: Propose(LocallyProposedValue(v)) + C1->>N: Proposal(SignedProposal(v)) + N->>C2: Proposal(SignedProposal(v)) - A1->>N: Full value v - N->>A2: Full value v - A2->>C2: ProposedValue(ProposedValue, ValueOrigin) (with v) + A1->>N: Full value V + N->>A2: Full value V + A2->>C2: ProposedValue(ProposedValue(v, validity)) - Note over C2: Has ID + full value → can proceed + Note over C2: Has v and its validity → can proceed ``` -In this mode of operation, the application is responsible to propagate the full value `v`. -Consequently, as a response to `getValue`, the application generates `Propose(LocallyProposedValue)` input -that does not contain the full value `v`, but instead carries a value identifier. As a result, the `PROPOSAL` -message generated by consensus upon processing this input includes only the value identifier, -not the full value. Nevertheless, in this mode, the consensus generates the `Publish` effect and this `PROPOSAL` -message is disseminated through the network via networking module. - -When a node receives a `PROPOSAL` message in this mode, the networking module generates the `Proposal(SignedProposal)` -input and passes it to the consensus. However, in this mode, receiving a `Proposal(SignedProposal)` alone is not -sufficient for the consensus to consider the proposal complete. The consensus must also receive confirmation from -the application that the full value corresponding to the identifier in the `PROPOSAL` has been obtained. - -This confirmation is delivered via the `ProposedValue(ProposedValue, ValueOrigin)` input, -which the application generates upon receiving the full value through the network. Only when the consensus has both -inputs `Proposal(SignedProposal)` with a value identifier and `ProposedValue(ProposedValue, ValueOrigin)` -with the matching full value, it can consider the proposal complete. At that point, it proceeds as it would -in the `ProposalOnly` mode upon receiving a `Proposal(SignedProposal)` input. - -In this approach, dissemination of the full value is delegated to the application. Specifically, the application -is responsible for distributing the full value across the network. So the application is albe to chunk -the value into parts, broadcasting these parts using the networking module, and then informing the consensus -once all parts have been received and reassembled. +In this mode, as a response to `getValue()`, the application generates `Propose(LocallyProposedValue(v))` input that does not contain the full value `V`, but instead carries a value reference. As a result, the proposal message generated by consensus upon processing this input includes only the value identifier, +not the full value. Nevertheless, in this mode, the consensus generates the `Publish` effect and the `Proposal(SignedProposal(v))` message is disseminated through the network via networking module. + +When a node receives the `Proposal(SignedProposal(v))` message it passes it to the consensus. However, in this mode, receiving a propsal alone is not sufficient for the consensus to consider the proposal complete. The consensus must also receive confirmation from the application that the corresponding full value is availble and was validated. + +This confirmation is delivered via the `ProposedValue(ProposedValue(v, validity))` input, which the application generates upon receiving the full value through the network. Only when the consensus has both `Proposal(SignedProposal(v))` and `ProposedValue(ProposedValue(v, validity))` , it can consider the proposal complete. ### PartsOnly +The same requirements as for `ProposalAndParts` apply. The only difference is that the application must also include `valid_round` in the dissemination protocol so that consensus can distinguish between L22 and L28 in the consensus algorithm. + +In addition the application doesn't need to define a `Proposal` protocol message as one is not used. The reason is that all the information that would be carried in such a message can be derived from `V` and its metatadata. + ```mermaid sequenceDiagram box Proposer @@ -214,31 +229,31 @@ sequenceDiagram end C1->>A1: getValue - A1->>C1: Propose(LocallyProposedValue) (value ID) + A1->>C1: Propose(LocallyProposedValue(v)) A1->>N: Full value v N->>A2: Full value v A2->>C2: ProposedValue(ProposedValue, ValueOrigin) (with v) - Note over C2: Has full value → can proceed + Note over C2: Has v and its validity → can proceed ``` This mode is very similar to `ProposalandParts` but the difference is that when receiving -`Propose(LocallyProposedValue)` input from the application the consensus doesn't do anything. Namely, -the `Publish` effect is not created and the `PROPOSAL` message is not sent through the network. Instead, -it waits to receive `ProposedValue(ProposedValue, ValueOrigin)` input from the application and when -this happens the consensus considers the proposal as complete and proceed. The application generates this -input upon receiving the full value from the network. As a result, in this case -value propagation is totally delegated to the application. +`Propose(LocallyProposedValue(v))` , and after processed by the consensus state machine, +the `Publish` effect is not created and a proposal message is not sent through the network. + +At the receiving side, consensus waits to receive `ProposedValue(ProposedValue(v, validity))` input from the application and when this happens the consensus considers the proposal as complete and proceeds. The application generates this input upon receiving the full value `V` from the network. As a result, in this case value propagation is totally delegated to the application. +### Summary To sum up, in different modes, different inputs are required to achieve the same effect as receiving the `PROPOSAL` message in the original Tendermint algorithm. -- In `ProposalOnly`, a single `Proposal(SignedProposal)` input is sufficient. -- In `ProposalAndParts`, both `Proposal(SignedProposal)` and `ProposedValue(ProposedValue, ValueOrigin)` inputs are needed. -- In `PartsOnly`, only `ProposedValue(ProposedValue, ValueOrigin)` input is enough, as no `PROPOSAL` message is sent over the network. +- In `ProposalOnly` and `ProposalAndParts`, both `Proposal(SignedProposal(x))` and `ProposedValue(ProposedValue(x, validity))` inputs are needed, with `x == V` for the former and `x == v` for the latter. +- In `PartsOnly`, only `ProposedValue(ProposedValue(v, validity))` input is enough, as no explicit proposal message is sent over the network. +Regardless of the mode of operation, the value that consensus operates at the proposal level is defined by the application in the `Value` trait concrete implementation. +The mode of operation is used outside the consensus driver and state machine in order to decide whether to send or accept explicit `Proposal` messages to the caller (via `Effect`). ### `TimeoutPropose` An important consideration is that, regardless of the mode of operation, all inputs required to complete a @@ -247,7 +262,7 @@ to accommodate for the time needed for a complete value propagation. This is esp in cases where the value is large and requires longer to be propagated through the network. -## Alternatives +## Value Propagation Considerations This section presents a (possibly not comprehensive) list of approaches to handle **Value Propagation** for consensus protocols in general, and for From ba2f599d8de60bf91eef05c4991f2bc080424fb2 Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Tue, 25 Mar 2025 13:41:48 +0100 Subject: [PATCH 17/52] Fix typos --- docs/architecture/adr-003-values-propagation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index a01d423c9..bc07aa892 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -204,7 +204,7 @@ sequenceDiagram In this mode, as a response to `getValue()`, the application generates `Propose(LocallyProposedValue(v))` input that does not contain the full value `V`, but instead carries a value reference. As a result, the proposal message generated by consensus upon processing this input includes only the value identifier, not the full value. Nevertheless, in this mode, the consensus generates the `Publish` effect and the `Proposal(SignedProposal(v))` message is disseminated through the network via networking module. -When a node receives the `Proposal(SignedProposal(v))` message it passes it to the consensus. However, in this mode, receiving a propsal alone is not sufficient for the consensus to consider the proposal complete. The consensus must also receive confirmation from the application that the corresponding full value is availble and was validated. +When a node receives the `Proposal(SignedProposal(v))` message it passes it to the consensus. However, in this mode, receiving a proposal alone is not sufficient for the consensus to consider the proposal complete. The consensus must also receive confirmation from the application that the corresponding full value is available and was validated. This confirmation is delivered via the `ProposedValue(ProposedValue(v, validity))` input, which the application generates upon receiving the full value through the network. Only when the consensus has both `Proposal(SignedProposal(v))` and `ProposedValue(ProposedValue(v, validity))` , it can consider the proposal complete. @@ -212,7 +212,7 @@ This confirmation is delivered via the `ProposedValue(ProposedValue(v, validity) The same requirements as for `ProposalAndParts` apply. The only difference is that the application must also include `valid_round` in the dissemination protocol so that consensus can distinguish between L22 and L28 in the consensus algorithm. -In addition the application doesn't need to define a `Proposal` protocol message as one is not used. The reason is that all the information that would be carried in such a message can be derived from `V` and its metatadata. +In addition the application doesn't need to define a `Proposal` protocol message as one is not used. The reason is that all the information that would be carried in such a message can be derived from `V` and its metadata. ```mermaid sequenceDiagram From 6d92a9a26b74c05ec6c5f6617337437ca0d9e352 Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Tue, 25 Mar 2025 14:14:18 +0100 Subject: [PATCH 18/52] Separate consensus core and engine, remove network --- .../adr-003-values-propagation.md | 53 ++++++++++--------- 1 file changed, 28 insertions(+), 25 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index bc07aa892..4e0440147 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -128,22 +128,24 @@ To reduce the size of the vote messages it is recommended that `id(V)` method im sequenceDiagram box Proposer participant A1 as Application - participant C1 as Consensus + participant C1 as Consensus Core + participant E1 as Consensus Engine end - participant N as Network - box Other nodes - participant C2 as Consensus + participant E2 as Consensus Engine + participant C2 as Consensus Core participant A2 as Application end C1->>A1: getValue() A1->>C1: Propose(LocallyProposedValue(V)) - C1->>N: Proposal(SignedProposal(V)) - N->>C2: Proposal(SignedProposal(V)) + C1->>E1: Effect::Publish(SignedProposal(V)) + E1->>E2: Proposal(SignedProposal(V)) + + E2->>C2: Proposal(SignedProposal(V)) C2-->>A2: Proposal(V) - A2-->>C2: ProposedValue(V, validity) + A2-->>C2: ProposedValue(ProposedValue(V, validity)) Note over C2: Has V _and its validity_ → can proceed @@ -179,23 +181,24 @@ In order to handle restarts the application should persist the undecided values sequenceDiagram box Proposer participant A1 as Application - participant C1 as Consensus + participant C1 as Consensus Core + participant E1 as Consensus Engine end - - participant N as Network box Other nodes - participant C2 as Consensus + participant E2 as Consensus Engine + participant C2 as Consensus Core participant A2 as Application end C1->>A1: getValue() A1->>C1: Propose(LocallyProposedValue(v)) - C1->>N: Proposal(SignedProposal(v)) - N->>C2: Proposal(SignedProposal(v)) + C1->>E1: Effect::Publish(SignedProposal(v)) + E1->>E2: Proposal(SignedProposal(v)) - A1->>N: Full value V - N->>A2: Full value V + E2->>C2: Proposal(SignedProposal(v)) + A1->>E2: Full value V + E2->>A2: Full value V A2->>C2: ProposedValue(ProposedValue(v, validity)) Note over C2: Has v and its validity → can proceed @@ -218,30 +221,30 @@ In addition the application doesn't need to define a `Proposal` protocol message sequenceDiagram box Proposer participant A1 as Application - participant C1 as Consensus + participant C1 as Consensus Core + participant E1 as Consensus Engine end - participant N as Network - box Other nodes - participant C2 as Consensus + participant E2 as Consensus Engine + participant C2 as Consensus Core participant A2 as Application end - C1->>A1: getValue + C1->>A1: getValue() A1->>C1: Propose(LocallyProposedValue(v)) - - A1->>N: Full value v - N->>A2: Full value v + C1--xE1: Effect::Publish(SignedProposal(v)) - A2->>C2: ProposedValue(ProposedValue, ValueOrigin) (with v) + A1->>E2: Full value V + E2->>A2: Full value V + A2->>C2: ProposedValue(ProposedValue(v, validity)) Note over C2: Has v and its validity → can proceed ``` This mode is very similar to `ProposalandParts` but the difference is that when receiving `Propose(LocallyProposedValue(v))` , and after processed by the consensus state machine, -the `Publish` effect is not created and a proposal message is not sent through the network. +the `Publish` effect is not emitted and a proposal message is not sent through the network. At the receiving side, consensus waits to receive `ProposedValue(ProposedValue(v, validity))` input from the application and when this happens the consensus considers the proposal as complete and proceeds. The application generates this input upon receiving the full value `V` from the network. As a result, in this case value propagation is totally delegated to the application. From 489dd731342906b031ca71c4f1b5bf9525805046 Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Tue, 25 Mar 2025 15:12:29 +0100 Subject: [PATCH 19/52] Correct diagrams --- .../adr-003-values-propagation.md | 74 ++++++++++--------- 1 file changed, 41 insertions(+), 33 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 4e0440147..c090332c0 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -100,9 +100,8 @@ can be found in [ADR-004 Coroutine-Based Effect System for Consensus](./adr-004- | `ProposedValue (ProposedValue, ValueOrigin)` | `height`, `round`, `valid_round`, `proposer`, `value`, `validity` | This input is also generated by the application when the application is responsible for disseminating values through the network. It informs the consensus that a value has been received and validated. | -When processing each of these inputs, the consensus may produce various effects -that must be handled by the environment to fully process the input. All such -effects are described in more detail in [ADR-004](./adr-004-coroutine-effect-system.md). +When processing each of these inputs, the consensus core may produce various effects that must be handled by the "environment" to fully process the input. All the effects produced by the consensus core are described in more detail in [ADR-004](./adr-004-coroutine-effect-system.md). These are especially relevant if the application integrates directly with the consensus core. Malachite offers a "Consensus Engine" crate that can be used as an integration point. Regardless the of the type of integration, the "Consensus Engine" is shown in this document as the part of the "environment" that handles the effects, relays messages between consensus core and the application, etc. + Here, our focus is limited to interactions related to value propagation and how they differ depending on the selected mode of operation. Towards this goal, in the following sections, the `height` , `round`, `valid_round`, and `value_origin` fields are omitted from the inputs. @@ -128,42 +127,44 @@ To reduce the size of the vote messages it is recommended that `id(V)` method im sequenceDiagram box Proposer participant A1 as Application - participant C1 as Consensus Core participant E1 as Consensus Engine + participant C1 as Consensus Core end box Other nodes - participant E2 as Consensus Engine participant C2 as Consensus Core + participant E2 as Consensus Engine participant A2 as Application end - C1->>A1: getValue() - A1->>C1: Propose(LocallyProposedValue(V)) + C1->>E1: Effect::GetValue() + E1->>A1: GetValue() + A1->>E1: Propose(LocallyProposedValue(V)) + E1->>C1: Propose(LocallyProposedValue(V)) C1->>E1: Effect::Publish(SignedProposal(V)) E1->>E2: Proposal(SignedProposal(V)) E2->>C2: Proposal(SignedProposal(V)) - C2-->>A2: Proposal(V) + E2-->>A2: Proposal(V) A2-->>C2: ProposedValue(ProposedValue(V, validity)) Note over C2: Has V _and its validity_ → can proceed ``` -In this mode, the consensus generates the `getValue` effect to request a value from the application. The application responds by sending the `Propose(LocallyProposedValue(V))` input to the consensus. +In this mode, the consensus core generates the `GetValue` effect to request a value from the application. The application responds by sending the `Propose(LocallyProposedValue(V))` input to the consensus core. -The consensus then generates a `Proposal(SignedProposal(V))` message that includes the actual +The consensus core then generates a `Proposal(SignedProposal(V))` message that includes the actual value `V` received via `LocallyProposedValue(V)`, and generates a `Publish` -effect. This effect will be handled by the networking module, and the proposal, -with the full value embedded in it, will be disseminated through the network. +effect. This effect is handled by the consensus engine, and the proposal, +with the full value embedded in it, is disseminated through the network. -Upon receiving a `Proposal(SignedProposal(V))` message from the network, the networking module passes it directly to the consensus for processing. -Consensus verifies the proposal is properly signed by the Proposer for the current height and round. +Upon receiving a `Proposal(SignedProposal(V))` message from the network, the engine passes it directly to the consensus core for processing. +Consensus core verifies the proposal is properly signed by the Proposer for the current height and round. -_(Implementation in progress) The consensus will then pass the unsigned `Proposal(V)` message to the application for validation. Once validation is performed the application generates the `ProposedValue(V, validity)` input and pass it to the consensus._ +_(Implementation in progress) The consensus engine (or core?) will also pass the unsigned `Proposal(V)` message to the application for validation. Once validation is performed the application generates the `ProposedValue(V, validity)` input and pass it to the consensus core._ -In this setup, the application only needs to provide a value to the consensus through `Propose(LocallyProposedValue(V))`, and value propagation is entirely handled by the networking module. The consensus only processes proposal messages that already contain the full value. +In this setup, the application only needs to provide a value to the consensus core through `Propose(LocallyProposedValue(V))`, and value propagation is entirely handled by the networking module. The consensus core only processes proposal messages that already contain the full value. This setup is the simplest, as the application is only responsible for providing the value to be ordered. However, if the value is large, the resulting proposal messages will also be large and must be propagated as such through the network. Any optimizations for value propagation, such as chunking the value into smaller parts and reassembling it on the receiving side, must be implemented at the networking level. This is because both the consensus and application remain unaware of how the value is transmitted. @@ -181,39 +182,43 @@ In order to handle restarts the application should persist the undecided values sequenceDiagram box Proposer participant A1 as Application - participant C1 as Consensus Core participant E1 as Consensus Engine + participant C1 as Consensus Core end box Other nodes - participant E2 as Consensus Engine participant C2 as Consensus Core + participant E2 as Consensus Engine participant A2 as Application end - C1->>A1: getValue() - A1->>C1: Propose(LocallyProposedValue(v)) + C1->>E1: Effect::GetValue() + E1->>A1: GetValue() + + A1->>E1: Propose(LocallyProposedValue(v)) + E1->>C1: Propose(LocallyProposedValue(v)) C1->>E1: Effect::Publish(SignedProposal(v)) E1->>E2: Proposal(SignedProposal(v)) E2->>C2: Proposal(SignedProposal(v)) A1->>E2: Full value V E2->>A2: Full value V - A2->>C2: ProposedValue(ProposedValue(v, validity)) + A2->>E2: ProposedValue(ProposedValue(v, validity)) + E2->>C2: ProposedValue(ProposedValue(v, validity)) Note over C2: Has v and its validity → can proceed ``` -In this mode, as a response to `getValue()`, the application generates `Propose(LocallyProposedValue(v))` input that does not contain the full value `V`, but instead carries a value reference. As a result, the proposal message generated by consensus upon processing this input includes only the value identifier, -not the full value. Nevertheless, in this mode, the consensus generates the `Publish` effect and the `Proposal(SignedProposal(v))` message is disseminated through the network via networking module. +In this mode, as a response to `GetValue()`, the application generates `Propose(LocallyProposedValue(v))` input that does not contain the full value `V`, but instead carries a value reference. As a result, the proposal message generated by consensus upon processing this input includes only the value reference, +not the full value. Nevertheless, in this mode, the consensus core generates the `Publish` effect and the `Proposal(SignedProposal(v))` message is disseminated through the network via networking module. -When a node receives the `Proposal(SignedProposal(v))` message it passes it to the consensus. However, in this mode, receiving a proposal alone is not sufficient for the consensus to consider the proposal complete. The consensus must also receive confirmation from the application that the corresponding full value is available and was validated. +When a node receives the `Proposal(SignedProposal(v))` message it passes it to the consensus. However, in this mode, receiving a proposal alone is not sufficient for the consensus core to consider the proposal complete. The consensus core must also receive confirmation from the application that the corresponding full value is available and was validated. -This confirmation is delivered via the `ProposedValue(ProposedValue(v, validity))` input, which the application generates upon receiving the full value through the network. Only when the consensus has both `Proposal(SignedProposal(v))` and `ProposedValue(ProposedValue(v, validity))` , it can consider the proposal complete. +This confirmation is delivered via the `ProposedValue(ProposedValue(v, validity))` input, which the application generates upon receiving the full value through the network. Only when the consensus core has both `Proposal(SignedProposal(v))` and `ProposedValue(ProposedValue(v, validity))` inputs, it can consider the proposal complete. ### PartsOnly -The same requirements as for `ProposalAndParts` apply. The only difference is that the application must also include `valid_round` in the dissemination protocol so that consensus can distinguish between L22 and L28 in the consensus algorithm. +The same requirements as for `ProposalAndParts` apply. The only difference is that the application must also include `valid_round` in the dissemination protocol so that consensus core can distinguish between L22 and L28 in the consensus algorithm. In addition the application doesn't need to define a `Proposal` protocol message as one is not used. The reason is that all the information that would be carried in such a message can be derived from `V` and its metadata. @@ -221,32 +226,35 @@ In addition the application doesn't need to define a `Proposal` protocol message sequenceDiagram box Proposer participant A1 as Application - participant C1 as Consensus Core participant E1 as Consensus Engine + participant C1 as Consensus Core end box Other nodes - participant E2 as Consensus Engine participant C2 as Consensus Core + participant E2 as Consensus Engine participant A2 as Application end - C1->>A1: getValue() - A1->>C1: Propose(LocallyProposedValue(v)) + C1->>E1: Effect::GetValue() + E1->>A1: GetValue() + A1->>E1: Propose(LocallyProposedValue(v)) + E1->>C1: Propose(LocallyProposedValue(v)) C1--xE1: Effect::Publish(SignedProposal(v)) A1->>E2: Full value V E2->>A2: Full value V - A2->>C2: ProposedValue(ProposedValue(v, validity)) + A2->>E2: ProposedValue(ProposedValue(v, validity)) + E2->>C2: ProposedValue(ProposedValue(v, validity)) Note over C2: Has v and its validity → can proceed ``` This mode is very similar to `ProposalandParts` but the difference is that when receiving -`Propose(LocallyProposedValue(v))` , and after processed by the consensus state machine, +`Propose(LocallyProposedValue(v))` , and after processed by the consensus core state machine, the `Publish` effect is not emitted and a proposal message is not sent through the network. -At the receiving side, consensus waits to receive `ProposedValue(ProposedValue(v, validity))` input from the application and when this happens the consensus considers the proposal as complete and proceeds. The application generates this input upon receiving the full value `V` from the network. As a result, in this case value propagation is totally delegated to the application. +At the receiving side, consensus core waits to receive `ProposedValue(ProposedValue(v, validity))` input and when this happens it considers the proposal as complete and proceeds. The application generates this input upon receiving the full value `V` from the network. As a result, in this case value propagation is totally delegated to the application. ### Summary To sum up, in different modes, different inputs are required to achieve the same effect as receiving From fd01fecf19859d0bb02a5f36181ca1dffbde8986 Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Tue, 25 Mar 2025 16:53:12 +0100 Subject: [PATCH 20/52] Address review comments --- docs/architecture/adr-003-values-propagation.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index c090332c0..4dbf3a63a 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -95,9 +95,9 @@ can be found in [ADR-004 Coroutine-Based Effect System for Consensus](./adr-004- | **Input** | **Fields** | **Description** | |-----------|------------|-----------------| -| `Proposal (SignedProposal)` | `height`, `round`, `value`, `value_origin` | This input is generated and passed to the consensus when a proposal message is received from the network. | -| `Propose (LocallyProposedValue)` | `height`, `round`, `value` | This input is produced by the application in response to a consensus request (`getValue`) for a value to propose. | -| `ProposedValue (ProposedValue, ValueOrigin)` | `height`, `round`, `valid_round`, `proposer`, `value`, `validity` | This input is also generated by the application when the application is responsible for disseminating values through the network. It informs the consensus that a value has been received and validated. | +| `Proposal (SignedProposal)` | `height`, `round`, `value`, `value_origin` | This input is generated and passed to the consensus when a proposal consensus message is received from the network. | +| `Propose (LocallyProposedValue)` | `height`, `round`, `value` | This input is produced by the application in response to a consensus request (`getValue()`) for a value to propose. | +| `ProposedValue (ProposedValue, ValueOrigin)` | `height`, `round`, `valid_round`, `proposer`, `value`, `validity` | This input is also generated by the application when the application is responsible for disseminating values through the network. It informs the consensus that a proposedvalue has been received and validated. | When processing each of these inputs, the consensus core may produce various effects that must be handled by the "environment" to fully process the input. All the effects produced by the consensus core are described in more detail in [ADR-004](./adr-004-coroutine-effect-system.md). These are especially relevant if the application integrates directly with the consensus core. Malachite offers a "Consensus Engine" crate that can be used as an integration point. Regardless the of the type of integration, the "Consensus Engine" is shown in this document as the part of the "environment" that handles the effects, relays messages between consensus core and the application, etc. From 6656c7b84e6587547510d47d5b8f8186056ed4e3 Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Tue, 25 Mar 2025 17:25:25 +0100 Subject: [PATCH 21/52] Add details for the different diagram blocks --- .../adr-003-values-propagation.md | 30 +++++++++++++++++-- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 4dbf3a63a..24e2e663b 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -61,6 +61,30 @@ Note: Not sure where to mention this but seems relevant. There have been some sl ## Current design +### Building Blocks +_(WIP)_ + +There are a number of entities that are involved in the value propagation process. +- Application - this is the software that uses the consensus. It is responsible for: + - providing the value to be propagated + - validating received values + - providing data availability for values (both undecided and decided) +- Consensus Engine - a component that is responsible for managing the interactions between the consensus core and the application, and between the consensus core and the networking layer. For the ones involved in the value propagation process, the consensus engine is responsible for: + - propagating values and/or proposals via the networking layer + - receiving values and/or proposals from the networking layer + - relaying values and/or proposals to the consensus core +- Consensus Core - this is the component that implements the tendermint consensus protocol. +- Networking - this is the component that is responsible for transmitting messages between nodes. +(Currently not shown in the diagrams) + +Malachite provides implementations for the consensus core, engine and networking. Applications can integrate with: + +- consensus core - see [ADR-004](./adr-004-coroutine-effect-system.md) +- engine and core integration - farcaster? +- engine, core, networking integration - channel and starknet test examples? + +### Value Payload Modes + At the moment, Malachite consensus implementation is supporting three different modes of operation to handle value propagation: 1) **ProposalOnly** @@ -86,7 +110,7 @@ pub struct Params { In the following, we examine each approach to understand how the consensus interacts with the environment, depending on the mode of operation adopted. -Specifically, we focus on the consensus inputs related to value propagation. +Specifically, we focus on the core consensus inputs related to value propagation. In general consensus inputs are how the consensus reacts to the events from the environment. A complete overview of all inputs processes by the consensus can be found in [ADR-004 Coroutine-Based Effect System for Consensus](./adr-004-coroutine-effect-system.md). @@ -121,7 +145,7 @@ This approach most closely follows the original Tendermint algorithm. It is expected to be used by applications that have small values to propose. The maximum value size is defined by the p2p network configuration. For example, if libp2p gossipsub is used, this will be the `max_transmit_size` configured for the gossipsub protocol. -To reduce the size of the vote messages it is recommended that `id(V)` method implementation should return a short representation of the value. +To reduce the size of the vote messages it is recommended that `id(V)` method implementation returns a short representation of the value. ```mermaid sequenceDiagram @@ -152,7 +176,7 @@ sequenceDiagram ``` -In this mode, the consensus core generates the `GetValue` effect to request a value from the application. The application responds by sending the `Propose(LocallyProposedValue(V))` input to the consensus core. +In this mode, the consensus core generates the `GetValue()` effect to request a value from the application. The application responds by sending the `Propose(LocallyProposedValue(V))` input to the consensus core. The consensus core then generates a `Proposal(SignedProposal(V))` message that includes the actual value `V` received via `LocallyProposedValue(V)`, and generates a `Publish` From 7426672a262f9a7d88e9e0f079379a6f91a1898b Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Tue, 25 Mar 2025 17:33:19 +0100 Subject: [PATCH 22/52] Add warnings for ProposalOnly --- docs/architecture/adr-003-values-propagation.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 24e2e663b..feb918a7a 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -55,7 +55,7 @@ implementations. In this document, we focus on the core parts of the consensus implementation responsible for **Value Propagation**. -Note: Not sure where to mention this but seems relevant. There have been some slight changes to the Tendermint consensus: +TODO: Not sure where to mention this but seems relevant. There have been some slight changes to the Tendermint consensus: - `getValue()` is asynchronous and the propose timeout is passed as a parameter in order to let the builder know how long it has to build a value. - `valid(v)` is checked when a `PROPOSAL` is available but before the algorithm runs. This is equivalent in a way with `PROPOSAL(h, r, v, vr, valid)` where valid = {Valid | Invalid}. @@ -139,7 +139,8 @@ The following notations are used in the following sections: - `v` is a short representation of the value `V`, it may be for example the hash of the value. ### ProposalOnly -Note: This mode is under development and not yet fully supported in Malachite (see the dotted lines in the diagram). +>[!WARNING] +> This mode is under development and not yet fully supported in Malachite (see the dotted lines in the diagram). This approach most closely follows the original Tendermint algorithm. @@ -325,6 +326,8 @@ for that height of consensus. Malachite, in `ProposalOnly` mode, when the application returns a full value in `Propose(LocallyProposedValue)`, follows this approach. +> [!WARNING] +> As mentioned in the [ProposalOnly](#proposalonly) section, this mode is under development and not yet fully supported in Malachite. ### Consensus by Reference From 864e8d669bd888e5a38948f4487e08ac7a2723df Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Tue, 25 Mar 2025 13:14:14 -0400 Subject: [PATCH 23/52] Apply suggestions from code review Co-authored-by: Daniel Signed-off-by: Anca Zamfir --- docs/architecture/adr-003-values-propagation.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index feb918a7a..c87efef15 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -58,7 +58,7 @@ implementation responsible for **Value Propagation**. TODO: Not sure where to mention this but seems relevant. There have been some slight changes to the Tendermint consensus: - `getValue()` is asynchronous and the propose timeout is passed as a parameter in order to let the builder know how long it has to build a value. - `valid(v)` is checked when a `PROPOSAL` is available but before the algorithm runs. This is equivalent in a way with `PROPOSAL(h, r, v, vr, valid)` where valid = {Valid | Invalid}. - +- `upon ⟨PROPOSAL, h, r, v, vr⟩ from proposer(h, r)` represents the receiving of a `PROPOSAL` message for a round `r` of height `h` for vale `v`. This is a necessary condition for the success of round `v` with the decision of the proposed value `v`. ## Current design ### Building Blocks @@ -85,7 +85,7 @@ Malachite provides implementations for the consensus core, engine and networking ### Value Payload Modes -At the moment, Malachite consensus implementation is supporting three +At the moment, Malachite supports three different modes of operation to handle value propagation: 1) **ProposalOnly** 2) **PartsOnly** @@ -324,7 +324,7 @@ If the round of consensus is successful, the value `v` carried by the round's `PROPOSAL` message is the value delivered to the application as the decision for that height of consensus. -Malachite, in `ProposalOnly` mode, when the application returns a full value in +Malachite, in [`ProposalOnly` mode](#proposalonly), when the application returns a full value in `Propose(LocallyProposedValue)`, follows this approach. > [!WARNING] > As mentioned in the [ProposalOnly](#proposalonly) section, this mode is under development and not yet fully supported in Malachite. From b93057673efed92237628741106f43354ae340bf Mon Sep 17 00:00:00 2001 From: Romain Ruetschi Date: Wed, 26 Mar 2025 10:25:31 +0100 Subject: [PATCH 24/52] Small changes --- .../adr-003-values-propagation.md | 146 +++++++++--------- 1 file changed, 75 insertions(+), 71 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index c87efef15..470e3dff0 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -7,86 +7,88 @@ ## Context -Malachite implements a consensus algorithm, [Tendermint][consensus-spec], -which allow processes to agree on a single decided value for each -consensus instance or *height*. - -Possible decision values are provided by the software that uses the consensus, -which we refer to generically as the *application*. There are -no assumptions about what a **value** represents—its semantics are defined -entirely by the application. For example, in blockchain applications, input +Malachite implements a consensus algorithm, [Tendermint][consensus-spec], +which allow processes to agree on a single decided value for each +consensus instance or *height*. + +Possible decision values are provided by the software that uses the consensus, +which we refer to generically as the *application*. There are +no assumptions about what a **value** represents—its semantics are defined +entirely by the application. For example, in blockchain applications, input values are blocks proposed to be appended to the blockchain. -Similarly, the consensus algorithm makes no assumptions about the **size** -of the proposed values; they may be of arbitrary byte size. However, -the application is expected to define a maximum size for proposed values -and to configure the consensus parameters accordingly—most notably, +Similarly, the consensus algorithm makes no assumptions about the **size** +of the proposed values; they may be of arbitrary byte size. However, +the application is expected to define a maximum size for proposed values +and to configure the consensus parameters accordingly—most notably, the durations of timeouts. -When value size becomes a relevant factor, it is important to recognize +When value size becomes a relevant factor, it is important to recognize that the consensus process comprises two distinct stages: - **Value Propagation**: Proposed values must be transmitted to all consensus participants. - **Value Decision**: One of the successfully propagated values is selected and decided. -The cost of **Value Propagation**, in terms of latency and bandwidth, -clearly depends on the size of the proposed values. In contrast, the -**Value Decision** stage should be independent of value size and +The cost of **Value Propagation**, in terms of latency and bandwidth, +clearly depends on the size of the proposed values. In contrast, the +**Value Decision** stage should be independent of value size and incur a roughly constant cost. -In Tendermint, **Value Propagation** is performed via the `PROPOSAL` -message, which is broadcast by the designated proposer of the round +In Tendermint, **Value Propagation** is performed via the `PROPOSAL` +message, which is broadcast by the designated proposer of the round and includes the proposed value `v`. -The **Value Decision** phase involves `PREVOTE` and `PRECOMMIT` -messages—collectively referred to as *votes*. Each vote includes -either an identifier `id(v)` of the proposed value `v` or the -special value `nil`. The function `id(v)` provides a compact +The **Value Decision** phase involves `PREVOTE` and `PRECOMMIT` +messages—collectively referred to as *votes*. Each vote includes +either an identifier `id(v)` of the proposed value `v` or the +special value `nil`. The function `id(v)` provides a compact representation of `v`, typically implemented as a fixed-size hash. -From this, we can see that **Value Propagation** is more challenging, -as it involves disseminating potentially large amounts of data through -the network. In contrast, the **Value Decision** phase requires only -the transmission of vote messages, which are relatively small and of -constant size. As a result, Malachite’s low-level API provides greater -flexibility in the value propagation stage to enable more optimized -implementations. +From this, we can see that **Value Propagation** is more challenging, +as it involves disseminating potentially large amounts of data through +the network. In contrast, the **Value Decision** phase requires only +the transmission of vote messages, which are relatively small and of +constant size. As a result, Malachite’s low-level API provides greater +flexibility in the value propagation stage to enable more optimized +implementations. -In this document, we focus on the core parts of the consensus +In this document, we focus on the core parts of the consensus implementation responsible for **Value Propagation**. TODO: Not sure where to mention this but seems relevant. There have been some slight changes to the Tendermint consensus: - `getValue()` is asynchronous and the propose timeout is passed as a parameter in order to let the builder know how long it has to build a value. - `valid(v)` is checked when a `PROPOSAL` is available but before the algorithm runs. This is equivalent in a way with `PROPOSAL(h, r, v, vr, valid)` where valid = {Valid | Invalid}. - `upon ⟨PROPOSAL, h, r, v, vr⟩ from proposer(h, r)` represents the receiving of a `PROPOSAL` message for a round `r` of height `h` for vale `v`. This is a necessary condition for the success of round `v` with the decision of the proposed value `v`. + ## Current design ### Building Blocks -_(WIP)_ - -There are a number of entities that are involved in the value propagation process. -- Application - this is the software that uses the consensus. It is responsible for: - - providing the value to be propagated - - validating received values - - providing data availability for values (both undecided and decided) -- Consensus Engine - a component that is responsible for managing the interactions between the consensus core and the application, and between the consensus core and the networking layer. For the ones involved in the value propagation process, the consensus engine is responsible for: - - propagating values and/or proposals via the networking layer - - receiving values and/or proposals from the networking layer - - relaying values and/or proposals to the consensus core -- Consensus Core - this is the component that implements the tendermint consensus protocol. -- Networking - this is the component that is responsible for transmitting messages between nodes. -(Currently not shown in the diagrams) + +There are a number of entities that are involved in the value propagation process: + +- **Application:** The software that uses the consensus engine. It is responsible for: + - Providing the value to be propagated + - Validating received values + - Providing data availability for values (both undecided and decided) +- **Consensus Engine:** The component responsible for managing the interactions between the consensus core and the application, and between the consensus core and the networking layer. For the ones involved in the value propagation process, the consensus engine is responsible for: + - Propagating values and/or proposals via the networking layer + - Receiving values and/or proposals from the networking layer + - Relaying values and/or proposals to the consensus core +- **Consensus Core:** The component that implements the tendermint consensus protocol. +- **Networking:** The component responsible for transmitting messages between nodes (currently not shown in the diagrams). Malachite provides implementations for the consensus core, engine and networking. Applications can integrate with: -- consensus core - see [ADR-004](./adr-004-coroutine-effect-system.md) -- engine and core integration - farcaster? -- engine, core, networking integration - channel and starknet test examples? +- Consensus core - See [ADR-004][adr-004] +- Engine without networking - [Snapchain](https://github.com/farcasterxyz/snapchain) +- Engine with networking (channel-based) - [Example application](https://github.com/informalsystems/malachite/tree/main/code/examples/channel) +- Engine with networking (actor-based) - [Starknet test application](https://github.com/informalsystems/malachite/tree/main/code/crates/starknet) ### Value Payload Modes At the moment, Malachite supports three -different modes of operation to handle value propagation: +different modes of operation to handle value propagation: + 1) **ProposalOnly** 2) **PartsOnly** 3) **ProposalAndParts** @@ -107,14 +109,12 @@ pub struct Params { ``` - -In the following, we examine each approach to understand how the consensus -interacts with the environment, depending on the mode of operation adopted. +In the following, we examine each approach to understand how the consensus +interacts with the environment, depending on the mode of operation adopted. Specifically, we focus on the core consensus inputs related to value propagation. -In general consensus inputs are how the consensus reacts to the events from -the environment. A complete overview of all inputs processes by the consensus -can be found in [ADR-004 Coroutine-Based Effect System for Consensus](./adr-004-coroutine-effect-system.md). - +In general consensus inputs are how the consensus reacts to the events from +the environment. A complete overview of all inputs processes by the consensus +can be found in [ADR-004 Coroutine-Based Effect System for Consensus][adr-004]. | **Input** | **Fields** | **Description** | @@ -124,9 +124,9 @@ can be found in [ADR-004 Coroutine-Based Effect System for Consensus](./adr-004- | `ProposedValue (ProposedValue, ValueOrigin)` | `height`, `round`, `valid_round`, `proposer`, `value`, `validity` | This input is also generated by the application when the application is responsible for disseminating values through the network. It informs the consensus that a proposedvalue has been received and validated. | -When processing each of these inputs, the consensus core may produce various effects that must be handled by the "environment" to fully process the input. All the effects produced by the consensus core are described in more detail in [ADR-004](./adr-004-coroutine-effect-system.md). These are especially relevant if the application integrates directly with the consensus core. Malachite offers a "Consensus Engine" crate that can be used as an integration point. Regardless the of the type of integration, the "Consensus Engine" is shown in this document as the part of the "environment" that handles the effects, relays messages between consensus core and the application, etc. +When processing each of these inputs, the consensus core may produce various effects that must be handled by the "environment" to fully process the input. All the effects produced by the consensus core are described in more detail in [ADR-004][adr-004]. These are especially relevant if the application integrates directly with the consensus core. Malachite offers a "Consensus Engine" crate that can be used as an integration point. Regardless the of the type of integration, the "Consensus Engine" is shown in this document as the part of the "environment" that handles the effects, relays messages between consensus core and the application, etc. -Here, our focus is limited to interactions related to value propagation and how they +Here, our focus is limited to interactions related to value propagation and how they differ depending on the selected mode of operation. Towards this goal, in the following sections, the `height` , `round`, `valid_round`, and `value_origin` fields are omitted from the inputs. ### Possible value types for Consensus Core? @@ -139,7 +139,8 @@ The following notations are used in the following sections: - `v` is a short representation of the value `V`, it may be for example the hash of the value. ### ProposalOnly ->[!WARNING] + +> [!WARNING] > This mode is under development and not yet fully supported in Malachite (see the dotted lines in the diagram). This approach most closely follows the original Tendermint algorithm. @@ -210,7 +211,7 @@ sequenceDiagram participant E1 as Consensus Engine participant C1 as Consensus Core end - + box Other nodes participant C2 as Consensus Core participant E2 as Consensus Engine @@ -275,15 +276,15 @@ sequenceDiagram Note over C2: Has v and its validity → can proceed ``` -This mode is very similar to `ProposalandParts` but the difference is that when receiving +This mode is very similar to `ProposalandParts` but the difference is that when receiving `Propose(LocallyProposedValue(v))` , and after processed by the consensus core state machine, the `Publish` effect is not emitted and a proposal message is not sent through the network. At the receiving side, consensus core waits to receive `ProposedValue(ProposedValue(v, validity))` input and when this happens it considers the proposal as complete and proceeds. The application generates this input upon receiving the full value `V` from the network. As a result, in this case value propagation is totally delegated to the application. ### Summary -To sum up, in different modes, different inputs are required to achieve the same effect as receiving -the `PROPOSAL` message in the original Tendermint algorithm. +To sum up, in different modes, different inputs are required to achieve the same effect as receiving +the `PROPOSAL` message in the original Tendermint algorithm. - In `ProposalOnly` and `ProposalAndParts`, both `Proposal(SignedProposal(x))` and `ProposedValue(ProposedValue(x, validity))` inputs are needed, with `x == V` for the former and `x == v` for the latter. - In `PartsOnly`, only `ProposedValue(ProposedValue(v, validity))` input is enough, as no explicit proposal message is sent over the network. @@ -292,8 +293,8 @@ Regardless of the mode of operation, the value that consensus operates at the pr The mode of operation is used outside the consensus driver and state machine in order to decide whether to send or accept explicit `Proposal` messages to the caller (via `Effect`). ### `TimeoutPropose` -An important consideration is that, regardless of the mode of operation, all inputs required to complete a -proposal must be received by the consensus before `timeoutPropose` expires. This timeout must be configured +An important consideration is that, regardless of the mode of operation, all inputs required to complete a +proposal must be received by the consensus before `timeoutPropose` expires. This timeout must be configured to accommodate for the time needed for a complete value propagation. This is especially important in cases where the value is large and requires longer to be propagated through the network. @@ -324,8 +325,9 @@ If the round of consensus is successful, the value `v` carried by the round's `PROPOSAL` message is the value delivered to the application as the decision for that height of consensus. -Malachite, in [`ProposalOnly` mode](#proposalonly), when the application returns a full value in +Malachite, in [`ProposalOnly` mode](#proposalonly), when the application returns a full value in `Propose(LocallyProposedValue)`, follows this approach. + > [!WARNING] > As mentioned in the [ProposalOnly](#proposalonly) section, this mode is under development and not yet fully supported in Malachite. @@ -371,22 +373,21 @@ referenced value `V` is known by the process. Therefore, a process where `v` is decided by the consensus implementation should be able to deliver the actual proposed value `V` to the application. -Malachite, when used in `ProposalAndParts` and `PartsOnly` modes, represents -a variant of this approach in which the dissemination of full values is +Malachite, when used in `ProposalAndParts` and `PartsOnly` modes, represents +a variant of this approach in which the dissemination of full values is entirely delegated to the application. ## Decision +No decision to be taken ## Status -Proposed +Accepted ## Consequences - - ### Positive ### Negative @@ -396,9 +397,12 @@ Proposed ## References * [Tendermint consensus specification][consensus-spec] +* [ADR 001 - Architecture][adr-001] +* [ADR 004 - Coroutine-Based Effect System for Consensus][adr-004] [consensus-spec]: ../../specs/consensus/README.md [consensus-code]: ../../specs/consensus/pseudo-code.md [consensus-proposals]: ../../specs/consensus/overview.md#proposals [consensus-votes]: ../../specs/consensus/overview.md#votes -[adr001]: ./adr-001-architecture.md +[adr-001]: ./adr-001-architecture.md +[adr-004]: ./adr-004-coroutine-effect-system.md From c2ae8a570aacd6499147bf07d6545fe0c0850e65 Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Wed, 26 Mar 2025 11:02:21 +0100 Subject: [PATCH 25/52] Remvove TODO --- docs/architecture/adr-003-values-propagation.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 470e3dff0..cd5b0191b 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -55,10 +55,10 @@ implementations. In this document, we focus on the core parts of the consensus implementation responsible for **Value Propagation**. -TODO: Not sure where to mention this but seems relevant. There have been some slight changes to the Tendermint consensus: -- `getValue()` is asynchronous and the propose timeout is passed as a parameter in order to let the builder know how long it has to build a value. -- `valid(v)` is checked when a `PROPOSAL` is available but before the algorithm runs. This is equivalent in a way with `PROPOSAL(h, r, v, vr, valid)` where valid = {Valid | Invalid}. -- `upon ⟨PROPOSAL, h, r, v, vr⟩ from proposer(h, r)` represents the receiving of a `PROPOSAL` message for a round `r` of height `h` for vale `v`. This is a necessary condition for the success of round `v` with the decision of the proposed value `v`. +Malachite includes the following changes to the Tendermint consensus: +- `getValue()` is asynchronous and the propose timeout is passed as a parameter in order to let the value (`V`) builder know how long it has to create it. +- `valid(v)` is checked when a `PROPOSAL` is available but before the algorithm runs. This is equivalent in a way with `PROPOSAL(h, r, v, vr, valid)` where `valid = {true | false}` with `valid(v)` already checked. + ## Current design From 799f4c50bc9dd362291ac5693cca3f46a69e5106 Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Wed, 26 Mar 2025 11:24:06 +0100 Subject: [PATCH 26/52] Show streaming before LocallyProposedValue, add own Proposal msg --- .../adr-003-values-propagation.md | 20 +++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index cd5b0191b..19c76f962 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -167,6 +167,7 @@ sequenceDiagram E1->>A1: GetValue() A1->>E1: Propose(LocallyProposedValue(V)) E1->>C1: Propose(LocallyProposedValue(V)) + C1->>C1: Proposal(SignedProposal(V)) C1->>E1: Effect::Publish(SignedProposal(V)) E1->>E2: Proposal(SignedProposal(V)) @@ -221,16 +222,19 @@ sequenceDiagram C1->>E1: Effect::GetValue() E1->>A1: GetValue() + A1->>E2: Full value V + E2->>A2: Full value V + A1->>E1: Propose(LocallyProposedValue(v)) E1->>C1: Propose(LocallyProposedValue(v)) + C1->>C1: Proposal(SignedProposal(V)) C1->>E1: Effect::Publish(SignedProposal(v)) - E1->>E2: Proposal(SignedProposal(v)) - E2->>C2: Proposal(SignedProposal(v)) - A1->>E2: Full value V - E2->>A2: Full value V A2->>E2: ProposedValue(ProposedValue(v, validity)) E2->>C2: ProposedValue(ProposedValue(v, validity)) + E1->>E2: Proposal(SignedProposal(v)) + + E2->>C2: Proposal(SignedProposal(v)) Note over C2: Has v and its validity → can proceed ``` @@ -264,14 +268,18 @@ sequenceDiagram C1->>E1: Effect::GetValue() E1->>A1: GetValue() + + A1->>E2: Full value V + E2->>A2: Full value V + A1->>E1: Propose(LocallyProposedValue(v)) E1->>C1: Propose(LocallyProposedValue(v)) + C1->>C1: Proposal(SignedProposal(V)) C1--xE1: Effect::Publish(SignedProposal(v)) - A1->>E2: Full value V - E2->>A2: Full value V A2->>E2: ProposedValue(ProposedValue(v, validity)) E2->>C2: ProposedValue(ProposedValue(v, validity)) + C2->>C2: Proposal(SignedProposal(v)) Note over C2: Has v and its validity → can proceed ``` From 0e63684acbf25a366a27506bf81d2c2cd1f10b83 Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Wed, 26 Mar 2025 11:45:40 +0100 Subject: [PATCH 27/52] Rearange receiver boxes --- docs/architecture/adr-003-values-propagation.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 19c76f962..923bc1132 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -158,9 +158,9 @@ sequenceDiagram end box Other nodes - participant C2 as Consensus Core - participant E2 as Consensus Engine participant A2 as Application + participant E2 as Consensus Engine + participant C2 as Consensus Core end C1->>E1: Effect::GetValue() @@ -171,9 +171,9 @@ sequenceDiagram C1->>E1: Effect::Publish(SignedProposal(V)) E1->>E2: Proposal(SignedProposal(V)) - E2->>C2: Proposal(SignedProposal(V)) E2-->>A2: Proposal(V) A2-->>C2: ProposedValue(ProposedValue(V, validity)) + E2->>C2: Proposal(SignedProposal(V)) Note over C2: Has V _and its validity_ → can proceed @@ -214,9 +214,9 @@ sequenceDiagram end box Other nodes - participant C2 as Consensus Core - participant E2 as Consensus Engine participant A2 as Application + participant E2 as Consensus Engine + participant C2 as Consensus Core end C1->>E1: Effect::GetValue() @@ -261,9 +261,9 @@ sequenceDiagram end box Other nodes - participant C2 as Consensus Core - participant E2 as Consensus Engine participant A2 as Application + participant E2 as Consensus Engine + participant C2 as Consensus Core end C1->>E1: Effect::GetValue() From d85bb5dff42ae95ae0116e8ae64f04492fb82214 Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Wed, 26 Mar 2025 07:22:12 -0400 Subject: [PATCH 28/52] Apply suggestions from code review Co-authored-by: nenadmilosevic95 <50905385+nenadmilosevic95@users.noreply.github.com> Signed-off-by: Anca Zamfir --- docs/architecture/adr-003-values-propagation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 923bc1132..3999d6cd8 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -186,7 +186,7 @@ value `V` received via `LocallyProposedValue(V)`, and generates a `Publish` effect. This effect is handled by the consensus engine, and the proposal, with the full value embedded in it, is disseminated through the network. -Upon receiving a `Proposal(SignedProposal(V))` message from the network, the engine passes it directly to the consensus core for processing. +Upon receiving a `Proposal(SignedProposal(V))` message from the network, the engine passes it directly to the consensus core for processing. Consensus core verifies the proposal is properly signed by the Proposer for the current height and round. _(Implementation in progress) The consensus engine (or core?) will also pass the unsigned `Proposal(V)` message to the application for validation. Once validation is performed the application generates the `ProposedValue(V, validity)` input and pass it to the consensus core._ From 0e8ba53688a03342c3ec7b560e2089d035394728 Mon Sep 17 00:00:00 2001 From: Adi Seredinschi Date: Wed, 26 Mar 2025 17:03:26 +0100 Subject: [PATCH 29/52] initial pass, ~50% done --- .../adr-003-values-propagation.md | 139 +++++++++--------- 1 file changed, 72 insertions(+), 67 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 3999d6cd8..bc1fba0c6 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -5,38 +5,40 @@ * 2025-03-18: Context and description of the problem * 2025-03-21: Current design description +## Summary + ## Context Malachite implements a consensus algorithm, [Tendermint][consensus-spec], -which allow processes to agree on a single decided value for each +which allows nodes to agree on a single *decision value* for each consensus instance or *height*. -Possible decision values are provided by the software that uses the consensus, -which we refer to generically as the *application*. There are -no assumptions about what a **value** represents—its semantics are defined -entirely by the application. For example, in blockchain applications, input -values are blocks proposed to be appended to the blockchain. +The software on top of Malachite provides the possible decision values; +we call this the *application*. There are +no assumptions about what a **value** represents—the application provides the semantics. +For example, in blockchain applications, input +values are blocks proposed to be appended to a blockchain. Similarly, the consensus algorithm makes no assumptions about the **size** -of the proposed values; they may be of arbitrary byte size. However, +of proposed values; they may be of arbitrary byte size. However, the application is expected to define a maximum size for proposed values and to configure the consensus parameters accordingly—most notably, -the durations of timeouts. +the durations of consensus timeouts. When value size becomes a relevant factor, it is important to recognize that the consensus process comprises two distinct stages: -- **Value Propagation**: Proposed values must be transmitted to all consensus participants. -- **Value Decision**: One of the successfully propagated values is selected and decided. +* **Value Propagation**: Proposed values must be transmitted to all consensus participants, eg system nodes. +* **Value Decision**: The consensus algorithm selects and decides on one of the successfully propagated values. The cost of **Value Propagation**, in terms of latency and bandwidth, clearly depends on the size of the proposed values. In contrast, the -**Value Decision** stage should be independent of value size and -incur a roughly constant cost. +**Value Decision** stage is in principle independent of value size and +incurs roughly a constant cost from the perspective of values. -In Tendermint, **Value Propagation** is performed via the `PROPOSAL` -message, which is broadcast by the designated proposer of the round -and includes the proposed value `v`. +In Tendermint, **Value Propagation** is performed by default via the +`PROPOSAL` message. The proposer node broadcasts this message, which +includes the proposed value `v`. The **Value Decision** phase involves `PREVOTE` and `PRECOMMIT` messages—collectively referred to as *votes*. Each vote includes @@ -44,45 +46,54 @@ either an identifier `id(v)` of the proposed value `v` or the special value `nil`. The function `id(v)` provides a compact representation of `v`, typically implemented as a fixed-size hash. -From this, we can see that **Value Propagation** is more challenging, -as it involves disseminating potentially large amounts of data through -the network. In contrast, the **Value Decision** phase requires only +### Problem statement + +From the above, we can observe that **Value Propagation** is more challenging, +as it involves disseminating potentially large amounts of data in +the network of system nodes. In contrast, the **Value Decision** phase requires only the transmission of vote messages, which are relatively small and of -constant size. As a result, Malachite’s low-level API provides greater -flexibility in the value propagation stage to enable more optimized -implementations. +constant size. -In this document, we focus on the core parts of the consensus -implementation responsible for **Value Propagation**. +Because value propagation can be a bottleneck, Malachite’s consensus API has to provide +enough flexibility so that applications can optimize the value propagation stage. -Malachite includes the following changes to the Tendermint consensus: -- `getValue()` is asynchronous and the propose timeout is passed as a parameter in order to let the value (`V`) builder know how long it has to create it. -- `valid(v)` is checked when a `PROPOSAL` is available but before the algorithm runs. This is equivalent in a way with `PROPOSAL(h, r, v, vr, valid)` where `valid = {true | false}` with `valid(v)` already checked. +### ADR scope +This ADR documents the design of the Malachite consensus API relevant for +the **Value Propagation** stage. Specifically, how Malachite enables +application builders flexibility for bypassing or mitigating the performance +implications of dealing with value propagation in a large system of nodes. -## Current design +## Decision ### Building Blocks There are a number of entities that are involved in the value propagation process: -- **Application:** The software that uses the consensus engine. It is responsible for: - - Providing the value to be propagated - - Validating received values - - Providing data availability for values (both undecided and decided) -- **Consensus Engine:** The component responsible for managing the interactions between the consensus core and the application, and between the consensus core and the networking layer. For the ones involved in the value propagation process, the consensus engine is responsible for: - - Propagating values and/or proposals via the networking layer - - Receiving values and/or proposals from the networking layer - - Relaying values and/or proposals to the consensus core -- **Consensus Core:** The component that implements the tendermint consensus protocol. -- **Networking:** The component responsible for transmitting messages between nodes (currently not shown in the diagrams). +* **Application:** The software that uses the consensus engine. It is responsible for: + * Providing the value to be propagated + * Validating received values + * Providing data availability for values (both undecided and decided) +* **Consensus Engine:** The component responsible for managing the interactions between the consensus core and the application, and between the consensus core and the networking layer. For the ones involved in the value propagation process, the consensus engine is responsible for: + * Propagating values and/or proposals via the networking layer + * Receiving values and/or proposals from the networking layer + * Relaying values and/or proposals to the consensus core (pending redesign in #943) +* **Consensus Core:** The component in Malachite that implements the Tendermint consensus protocol. +* **Networking:** The component responsible for transmitting messages between nodes (currently not shown in the diagrams). + +Malachite provides implementations for the consensus core, engine and networking. Applications can integrate with Malachite via various interfaces: + +* Consensus core - See [ADR-004][adr-004] +* Engine without networking - [Snapchain](https://github.com/farcasterxyz/snapchain) +* Engine with networking (channel-based) - [Example application](https://github.com/informalsystems/malachite/tree/main/code/examples/channel) +* Engine with networking (actor-based) - [Starknet test application](https://github.com/informalsystems/malachite/tree/main/code/crates/starknet) -Malachite provides implementations for the consensus core, engine and networking. Applications can integrate with: +#### Changes to Tendermint -- Consensus core - See [ADR-004][adr-004] -- Engine without networking - [Snapchain](https://github.com/farcasterxyz/snapchain) -- Engine with networking (channel-based) - [Example application](https://github.com/informalsystems/malachite/tree/main/code/examples/channel) -- Engine with networking (actor-based) - [Starknet test application](https://github.com/informalsystems/malachite/tree/main/code/crates/starknet) +Malachite deviates from the vanilla Tendermint consensus in these two key methods: + +* `getValue()` is asynchronous and the propose timeout is passed as a parameter in order to let the value (`V`) builder know how long it has to create it. +* `valid(v)` is checked when a `PROPOSAL` is available but before the algorithm runs. This is equivalent in a way with `PROPOSAL(h, r, v, vr, valid)` where `valid = {true | false}` with `valid(v)` already checked. ### Value Payload Modes @@ -109,20 +120,18 @@ pub struct Params { ``` -In the following, we examine each approach to understand how the consensus +In the following, we examine each approach to understand how consensus core interacts with the environment, depending on the mode of operation adopted. Specifically, we focus on the core consensus inputs related to value propagation. -In general consensus inputs are how the consensus reacts to the events from -the environment. A complete overview of all inputs processes by the consensus +In general consensus inputs are how the core reacts to the events from +the environment. A complete overview of all inputs processes by consensus core can be found in [ADR-004 Coroutine-Based Effect System for Consensus][adr-004]. - | **Input** | **Fields** | **Description** | |-----------|------------|-----------------| -| `Proposal (SignedProposal)` | `height`, `round`, `value`, `value_origin` | This input is generated and passed to the consensus when a proposal consensus message is received from the network. | -| `Propose (LocallyProposedValue)` | `height`, `round`, `value` | This input is produced by the application in response to a consensus request (`getValue()`) for a value to propose. | -| `ProposedValue (ProposedValue, ValueOrigin)` | `height`, `round`, `valid_round`, `proposer`, `value`, `validity` | This input is also generated by the application when the application is responsible for disseminating values through the network. It informs the consensus that a proposedvalue has been received and validated. | - +| `Proposal (SignedProposal)` | `height`, `round`, `value`, `value_origin` | This input is generated and passed to consensus core when a proposal message is received from the network. | +| `Propose (LocallyProposedValue)` | `height`, `round`, `value` | This input is produced by the application in response to a request (`getValue()` specifically) for a value to propose. | +| `ProposedValue (ProposedValue, ValueOrigin)` | `height`, `round`, `valid_round`, `proposer`, `value`, `validity` | This input is also generated by the application when the application is responsible for disseminating values through the network. It informs the consensus core that a proposed value has been received and validated. | When processing each of these inputs, the consensus core may produce various effects that must be handled by the "environment" to fully process the input. All the effects produced by the consensus core are described in more detail in [ADR-004][adr-004]. These are especially relevant if the application integrates directly with the consensus core. Malachite offers a "Consensus Engine" crate that can be used as an integration point. Regardless the of the type of integration, the "Consensus Engine" is shown in this document as the part of the "environment" that handles the effects, relays messages between consensus core and the application, etc. @@ -135,8 +144,9 @@ The concrete value type is defined by the context `Ctx` and is passed to consens The value type must implement the `Value` trait, including the `id()` method. Consensus core uses the `id()` when generating the votes. The following notations are used in the following sections: -- `V` denotes the full application value, this for example can be a block in a blockchain application -- `v` is a short representation of the value `V`, it may be for example the hash of the value. + +* `V` denotes the full application value, this for example can be a block in a blockchain application +* `v` is a short representation of the value `V`, it may be for example the hash of the value. ### ProposalOnly @@ -189,7 +199,7 @@ with the full value embedded in it, is disseminated through the network. Upon receiving a `Proposal(SignedProposal(V))` message from the network, the engine passes it directly to the consensus core for processing. Consensus core verifies the proposal is properly signed by the Proposer for the current height and round. -_(Implementation in progress) The consensus engine (or core?) will also pass the unsigned `Proposal(V)` message to the application for validation. Once validation is performed the application generates the `ProposedValue(V, validity)` input and pass it to the consensus core._ +*(Implementation in progress) The consensus engine (or core?) will also pass the unsigned `Proposal(V)` message to the application for validation. Once validation is performed the application generates the `ProposedValue(V, validity)` input and pass it to the consensus core.* In this setup, the application only needs to provide a value to the consensus core through `Propose(LocallyProposedValue(V))`, and value propagation is entirely handled by the networking module. The consensus core only processes proposal messages that already contain the full value. @@ -204,7 +214,6 @@ The application communicates to consensus that a value is available using a refe In order to handle restarts the application should persist the undecided values `V`. - ```mermaid sequenceDiagram box Proposer @@ -291,22 +300,23 @@ the `Publish` effect is not emitted and a proposal message is not sent through t At the receiving side, consensus core waits to receive `ProposedValue(ProposedValue(v, validity))` input and when this happens it considers the proposal as complete and proceeds. The application generates this input upon receiving the full value `V` from the network. As a result, in this case value propagation is totally delegated to the application. ### Summary + To sum up, in different modes, different inputs are required to achieve the same effect as receiving the `PROPOSAL` message in the original Tendermint algorithm. -- In `ProposalOnly` and `ProposalAndParts`, both `Proposal(SignedProposal(x))` and `ProposedValue(ProposedValue(x, validity))` inputs are needed, with `x == V` for the former and `x == v` for the latter. -- In `PartsOnly`, only `ProposedValue(ProposedValue(v, validity))` input is enough, as no explicit proposal message is sent over the network. +* In `ProposalOnly` and `ProposalAndParts`, both `Proposal(SignedProposal(x))` and `ProposedValue(ProposedValue(x, validity))` inputs are needed, with `x == V` for the former and `x == v` for the latter. +* In `PartsOnly`, only `ProposedValue(ProposedValue(v, validity))` input is enough, as no explicit proposal message is sent over the network. Regardless of the mode of operation, the value that consensus operates at the proposal level is defined by the application in the `Value` trait concrete implementation. The mode of operation is used outside the consensus driver and state machine in order to decide whether to send or accept explicit `Proposal` messages to the caller (via `Effect`). ### `TimeoutPropose` + An important consideration is that, regardless of the mode of operation, all inputs required to complete a proposal must be received by the consensus before `timeoutPropose` expires. This timeout must be configured to accommodate for the time needed for a complete value propagation. This is especially important in cases where the value is large and requires longer to be propagated through the network. - ## Value Propagation Considerations This section presents a (possibly not comprehensive) list of approaches to @@ -339,7 +349,6 @@ Malachite, in [`ProposalOnly` mode](#proposalonly), when the application returns > [!WARNING] > As mentioned in the [ProposalOnly](#proposalonly) section, this mode is under development and not yet fully supported in Malachite. - ### Consensus by Reference In this approach, the application is responsible for implementing the @@ -385,21 +394,20 @@ Malachite, when used in `ProposalAndParts` and `PartsOnly` modes, represents a variant of this approach in which the dissemination of full values is entirely delegated to the application. - -## Decision - -No decision to be taken - ## Status -Accepted +Accepted and implemented as of `ec9c421` (March 2025). ## Consequences ### Positive +* The three modes **ProposalOnly**, **PartsOnly**, and **ProposalAndParts** provide application builders with flexibility + ### Negative +* Codebase and interactions may be unnecessarily complex or difficult to udnerstand. + ### Neutral ## References @@ -409,8 +417,5 @@ Accepted * [ADR 004 - Coroutine-Based Effect System for Consensus][adr-004] [consensus-spec]: ../../specs/consensus/README.md -[consensus-code]: ../../specs/consensus/pseudo-code.md -[consensus-proposals]: ../../specs/consensus/overview.md#proposals -[consensus-votes]: ../../specs/consensus/overview.md#votes [adr-001]: ./adr-001-architecture.md [adr-004]: ./adr-004-coroutine-effect-system.md From d7a0d06be70463287c3da30e7b5bcb3c1513e566 Mon Sep 17 00:00:00 2001 From: Nenad Milosevic Date: Wed, 26 Mar 2025 17:37:25 +0100 Subject: [PATCH 30/52] pass on value propagation considerations section --- .../adr-003-values-propagation.md | 106 ++++++++---------- 1 file changed, 47 insertions(+), 59 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 3999d6cd8..37ac700f6 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -315,75 +315,63 @@ Tendermint in particular, discussing the pros and cons of each of them. ### Consensus by Value -In this approach, the consensus implementation plays both the -**Value Propagation** and **Value Decision** roles. +In this approach, the actual value that will eventually be delivered to +the application is disseminated *through the consensus protocol itself*. -This means that a `PROPOSAL(h, r, v, vr)` consensus message broadcast by a -process carries its proposed value `v`. -Other processes learn the proposed value `v` when they receive the associated -`PROPOSAL` message. +Specifically, the `PROPOSAL(h, r, v, vr)` message broadcast by a process +carries the full value `v` being proposed. Other processes learn `v` by +receiving this message. Since vote messages (`PREVOTE`, `PRECOMMIT`) only +include a compact identifier `id(v)`, a process cannot vote for a value +unless it has first received the full value `v` via the `PROPOSAL` message. -As previously discussed, the vote messages (`PREVOTE` and `PRECOMMIT`) do not -carry the proposed value `v`, but a short representation `id(v)` of it. -A process cannot sign a vote for `id(v)` if it does not know the value `v`. -So receiving `PROPOSAL` message carrying the value `v` is a requirement for -signing a vote for `id(v)`. +If the consensus round is successful, the value `v` carried in the round’s +`PROPOSAL` message is delivered to the application as the decided value for +that height. -If the round of consensus is successful, the value `v` carried by the round's -`PROPOSAL` message is the value delivered to the application as the decision -for that height of consensus. +This mode ensures that value dissemination and consensus are tightly coupled: +consensus only progresses on values that are known to all participants, as +`v` must be received to vote. -Malachite, in [`ProposalOnly` mode](#proposalonly), when the application returns a full value in -`Propose(LocallyProposedValue)`, follows this approach. +Malachite follows this approach in [`ProposalOnly` mode](#proposalonly), +when the application returns the full value directly in `Propose(LocallyProposedValue)`. > [!WARNING] > As mentioned in the [ProposalOnly](#proposalonly) section, this mode is under development and not yet fully supported in Malachite. - ### Consensus by Reference -In this approach, the application is responsible for implementing the -**Value Propagation** stage, -while the consensus algorithm implements **Value Decision** stage. - -This means that a `PROPOSAL(h, r, v, vr)` consensus message broadcast by a -process does not carry the value proposed by this process. -The value `v` ordered by the consensus algorithm is instead a reference, a -description, or an identifier of the value actually proposed by the process, -whose propagation is a responsibility of the application. - -So if a process wants to propose a value `V` using this approach, it has: -(i) to propagate `V` to all processes, then (ii) produce a reference `v` of the -proposed value `V` and provide `v` to the consensus implementation. -On the receive side, a process that receives a `PROPOSAL` carrying `v` should -ensure that the referenced value `V` has been received as well. -Only in this case, the process can deliver the `PROPOSAL` for `v` to the -consensus implementation. - -Since the values that are proposed and decided by consensus are references to -actually proposed values, `v` is expected to be a short representation of `V`. -For this reason, the optimization of having vote messages carrying `id(v)` -instead of `v` becomes pretty much irrelevant. - -If the round of consensus is successful, the reference `v` carried by the -round's `PROPOSAL` message is the value delivered by the consensus -implementation as the decision for that height. -But the actual decision value for that height of consensus is `V`, the value -referenced by `v`. -It is `V` and not `v` that should be delivered to the application. - -Notice, however, that a value can only be decided by the consensus -implementation if a `PROPOSAL` message carrying that value was previously -decided. -As already mentioned, in this approach, a `PROPOSAL` message carrying a -reference `v` can only be delivered to the consensus implementation if the -referenced value `V` is known by the process. -Therefore, a process where `v` is decided by the consensus implementation -should be able to deliver the actual proposed value `V` to the application. - -Malachite, when used in `ProposalAndParts` and `PartsOnly` modes, represents -a variant of this approach in which the dissemination of full values is -entirely delegated to the application. +In this approach, the value disseminated by the consensus protocol is not +the actual full value `V`, but a *reference* to it—such as a hash or identifier `v`. +The responsibility of propagating the full value `V` is delegated to the application +or an external component. + +Here, the `PROPOSAL(h, r, v, vr)` message carries only the identifier `v`, +not the full value `V`. To propose a value, a process must: +1. Disseminate `V` to other nodes through a separate mechanism (e.g., application layer), and +2. Provide the reference `v` to the consensus protocol for inclusion in the `PROPOSAL`. + +On the receiving side, a process that gets a `PROPOSAL` carrying reference `v` +must *ensure* that the corresponding value `V` is available before voting for +the proposal. Importantly, consensus does *not* need to have +the full value `V` locally in order to proceed—only assurance that the full value +is available or will be available is required. + +Since consensus proceeds on references rather than values, the benefit of encoding +votes with `id(v)` becomes less significant, as `v` is already a compact identifier. + +If the consensus round succeeds, the value `v` is what is decided at the consensus layer. +However, the actual value delivered to the application is the corresponding full +value `V` that `v` refers to. + +Malachite, in `ProposalAndParts` and `PartsOnly` modes, represent a variant of this approach. +In these modes, the responsibility for disseminating full values is entirely delegated to +the application. When the application provides a `ProposedValue(v)`, it signals to the +consensus layer that it possesses the full value `V` corresponding to the identifier `v`, +allowing the consensus logic to proceed safely. + +Since value dissemination is not handled by the consensus protocol but explicitly by the application, +the consensus module only decides on the identifier `v` and informs the application of this decision. +It is then the application's responsibility to match `v` to the full value `V` and deliver `V` accordingly. ## Decision From bc6b588f717967f046672c0ab35be136c6d2b44f Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Wed, 26 Mar 2025 17:42:24 +0100 Subject: [PATCH 31/52] Generate proposal for PartsOnly in the engine --- docs/architecture/adr-003-values-propagation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 4f9361486..c9c0caf43 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -181,9 +181,9 @@ sequenceDiagram C1->>E1: Effect::Publish(SignedProposal(V)) E1->>E2: Proposal(SignedProposal(V)) - E2-->>A2: Proposal(V) - A2-->>C2: ProposedValue(ProposedValue(V, validity)) E2->>C2: Proposal(SignedProposal(V)) + C2-->>A2: Proposal(V) + A2-->>C2: ProposedValue(ProposedValue(V, validity)) Note over C2: Has V _and its validity_ → can proceed From af34a6268701b9129d5737fea3dd91180b383e0b Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Wed, 26 Mar 2025 17:47:44 +0100 Subject: [PATCH 32/52] Explicit requirements for PartsOnly --- docs/architecture/adr-003-values-propagation.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index c9c0caf43..6e935bacb 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -257,7 +257,9 @@ This confirmation is delivered via the `ProposedValue(ProposedValue(v, validity) ### PartsOnly -The same requirements as for `ProposalAndParts` apply. The only difference is that the application must also include `valid_round` in the dissemination protocol so that consensus core can distinguish between L22 and L28 in the consensus algorithm. +In this mode of operation, the application is responsible to define and implement the dissemination protocol for the full value `V` and its metadata (`height`, `round` and possibly `valid_round`). It is expected that the application splits the value `V` into parts, signs each part individually, and disseminates them throughout the network. At the receiving end, the application should verify that the parts are properly signed by the Proposer for the `height` and `round` as derived from the parts and reassemble the full value `V`. +The application communicates to consensus that a value is available using a reference `v` to `V`. `v` is expected to be short representation of `V` and a possible (but not mandatory) implementation for `id` is `id(v) = v`. +The application must also include `valid_round` in the dissemination protocol so that consensus core can distinguish between L22 and L28 in the consensus algorithm. In addition the application doesn't need to define a `Proposal` protocol message as one is not used. The reason is that all the information that would be carried in such a message can be derived from `V` and its metadata. From 23093ce80a2bc5330d26db92f38097dfe4f3c0ad Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Wed, 26 Mar 2025 18:02:27 +0100 Subject: [PATCH 33/52] Add reference to tendermint consensus algo changes --- docs/architecture/adr-003-values-propagation.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 6e935bacb..c486c8b62 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -92,8 +92,8 @@ Malachite provides implementations for the consensus core, engine and networking Malachite deviates from the vanilla Tendermint consensus in these two key methods: -* `getValue()` is asynchronous and the propose timeout is passed as a parameter in order to let the value (`V`) builder know how long it has to create it. -* `valid(v)` is checked when a `PROPOSAL` is available but before the algorithm runs. This is equivalent in a way with `PROPOSAL(h, r, v, vr, valid)` where `valid = {true | false}` with `valid(v)` already checked. +* `getValue()` is asynchronous and the propose timeout is passed as a parameter in order to let the value (`V`) builder know how long it has to create it. See [Malachite Async GetValue][getvalue-changes] for more details. +* `valid(v)` is checked when a `PROPOSAL` is available but before the algorithm runs. This is equivalent in a way with `PROPOSAL(h, r, v, vr, valid)` where `valid = {true | false}` with `valid(v)` already checked. See [Malachite Validity Checks][validity-changes] for more details. ### Value Payload Modes @@ -404,9 +404,13 @@ Accepted and implemented as of `ec9c421` (March 2025). ## References * [Tendermint consensus specification][consensus-spec] +* [Malachite Async GetValue][getvalue-changes] +* [Malachite Validity Checks][validity-changes] * [ADR 001 - Architecture][adr-001] * [ADR 004 - Coroutine-Based Effect System for Consensus][adr-004] [consensus-spec]: ../../specs/consensus/README.md +[getvalue-changes]: ../../specs/consensus/design.md#asynchronous-getvalue-and-proposevaluev +[validity-changes]: ../../specs/consensus/design.md#validity-checks [adr-001]: ./adr-001-architecture.md [adr-004]: ./adr-004-coroutine-effect-system.md From aa5742135e3b5e73fed5c9160f790ff63e14a810 Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Wed, 26 Mar 2025 18:07:40 +0100 Subject: [PATCH 34/52] Spelling --- docs/architecture/adr-003-values-propagation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index c486c8b62..b1e92226a 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -397,7 +397,7 @@ Accepted and implemented as of `ec9c421` (March 2025). ### Negative -* Codebase and interactions may be unnecessarily complex or difficult to udnerstand. +* Codebase and interactions may be unnecessarily complex or difficult to understand. ### Neutral From ce82a678fbb9d71e9a52401238c11baeb7cafedb Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Wed, 26 Mar 2025 18:14:58 +0100 Subject: [PATCH 35/52] Small fixes --- docs/architecture/adr-003-values-propagation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index b1e92226a..98b63d482 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -357,7 +357,7 @@ the actual full value `V`, but a *reference* to it—such as a hash or identifie The responsibility of propagating the full value `V` is delegated to the application or an external component. -Here, the `PROPOSAL(h, r, v, vr)` message carries only the identifier `v`, +Here, the `PROPOSAL(h, r, v, vr)` message carries only the reference `v`, not the full value `V`. To propose a value, a process must: 1. Disseminate `V` to other nodes through a separate mechanism (e.g., application layer), and 2. Provide the reference `v` to the consensus protocol for inclusion in the `PROPOSAL`. @@ -366,7 +366,7 @@ On the receiving side, a process that gets a `PROPOSAL` carrying reference `v` must *ensure* that the corresponding value `V` is available before voting for the proposal. Importantly, consensus does *not* need to have the full value `V` locally in order to proceed—only assurance that the full value -is available or will be available is required. +is available or will be available when required. Since consensus proceeds on references rather than values, the benefit of encoding votes with `id(v)` becomes less significant, as `v` is already a compact identifier. From 369f61af1f11d933ecf4909da1137fa687f41ddb Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Wed, 26 Mar 2025 18:21:55 +0100 Subject: [PATCH 36/52] Clarify ProposalOnly for large values --- docs/architecture/adr-003-values-propagation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 98b63d482..aa6228f28 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -203,7 +203,7 @@ Consensus core verifies the proposal is properly signed by the Proposer for the In this setup, the application only needs to provide a value to the consensus core through `Propose(LocallyProposedValue(V))`, and value propagation is entirely handled by the networking module. The consensus core only processes proposal messages that already contain the full value. -This setup is the simplest, as the application is only responsible for providing the value to be ordered. However, if the value is large, the resulting proposal messages will also be large and must be propagated as such through the network. Any optimizations for value propagation, such as chunking the value into smaller parts and reassembling it on the receiving side, must be implemented at the networking level. This is because both the consensus and application remain unaware of how the value is transmitted. +This setup is the simplest, as the application is only responsible for providing the value to be ordered. However, if the value is large, the resulting proposal messages will also be large and must be propagated as such through the network. Any optimizations for value propagation, such as chunking the value into smaller parts and reassembling it on the receiving side, must be implemented outside the consensus core. This is because both the consensus and application remain unaware of how the value is transmitted. The other two modes of operation are designed to support such optimizations at the application level rather than at the network level. We will explore how this is achieved in the following sections. From 90a25d6f0f3f9bd7ab3a20c235158291fb91d1c0 Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Wed, 26 Mar 2025 18:25:14 +0100 Subject: [PATCH 37/52] Fix Engine description --- docs/architecture/adr-003-values-propagation.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index aa6228f28..c21cdebbb 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -77,7 +77,8 @@ There are a number of entities that are involved in the value propagation proces * **Consensus Engine:** The component responsible for managing the interactions between the consensus core and the application, and between the consensus core and the networking layer. For the ones involved in the value propagation process, the consensus engine is responsible for: * Propagating values and/or proposals via the networking layer * Receiving values and/or proposals from the networking layer - * Relaying values and/or proposals to the consensus core (pending redesign in #943) + * Relaying proposals to the consensus core + * Relaying values to the application * **Consensus Core:** The component in Malachite that implements the Tendermint consensus protocol. * **Networking:** The component responsible for transmitting messages between nodes (currently not shown in the diagrams). From 47953d708b180e0627e06b64094f1f33d9f2c4b1 Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Wed, 26 Mar 2025 13:27:02 -0400 Subject: [PATCH 38/52] Apply suggestions from code review Co-authored-by: Daniel Signed-off-by: Anca Zamfir --- docs/architecture/adr-003-values-propagation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index c21cdebbb..13df7ecd3 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -80,7 +80,7 @@ There are a number of entities that are involved in the value propagation proces * Relaying proposals to the consensus core * Relaying values to the application * **Consensus Core:** The component in Malachite that implements the Tendermint consensus protocol. -* **Networking:** The component responsible for transmitting messages between nodes (currently not shown in the diagrams). +* **Networking:** The component responsible for transmitting messages between nodes (not shown in the diagrams below). Malachite provides implementations for the consensus core, engine and networking. Applications can integrate with Malachite via various interfaces: From 67ef52801fac39ddb75f36cf4b7f2c96e132df2b Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Wed, 26 Mar 2025 18:33:52 +0100 Subject: [PATCH 39/52] Ref fixes --- docs/architecture/adr-003-values-propagation.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 13df7ecd3..63844545b 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -93,8 +93,8 @@ Malachite provides implementations for the consensus core, engine and networking Malachite deviates from the vanilla Tendermint consensus in these two key methods: -* `getValue()` is asynchronous and the propose timeout is passed as a parameter in order to let the value (`V`) builder know how long it has to create it. See [Malachite Async GetValue][getvalue-changes] for more details. -* `valid(v)` is checked when a `PROPOSAL` is available but before the algorithm runs. This is equivalent in a way with `PROPOSAL(h, r, v, vr, valid)` where `valid = {true | false}` with `valid(v)` already checked. See [Malachite Validity Checks][validity-changes] for more details. +* `getValue()` is asynchronous and the propose timeout is passed as a parameter in order to let the value (`V`) builder know how long it has to create it. See [Malachite async GetValue][getvalue-changes] for more details. +* `valid(v)` is checked when a `PROPOSAL` is available but before the algorithm runs. This is equivalent in a way with `PROPOSAL(h, r, v, vr, valid)` where `valid = {true | false}` with `valid(v)` already checked. See [Malachite validity checks][validity-changes] for more details. ### Value Payload Modes @@ -405,8 +405,8 @@ Accepted and implemented as of `ec9c421` (March 2025). ## References * [Tendermint consensus specification][consensus-spec] -* [Malachite Async GetValue][getvalue-changes] -* [Malachite Validity Checks][validity-changes] +* [Malachite async GetValue][getvalue-changes] +* [Malachite validity checks][validity-changes] * [ADR 001 - Architecture][adr-001] * [ADR 004 - Coroutine-Based Effect System for Consensus][adr-004] From 5db0a31a1cdcc43b8cc80a409f80f3c9736e4199 Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Wed, 26 Mar 2025 13:49:36 -0400 Subject: [PATCH 40/52] Update docs/architecture/adr-003-values-propagation.md Co-authored-by: Daniel Signed-off-by: Anca Zamfir --- docs/architecture/adr-003-values-propagation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 63844545b..6b9dd5034 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -91,7 +91,7 @@ Malachite provides implementations for the consensus core, engine and networking #### Changes to Tendermint -Malachite deviates from the vanilla Tendermint consensus in these two key methods: +Malachite deviates from the [vanilla Tendermint consensus][tendermint-code] in these two key methods, relate to proposal propagation and validation: * `getValue()` is asynchronous and the propose timeout is passed as a parameter in order to let the value (`V`) builder know how long it has to create it. See [Malachite async GetValue][getvalue-changes] for more details. * `valid(v)` is checked when a `PROPOSAL` is available but before the algorithm runs. This is equivalent in a way with `PROPOSAL(h, r, v, vr, valid)` where `valid = {true | false}` with `valid(v)` already checked. See [Malachite validity checks][validity-changes] for more details. From 77d21f25afa203be345ce44da836e226d8fd0a74 Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Wed, 26 Mar 2025 13:49:55 -0400 Subject: [PATCH 41/52] Update docs/architecture/adr-003-values-propagation.md Co-authored-by: Daniel Signed-off-by: Anca Zamfir --- docs/architecture/adr-003-values-propagation.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 6b9dd5034..e1e377caa 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -411,6 +411,7 @@ Accepted and implemented as of `ec9c421` (March 2025). * [ADR 004 - Coroutine-Based Effect System for Consensus][adr-004] [consensus-spec]: ../../specs/consensus/README.md +[tendermint-code]: ../../specs/consensus/pseudo-code.md [getvalue-changes]: ../../specs/consensus/design.md#asynchronous-getvalue-and-proposevaluev [validity-changes]: ../../specs/consensus/design.md#validity-checks [adr-001]: ./adr-001-architecture.md From a044bdb854eb64db381fc6c6c8587d48ffb2976c Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Wed, 26 Mar 2025 14:05:50 -0400 Subject: [PATCH 42/52] Apply suggestions from code review Co-authored-by: Daniel Signed-off-by: Anca Zamfir --- .../adr-003-values-propagation.md | 39 +++++++++++-------- 1 file changed, 23 insertions(+), 16 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index e1e377caa..1ab8f0a22 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -4,6 +4,7 @@ * 2025-03-18: Context and description of the problem * 2025-03-21: Current design description +* 2025-03-25: Diagrams and more detailed description of the three operation options ## Summary @@ -111,14 +112,13 @@ A specific mode can be set as a `value_payload` consensus parameter: /// Consensus parameters. #[derive_where(Clone, Debug)] pub struct Params { - ... + ... /// The messages required to deliver proposals pub value_payload: ValuePayload, ... } - ``` In the following, we examine each approach to understand how consensus core @@ -132,7 +132,7 @@ can be found in [ADR-004 Coroutine-Based Effect System for Consensus][adr-004]. |-----------|------------|-----------------| | `Proposal (SignedProposal)` | `height`, `round`, `value`, `value_origin` | This input is generated and passed to consensus core when a proposal message is received from the network. | | `Propose (LocallyProposedValue)` | `height`, `round`, `value` | This input is produced by the application in response to a request (`getValue()` specifically) for a value to propose. | -| `ProposedValue (ProposedValue, ValueOrigin)` | `height`, `round`, `valid_round`, `proposer`, `value`, `validity` | This input is also generated by the application when the application is responsible for disseminating values through the network. It informs the consensus core that a proposed value has been received and validated. | +| `ProposedValue (ProposedValue, ValueOrigin)` | `height`, `round`, `valid_round`, `proposer`, `value`, `validity` | This input is also generated by the application when the application is responsible for disseminating values through the network. It informs the consensus core that a proposed `v` value has been received and validated (`valid(v)`). | When processing each of these inputs, the consensus core may produce various effects that must be handled by the "environment" to fully process the input. All the effects produced by the consensus core are described in more detail in [ADR-004][adr-004]. These are especially relevant if the application integrates directly with the consensus core. Malachite offers a "Consensus Engine" crate that can be used as an integration point. Regardless the of the type of integration, the "Consensus Engine" is shown in this document as the part of the "environment" that handles the effects, relays messages between consensus core and the application, etc. @@ -158,7 +158,7 @@ This approach most closely follows the original Tendermint algorithm. It is expected to be used by applications that have small values to propose. The maximum value size is defined by the p2p network configuration. For example, if libp2p gossipsub is used, this will be the `max_transmit_size` configured for the gossipsub protocol. -To reduce the size of the vote messages it is recommended that `id(V)` method implementation returns a short representation of the value. +To reduce the size of the vote messages it is expected that `id(V)` method implementation returns a short representation of the proposed value `V`. ```mermaid sequenceDiagram @@ -200,20 +200,23 @@ with the full value embedded in it, is disseminated through the network. Upon receiving a `Proposal(SignedProposal(V))` message from the network, the engine passes it directly to the consensus core for processing. Consensus core verifies the proposal is properly signed by the Proposer for the current height and round. -*(Implementation in progress) The consensus engine (or core?) will also pass the unsigned `Proposal(V)` message to the application for validation. Once validation is performed the application generates the `ProposedValue(V, validity)` input and pass it to the consensus core.* +*(Implementation in progress) The consensus engine implementation should also pass the unsigned `Proposal(V)` message to the application for validation. +Once validation is performed the application generates the `ProposedValue(V, valid(V))` input and provide it as input to consensus.* -In this setup, the application only needs to provide a value to the consensus core through `Propose(LocallyProposedValue(V))`, and value propagation is entirely handled by the networking module. The consensus core only processes proposal messages that already contain the full value. +In this setup, the application only needs to provide a value to the consensus core through `Propose(LocallyProposedValue(V))`, and value propagation is entirely handled by the networking module. The consensus core processes proposal messages that already contain the proposed value `V`. This setup is the simplest, as the application is only responsible for providing the value to be ordered. However, if the value is large, the resulting proposal messages will also be large and must be propagated as such through the network. Any optimizations for value propagation, such as chunking the value into smaller parts and reassembling it on the receiving side, must be implemented outside the consensus core. This is because both the consensus and application remain unaware of how the value is transmitted. -The other two modes of operation are designed to support such optimizations at the application level rather than at the network level. We will explore how this is achieved in the following sections. +The other two modes of operation are designed to support such optimizations at the application level rather than at the network level. +We will explore how this is achieved in the following sections. ### ProposalAndParts In this mode of operation, the application is responsible to define and implement the dissemination protocol for the full value `V` and its metadata (`height`, `round` and possibly `valid_round`). It is expected that the application splits the value `V` into parts, signs each part individually, and disseminates them throughout the network. At the receiving end, the application should verify that the parts are properly signed by the Proposer for the `height` and `round` as derived from the parts and reassemble the full value `V`. -The application communicates to consensus that a value is available using a reference `v` to `V`. `v` is expected to be short representation of `V` and a possible (but not mandatory) implementation for `id` is `id(v) = v`. +The application communicates to consensus that a value is available using a reference `v` to `V`. +`v` is expected to be short representation of `V` and a possible (but not mandatory) implementation for the`id()` trait is the trivial `id(v) = v`. -In order to handle restarts the application should persist the undecided values `V`. +In order to handle restarts, the application should persist the both the produced and received values `V`. ```mermaid sequenceDiagram @@ -249,7 +252,8 @@ sequenceDiagram Note over C2: Has v and its validity → can proceed ``` -In this mode, as a response to `GetValue()`, the application generates `Propose(LocallyProposedValue(v))` input that does not contain the full value `V`, but instead carries a value reference. As a result, the proposal message generated by consensus upon processing this input includes only the value reference, +In this mode, as a response to `GetValue()`, the application generates `Propose(LocallyProposedValue(v))` input that does not contain the full value `V`, but instead carries a value reference `v`. +As a result, the `Proposal` message generated by consensus upon processing this input includes only the value reference, not the full value. Nevertheless, in this mode, the consensus core generates the `Publish` effect and the `Proposal(SignedProposal(v))` message is disseminated through the network via networking module. When a node receives the `Proposal(SignedProposal(v))` message it passes it to the consensus. However, in this mode, receiving a proposal alone is not sufficient for the consensus core to consider the proposal complete. The consensus core must also receive confirmation from the application that the corresponding full value is available and was validated. @@ -262,7 +266,9 @@ In this mode of operation, the application is responsible to define and implemen The application communicates to consensus that a value is available using a reference `v` to `V`. `v` is expected to be short representation of `V` and a possible (but not mandatory) implementation for `id` is `id(v) = v`. The application must also include `valid_round` in the dissemination protocol so that consensus core can distinguish between L22 and L28 in the consensus algorithm. -In addition the application doesn't need to define a `Proposal` protocol message as one is not used. The reason is that all the information that would be carried in such a message can be derived from `V` and its metadata. +In addition the consensus implementation does not produce and publish a `Proposal` message. +The reason is that all the information that would be carried in such a message can be derived from `V` and its metadata. +In other words, the application is able to reconstruct the `Proposal` message from the disseminated data, thus rendering the consensus-level `Proposal` message redundant. ```mermaid sequenceDiagram @@ -298,7 +304,7 @@ sequenceDiagram This mode is very similar to `ProposalandParts` but the difference is that when receiving `Propose(LocallyProposedValue(v))` , and after processed by the consensus core state machine, -the `Publish` effect is not emitted and a proposal message is not sent through the network. +the `Publish` effect is not emitted and a `Proposal` message is not sent through the network. At the receiving side, consensus core waits to receive `ProposedValue(ProposedValue(v, validity))` input and when this happens it considers the proposal as complete and proceeds. The application generates this input upon receiving the full value `V` from the network. As a result, in this case value propagation is totally delegated to the application. @@ -309,6 +315,7 @@ the `PROPOSAL` message in the original Tendermint algorithm. * In `ProposalOnly` and `ProposalAndParts`, both `Proposal(SignedProposal(x))` and `ProposedValue(ProposedValue(x, validity))` inputs are needed, with `x == V` for the former and `x == v` for the latter. * In `PartsOnly`, only `ProposedValue(ProposedValue(v, validity))` input is enough, as no explicit proposal message is sent over the network. +In other words, the Proposal propagation role is entirely up to the application. Regardless of the mode of operation, the value that consensus operates at the proposal level is defined by the application in the `Value` trait concrete implementation. The mode of operation is used outside the consensus driver and state machine in order to decide whether to send or accept explicit `Proposal` messages to the caller (via `Effect`). @@ -317,7 +324,7 @@ The mode of operation is used outside the consensus driver and state machine in An important consideration is that, regardless of the mode of operation, all inputs required to complete a proposal must be received by the consensus before `timeoutPropose` expires. This timeout must be configured -to accommodate for the time needed for a complete value propagation. This is especially important +to accommodate for the time needed for a complete value production and propagation. This is especially important in cases where the value is large and requires longer to be propagated through the network. ## Value Propagation Considerations @@ -343,7 +350,7 @@ that height. This mode ensures that value dissemination and consensus are tightly coupled: consensus only progresses on values that are known to all participants, as -`v` must be received to vote. +`v` must be received as a condition for a process to vote (on `id(v)`). Malachite follows this approach in [`ProposalOnly` mode](#proposalonly), when the application returns the full value directly in `Propose(LocallyProposedValue)`. @@ -379,11 +386,11 @@ value `V` that `v` refers to. Malachite, in `ProposalAndParts` and `PartsOnly` modes, represent a variant of this approach. In these modes, the responsibility for disseminating full values is entirely delegated to the application. When the application provides a `ProposedValue(v)`, it signals to the -consensus layer that it possesses the full value `V` corresponding to the identifier `v`, +consensus layer that it possesses the full value `V` corresponding to the reference `v`, allowing the consensus logic to proceed safely. Since value dissemination is not handled by the consensus protocol but explicitly by the application, -the consensus module only decides on the identifier `v` and informs the application of this decision. +the consensus module only decides on the reference `v` and informs the application of this decision. It is then the application's responsibility to match `v` to the full value `V` and deliver `V` accordingly. ## Status From 0370ded5a830cdaf7a441dc2e57a6c579c703766 Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Wed, 26 Mar 2025 19:14:10 +0100 Subject: [PATCH 43/52] valid_round mandatory for PartsOnly --- docs/architecture/adr-003-values-propagation.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 1ab8f0a22..059cb69c4 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -262,9 +262,8 @@ This confirmation is delivered via the `ProposedValue(ProposedValue(v, validity) ### PartsOnly -In this mode of operation, the application is responsible to define and implement the dissemination protocol for the full value `V` and its metadata (`height`, `round` and possibly `valid_round`). It is expected that the application splits the value `V` into parts, signs each part individually, and disseminates them throughout the network. At the receiving end, the application should verify that the parts are properly signed by the Proposer for the `height` and `round` as derived from the parts and reassemble the full value `V`. +In this mode of operation, the application is responsible to define and implement the dissemination protocol for the full value `V` and its metadata (`height`, `round` and `valid_round`). It is expected that the application splits the value `V` into parts, signs each part individually, and disseminates them throughout the network. At the receiving end, the application should verify that the parts are properly signed by the Proposer for the `height` and `round` as derived from the parts and reassemble the full value `V`. The application communicates to consensus that a value is available using a reference `v` to `V`. `v` is expected to be short representation of `V` and a possible (but not mandatory) implementation for `id` is `id(v) = v`. -The application must also include `valid_round` in the dissemination protocol so that consensus core can distinguish between L22 and L28 in the consensus algorithm. In addition the consensus implementation does not produce and publish a `Proposal` message. The reason is that all the information that would be carried in such a message can be derived from `V` and its metadata. From fa36fbf42db200fba676ba5ba8236271f7020900 Mon Sep 17 00:00:00 2001 From: Adi Seredinschi Date: Wed, 26 Mar 2025 21:14:06 +0100 Subject: [PATCH 44/52] second pass --- .../adr-003-values-propagation.md | 44 ++++++++++++++----- docs/architecture/readme.md | 4 +- 2 files changed, 34 insertions(+), 14 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 059cb69c4..24cd68578 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -5,8 +5,23 @@ * 2025-03-18: Context and description of the problem * 2025-03-21: Current design description * 2025-03-25: Diagrams and more detailed description of the three operation options +* 2025-03-26: Reviewed & accepted -## Summary +## Overview + +This ADR documents the current architecture that we adopted in Malachite +for handling the propagation of proposed values. +The main goal of this architecture is to +enable flexibility for application developers. +The architectural choice applies both to the core libraries +as well as to the engine. + +Propagation of proposed values among system nodes is often critical for +system performance. Hence, allowing different optimizations and modes of +propagation is an important architectural requirement for Malachite. +We discuss the three existing modes: +ProposalOnly, PartsOnly, and ProposalAndParts, documenting their +respective use-case and design. ## Context @@ -37,9 +52,10 @@ clearly depends on the size of the proposed values. In contrast, the **Value Decision** stage is in principle independent of value size and incurs roughly a constant cost from the perspective of values. -In Tendermint, **Value Propagation** is performed by default via the -`PROPOSAL` message. The proposer node broadcasts this message, which -includes the proposed value `v`. +In the original, abstract Tendermint algorithm, the `PROPOSAL` message performs +the task of **Value Propagation**. The proposer node broadcasts this message, which +includes the proposed value `v`. As the present ADR will show, different implementations +can achieve this abstract task. The **Value Decision** phase involves `PREVOTE` and `PRECOMMIT` messages—collectively referred to as *votes*. Each vote includes @@ -99,7 +115,7 @@ Malachite deviates from the [vanilla Tendermint consensus][tendermint-code] in t ### Value Payload Modes -At the moment, Malachite supports three +At the moment, the Malachite core consensus library supports three different modes of operation to handle value propagation: 1) **ProposalOnly** @@ -141,7 +157,7 @@ differ depending on the selected mode of operation. Towards this goal, in the fo ### Possible value types for Consensus Core? -The concrete value type is defined by the context `Ctx` and is passed to consensus as a type parameter. Each application using the consensus will define its own concrete value type. +The concrete value type is defined by the context `Ctx` and is passed to consensus as a type parameter. Each application using Malachite will define its own concrete value type. The value type must implement the `Value` trait, including the `id()` method. Consensus core uses the `id()` when generating the votes. The following notations are used in the following sections: @@ -203,9 +219,9 @@ Consensus core verifies the proposal is properly signed by the Proposer for the *(Implementation in progress) The consensus engine implementation should also pass the unsigned `Proposal(V)` message to the application for validation. Once validation is performed the application generates the `ProposedValue(V, valid(V))` input and provide it as input to consensus.* -In this setup, the application only needs to provide a value to the consensus core through `Propose(LocallyProposedValue(V))`, and value propagation is entirely handled by the networking module. The consensus core processes proposal messages that already contain the proposed value `V`. +In this mode, the application only needs to provide a value to the consensus core through `Propose(LocallyProposedValue(V))`, and value propagation is entirely handled by the networking module. The consensus core processes proposal messages that already contain the proposed value `V`. -This setup is the simplest, as the application is only responsible for providing the value to be ordered. However, if the value is large, the resulting proposal messages will also be large and must be propagated as such through the network. Any optimizations for value propagation, such as chunking the value into smaller parts and reassembling it on the receiving side, must be implemented outside the consensus core. This is because both the consensus and application remain unaware of how the value is transmitted. +This mode is the simplest, as the application is only responsible for providing the value to be ordered. However, if the value is large, the resulting proposal messages will also be large and must be propagated as such through the network. Any optimizations for value propagation, such as chunking the value into smaller parts and reassembling it on the receiving side, must be implemented outside the consensus core. This is because both the consensus and application remain unaware of how the value is transmitted. The other two modes of operation are designed to support such optimizations at the application level rather than at the network level. We will explore how this is achieved in the following sections. @@ -309,8 +325,8 @@ At the receiving side, consensus core waits to receive `ProposedValue(ProposedVa ### Summary -To sum up, in different modes, different inputs are required to achieve the same effect as receiving -the `PROPOSAL` message in the original Tendermint algorithm. +To sum up, different modes rely on different inputs to achieve the same effect as +the original Tendermint algorithm achieve via the `PROPOSAL` message. * In `ProposalOnly` and `ProposalAndParts`, both `Proposal(SignedProposal(x))` and `ProposedValue(ProposedValue(x, validity))` inputs are needed, with `x == V` for the former and `x == v` for the latter. * In `PartsOnly`, only `ProposedValue(ProposedValue(v, validity))` input is enough, as no explicit proposal message is sent over the network. @@ -326,7 +342,7 @@ proposal must be received by the consensus before `timeoutPropose` expires. This to accommodate for the time needed for a complete value production and propagation. This is especially important in cases where the value is large and requires longer to be propagated through the network. -## Value Propagation Considerations +## Additional considerations about value propagation This section presents a (possibly not comprehensive) list of approaches to handle **Value Propagation** for consensus protocols in general, and for @@ -351,6 +367,8 @@ This mode ensures that value dissemination and consensus are tightly coupled: consensus only progresses on values that are known to all participants, as `v` must be received as a condition for a process to vote (on `id(v)`). +#### Relevance to present ADR + Malachite follows this approach in [`ProposalOnly` mode](#proposalonly), when the application returns the full value directly in `Propose(LocallyProposedValue)`. @@ -382,6 +400,8 @@ If the consensus round succeeds, the value `v` is what is decided at the consens However, the actual value delivered to the application is the corresponding full value `V` that `v` refers to. +#### Relevance to present ADR + Malachite, in `ProposalAndParts` and `PartsOnly` modes, represent a variant of this approach. In these modes, the responsibility for disseminating full values is entirely delegated to the application. When the application provides a `ProposedValue(v)`, it signals to the @@ -404,7 +424,7 @@ Accepted and implemented as of `ec9c421` (March 2025). ### Negative -* Codebase and interactions may be unnecessarily complex or difficult to understand. +* Codebase and interactions may be complex or difficult to understand. ### Neutral diff --git a/docs/architecture/readme.md b/docs/architecture/readme.md index 3de2afa5e..02d8bab5d 100644 --- a/docs/architecture/readme.md +++ b/docs/architecture/readme.md @@ -28,5 +28,5 @@ To suggest an ADR, please make use of the [ADR template](./adr-template.md) prov | ADR \# | Description | Status | |----------------------------------------|-----------------------------------------|----------| | [001](./adr-001-architecture.md) | High Level Architecture | Accepted | -| [002](./adr-002-node-actor.md) | Node Architecture using the Actor Model | Proposed | -| [003](./adr-003-values-propagation.md) | Propagation of Proposed Values | Proposed | +| [002](./adr-002-node-actor.md) | Node Architecture using the Actor Model | Accepted | +| [003](./adr-003-values-propagation.md) | Propagation of Proposed Values | Accepted | From 2fdd9366f5e702fd57933655dc65abd0ab29cb96 Mon Sep 17 00:00:00 2001 From: Adi Seredinschi Date: Wed, 26 Mar 2025 21:23:16 +0100 Subject: [PATCH 45/52] minor rephrasing --- docs/architecture/adr-003-values-propagation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 24cd68578..0377d34db 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -4,7 +4,7 @@ * 2025-03-18: Context and description of the problem * 2025-03-21: Current design description -* 2025-03-25: Diagrams and more detailed description of the three operation options +* 2025-03-25: Diagrams and more detailed description of the three modes of operation * 2025-03-26: Reviewed & accepted ## Overview From 0d33f112261ae79bfc07f4d09a3a3c97d57a0fa5 Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Thu, 27 Mar 2025 10:36:07 +0100 Subject: [PATCH 46/52] Cleanup --- .../adr-003-values-propagation.md | 43 ++++++++++--------- 1 file changed, 22 insertions(+), 21 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 0377d34db..c1f9eae8b 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -9,7 +9,7 @@ ## Overview -This ADR documents the current architecture that we adopted in Malachite +This ADR documents the current architecture adopted in Malachite for handling the propagation of proposed values. The main goal of this architecture is to enable flexibility for application developers. @@ -19,9 +19,13 @@ as well as to the engine. Propagation of proposed values among system nodes is often critical for system performance. Hence, allowing different optimizations and modes of propagation is an important architectural requirement for Malachite. -We discuss the three existing modes: -ProposalOnly, PartsOnly, and ProposalAndParts, documenting their -respective use-case and design. + +Malachite supports three modes for value propagation: +* `ProposalOnly` +* `PartsOnly` +* `ProposalAndParts` + +Their respective use-case and design are documented in the following sections. ## Context @@ -30,10 +34,9 @@ which allows nodes to agree on a single *decision value* for each consensus instance or *height*. The software on top of Malachite provides the possible decision values; -we call this the *application*. There are -no assumptions about what a **value** represents—the application provides the semantics. -For example, in blockchain applications, input -values are blocks proposed to be appended to a blockchain. +in this document, this is referred to as the *application*. +There are no assumptions in the consensus algorithm about what a **value** represents, the application provides the semantics. +For example, in blockchain applications, input values are blocks proposed to be appended to a blockchain. Similarly, the consensus algorithm makes no assumptions about the **size** of proposed values; they may be of arbitrary byte size. However, @@ -65,14 +68,13 @@ representation of `v`, typically implemented as a fixed-size hash. ### Problem statement -From the above, we can observe that **Value Propagation** is more challenging, +From the above, it can be observed that **Value Propagation** is more challenging, as it involves disseminating potentially large amounts of data in the network of system nodes. In contrast, the **Value Decision** phase requires only the transmission of vote messages, which are relatively small and of constant size. -Because value propagation can be a bottleneck, Malachite’s consensus API has to provide -enough flexibility so that applications can optimize the value propagation stage. +Because value propagation can be a bottleneck, Malachite's consensus API has to provide enough flexibility so that applications can optimize the value propagation stage. ### ADR scope @@ -137,12 +139,11 @@ pub struct Params { } ``` -In the following, we examine each approach to understand how consensus core -interacts with the environment, depending on the mode of operation adopted. -Specifically, we focus on the core consensus inputs related to value propagation. -In general consensus inputs are how the core reacts to the events from -the environment. A complete overview of all inputs processes by consensus core -can be found in [ADR-004 Coroutine-Based Effect System for Consensus][adr-004]. +In the following sections, for each of the three modes of operation, the consensus core interactions with the environment are described. + +In general, events from the environment may trigger different inputs to the consensus core, and a complete overview of all inputs can be found in [ADR-004 Coroutine-Based Effect System for Consensus][adr-004]. + +This documents focuses on the core consensus inputs related to value propagation. | **Input** | **Fields** | **Description** | |-----------|------------|-----------------| @@ -152,7 +153,7 @@ can be found in [ADR-004 Coroutine-Based Effect System for Consensus][adr-004]. When processing each of these inputs, the consensus core may produce various effects that must be handled by the "environment" to fully process the input. All the effects produced by the consensus core are described in more detail in [ADR-004][adr-004]. These are especially relevant if the application integrates directly with the consensus core. Malachite offers a "Consensus Engine" crate that can be used as an integration point. Regardless the of the type of integration, the "Consensus Engine" is shown in this document as the part of the "environment" that handles the effects, relays messages between consensus core and the application, etc. -Here, our focus is limited to interactions related to value propagation and how they +Here, the focus is limited to interactions related to value propagation and how they differ depending on the selected mode of operation. Towards this goal, in the following sections, the `height` , `round`, `valid_round`, and `value_origin` fields are omitted from the inputs. ### Possible value types for Consensus Core? @@ -224,7 +225,7 @@ In this mode, the application only needs to provide a value to the consensus cor This mode is the simplest, as the application is only responsible for providing the value to be ordered. However, if the value is large, the resulting proposal messages will also be large and must be propagated as such through the network. Any optimizations for value propagation, such as chunking the value into smaller parts and reassembling it on the receiving side, must be implemented outside the consensus core. This is because both the consensus and application remain unaware of how the value is transmitted. The other two modes of operation are designed to support such optimizations at the application level rather than at the network level. -We will explore how this is achieved in the following sections. +The following sections describe how this is achieved. ### ProposalAndParts @@ -317,7 +318,7 @@ sequenceDiagram Note over C2: Has v and its validity → can proceed ``` -This mode is very similar to `ProposalandParts` but the difference is that when receiving +This mode is very similar to `ProposalAndParts` but the difference is that when receiving `Propose(LocallyProposedValue(v))` , and after processed by the consensus core state machine, the `Publish` effect is not emitted and a `Proposal` message is not sent through the network. @@ -359,7 +360,7 @@ receiving this message. Since vote messages (`PREVOTE`, `PRECOMMIT`) only include a compact identifier `id(v)`, a process cannot vote for a value unless it has first received the full value `v` via the `PROPOSAL` message. -If the consensus round is successful, the value `v` carried in the round’s +If the consensus round is successful, the value `v` carried in the round's `PROPOSAL` message is delivered to the application as the decided value for that height. From 753d9d659e2569bd10f0ba1e9ff0b5ce6a9e152c Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Thu, 27 Mar 2025 10:46:34 +0100 Subject: [PATCH 47/52] Review comment --- docs/architecture/adr-003-values-propagation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index c1f9eae8b..88d4cb97d 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -112,7 +112,7 @@ Malachite provides implementations for the consensus core, engine and networking Malachite deviates from the [vanilla Tendermint consensus][tendermint-code] in these two key methods, relate to proposal propagation and validation: -* `getValue()` is asynchronous and the propose timeout is passed as a parameter in order to let the value (`V`) builder know how long it has to create it. See [Malachite async GetValue][getvalue-changes] for more details. +* `getValue()` is asynchronous and the propose timeout is passed as a parameter in order to let the value (`v`) builder know how long it has to create it. See [Malachite async GetValue][getvalue-changes] for more details. * `valid(v)` is checked when a `PROPOSAL` is available but before the algorithm runs. This is equivalent in a way with `PROPOSAL(h, r, v, vr, valid)` where `valid = {true | false}` with `valid(v)` already checked. See [Malachite validity checks][validity-changes] for more details. ### Value Payload Modes From 05c95dc1a67fe4d8dd4a87e08208ec20be325333 Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Thu, 27 Mar 2025 11:16:31 +0100 Subject: [PATCH 48/52] Add restreaming section --- .../adr-003-values-propagation.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 88d4cb97d..711339083 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -324,6 +324,25 @@ the `Publish` effect is not emitted and a `Proposal` message is not sent through At the receiving side, consensus core waits to receive `ProposedValue(ProposedValue(v, validity))` input and when this happens it considers the proposal as complete and proceeds. The application generates this input upon receiving the full value `V` from the network. As a result, in this case value propagation is totally delegated to the application. +### Value Restreaming + +When operating in either `ProposalAndParts` or `PartsOnly` modes, there are cases where a proposer needs to re-propose a value that was previously seen in an earlier round. This occurs when implementing L16 of the Tendermint algorithm, where a proposer enters a new round with a valid value from a previous round. + +The restreaming flow follows a similar pattern to the initial proposal flow, but with these key differences: + +1. **Effect Generation**: + - Instead of `GetValue`, the consensus core generates a `RestreamProposal` effect + - This signals to the application that it should re-disseminate an existing value rather than generate a new one + +2. **Proposer Behavior**: + - The consensus core already has the valid value from the previous round + - Unlike the initial proposal flow, it does not need to wait for a `LocallyProposedValue` input + - The application only needs to handle re-dissemination of the value parts through the network + +3. **Non-Proposer Behavior**: + - If a non-proposer node has already validated this value in a previous round, it can skip waiting for the `ProposedValue` input + - This optimization helps reduce unnecessary validation work and speeds up consensus + ### Summary To sum up, different modes rely on different inputs to achieve the same effect as From c33540f6d53205d987c3d07d9c557510b05f39df Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Thu, 27 Mar 2025 11:57:56 +0100 Subject: [PATCH 49/52] Review comments - correct restreaming behavior at non-proposer --- docs/architecture/adr-003-values-propagation.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 711339083..dbf52424a 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -340,8 +340,7 @@ The restreaming flow follows a similar pattern to the initial proposal flow, but - The application only needs to handle re-dissemination of the value parts through the network 3. **Non-Proposer Behavior**: - - If a non-proposer node has already validated this value in a previous round, it can skip waiting for the `ProposedValue` input - - This optimization helps reduce unnecessary validation work and speeds up consensus + - In `ProposalAndParts` mode, if a non-proposer node has already validated this value in a previous round, it can skip waiting for the `ProposedValue` input and proceed when the `Proposal` message is received. ### Summary From 7be2b78cb9a213097c77103c74aae7271da3b430 Mon Sep 17 00:00:00 2001 From: Anca Zamfir Date: Thu, 27 Mar 2025 12:06:41 +0100 Subject: [PATCH 50/52] Review comments --- docs/architecture/adr-003-values-propagation.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index dbf52424a..1e29b083a 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -330,16 +330,14 @@ When operating in either `ProposalAndParts` or `PartsOnly` modes, there are case The restreaming flow follows a similar pattern to the initial proposal flow, but with these key differences: -1. **Effect Generation**: +1. **Proposer Node Behavior**: - Instead of `GetValue`, the consensus core generates a `RestreamProposal` effect - This signals to the application that it should re-disseminate an existing value rather than generate a new one - -2. **Proposer Behavior**: - The consensus core already has the valid value from the previous round - Unlike the initial proposal flow, it does not need to wait for a `LocallyProposedValue` input - The application only needs to handle re-dissemination of the value parts through the network -3. **Non-Proposer Behavior**: +2. **Other Node Behavior**: - In `ProposalAndParts` mode, if a non-proposer node has already validated this value in a previous round, it can skip waiting for the `ProposedValue` input and proceed when the `Proposal` message is received. ### Summary From e9aff4cd282db995c4121aa065e987df4c2d9965 Mon Sep 17 00:00:00 2001 From: nenadmilosevic95 <50905385+nenadmilosevic95@users.noreply.github.com> Date: Thu, 27 Mar 2025 12:30:40 +0100 Subject: [PATCH 51/52] small updates Signed-off-by: nenadmilosevic95 <50905385+nenadmilosevic95@users.noreply.github.com> --- docs/architecture/adr-003-values-propagation.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 1e29b083a..5912cf4b5 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -326,19 +326,20 @@ At the receiving side, consensus core waits to receive `ProposedValue(ProposedVa ### Value Restreaming -When operating in either `ProposalAndParts` or `PartsOnly` modes, there are cases where a proposer needs to re-propose a value that was previously seen in an earlier round. This occurs when implementing L16 of the Tendermint algorithm, where a proposer enters a new round with a valid value from a previous round. +When operating in either `ProposalAndParts` or `PartsOnly` mode, there are situations where the proposer must re-propose a value observed in a previous round. This scenario arises when implementing line 16 of the Tendermint algorithm, where the proposer enters a new round already holding a valid value from an earlier round. -The restreaming flow follows a similar pattern to the initial proposal flow, but with these key differences: +> **Note:** The application is responsible for storing all previously received propagated values from earlier rounds to support such re-proposals. + +The restreaming flow mirrors the initial proposal flow, with the following key differences: 1. **Proposer Node Behavior**: - - Instead of `GetValue`, the consensus core generates a `RestreamProposal` effect - - This signals to the application that it should re-disseminate an existing value rather than generate a new one - - The consensus core already has the valid value from the previous round - - Unlike the initial proposal flow, it does not need to wait for a `LocallyProposedValue` input - - The application only needs to handle re-dissemination of the value parts through the network + - Instead of invoking `GetValue()`, the consensus core emits a `RestreamProposal()` effect, signaling the application to re-disseminate an existing value rather than generating a new one. + - The application is responsible for re-disseminating the parts of the value across the network. + - The consensus core does not wait for a `Propose(LocallyProposedValue(v))` input, as it already holds a valid value from the previous round. 2. **Other Node Behavior**: - - In `ProposalAndParts` mode, if a non-proposer node has already validated this value in a previous round, it can skip waiting for the `ProposedValue` input and proceed when the `Proposal` message is received. + - In `ProposalAndParts` mode, if a non-proposer node has already validated the value in a previous round, it can skip waiting for the `ProposedValue(ProposedValue(v, validity))` input, and proceed as soon as it receives the `Proposal(SignedProposal(v))` message. + ### Summary From 77cc03a73df4d81c2e11116cd2ce8270c901cc21 Mon Sep 17 00:00:00 2001 From: nenadmilosevic95 <50905385+nenadmilosevic95@users.noreply.github.com> Date: Thu, 27 Mar 2025 13:09:23 +0100 Subject: [PATCH 52/52] add `V` in consensus by value section Signed-off-by: nenadmilosevic95 <50905385+nenadmilosevic95@users.noreply.github.com> --- .../adr-003-values-propagation.md | 31 +++++++++---------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/docs/architecture/adr-003-values-propagation.md b/docs/architecture/adr-003-values-propagation.md index 5912cf4b5..b482f7bd8 100644 --- a/docs/architecture/adr-003-values-propagation.md +++ b/docs/architecture/adr-003-values-propagation.md @@ -368,22 +368,21 @@ Tendermint in particular, discussing the pros and cons of each of them. ### Consensus by Value -In this approach, the actual value that will eventually be delivered to -the application is disseminated *through the consensus protocol itself*. - -Specifically, the `PROPOSAL(h, r, v, vr)` message broadcast by a process -carries the full value `v` being proposed. Other processes learn `v` by -receiving this message. Since vote messages (`PREVOTE`, `PRECOMMIT`) only -include a compact identifier `id(v)`, a process cannot vote for a value -unless it has first received the full value `v` via the `PROPOSAL` message. - -If the consensus round is successful, the value `v` carried in the round's -`PROPOSAL` message is delivered to the application as the decided value for -that height. - -This mode ensures that value dissemination and consensus are tightly coupled: -consensus only progresses on values that are known to all participants, as -`v` must be received as a condition for a process to vote (on `id(v)`). +In this approach, the consensus protocol itself handles the dissemination +of the actual full value `V` that will eventually be delivered by the application. + +The proposer broadcasts a `PROPOSAL(h, r, v, vr)` message, where the field `v` contains +the full value `V`. Other processes learn the full value by receiving this `PROPOSAL` message. +Since subsequent vote messages (`PREVOTE`, `PRECOMMIT`) include only a compact identifier `id(v)`, +a process must first receive the full value `v` through the `PROPOSAL` before it can cast a vote. + +When the consensus round succeeds, the protocol delivers the value `v` from the `PROPOSAL` message +to the application as the decided value for that height. + +This mode tightly couples value dissemination with consensus: the protocol allows progress only on +values that all participants have received, since each process must obtain the full value `v` before +voting on `id(v)`. + #### Relevance to present ADR