Staking-next integration aka. "Staking Next Command Center" #11401
Labels
P1 - High
Essential for progress, blocks other tasks. Must be completed soon to maintain project flow.
Comments
|
Amazing doc thank you! This is going to have to be a high priority. I agree that we'll have to have some secondary page cc: @ap211unitech Let's have a discussion soon on how we can tackle this and create the secondary page based on the doc Kian put together above. |
TarikGul
added
the
P1 - High
40 tasks
|
Hi @kianenigma , I am having some trouble running node locally according to this doc. Given commit here doesn't compile.
cc: @TarikGul |
|
All good now — we got it working! Huge shoutout to @valentinfernandez1 for the help! 🙌 |
|
Please note that the development branch has been moved to paritytech/polkadot-sdk#8127. Latest commit should all be stable to run. Nothing related to this line of work has chaned 👍 |
|
I won't open a new issue, but a note for @TarikGul: The current staking overview page also doesn't seem to take into account the pagination of exposures. For example, this is what I see now in westend: 
But the correct way to calculate the sum of nominators backing each active validator is: |
Thanks for identifying another issue with the staking page. If i remember correctly there were a lot of entry points where issues arised - albeit I fixed everything on the API level properly, but there might have been a few areas in the staking page I missed. |
17 tasks
github-merge-queue bot
pushed a commit
to paritytech/polkadot-sdk
that referenced
this issue
Apr 21, 2025
Moved from: #7601. Follow ups to: #7282. Closes: #8146 --- This PR is the final outcome of a multi-month development period, with a lot of background work since 2022. Its main aim is to make pallet-staking, alongside its `type ElectionProvider` compatible to be used in a parachain, and report back the validator set to a relay-chain. This setup is intended to be used for Polkadot, Kusama and Westend relay-chains, with the corresponding AssetHubs hosting the staking system. While this PR is quite big, a lot of the diffs are due to adding a relay and parachain runtime for testing. The following is a guide to help reviewers/auditors distinguish what has actually changed in this PR. > Additional reading: See polkadot-js/apps#11401, and the hackmd shared in there, which contains more in-depth explanation of how RC <> AH communicate. ## Added > [This shows the partial diff](ankn/diff-staking-async...ankn/diff-staking-async-1) introduced in pallet-staking-async and election-provider-multi-block relative to the existing (in master) pallet-staking and election-provider-multi-phase. This PR adds the following new pallets, all of which are not used anywhere yet, with the exception of one (see `westend-runtime` changes below). #### `pallet-election-provider-multi-block` This is a set of 4 pallets, capable of implementing an async, multi-page `ElectionProvider`. This pallet is not used in any real runtime yet, and is intended to be used in `AssetHub`, next to `pallet-staking-async`. #### `pallet-staking-async` A fork of the old `pallet-staking`, with a number of key differences, making it suitable to be used in a parachain: 1. It no longer has access to a secure timestamp, previously used to calculate the duration of an era. 2. It no longer has access to a `pallet-session`. 3. It no longer has access to a `pallet-authorship`. 5. It is capable of working with a multi-page `ElectionProvider`, aka. `pallet-election-provider-multi-block`. To compensate for the above, this pallet relies on XCM messages coming from the relay-chain, informing the pallet of: * When a new era should be activated, and how long its duration was * When an offence has happened on the relay relay-chain * When a session ends on the relay-chain, and how many reward points were accumulated for each validators during that period. #### `pallet-staking-async-ah-client` and `pallet-staking-async-rc-client` Are the two new pallets that facilitate the above communication. #### `pallet-ahm-test` A test-only crate that contains e2e rust-based unit test for all of the above. #### `pallet-staking-async-rc-runtime` and `pallet-staking-async-parachain-runtime` Forks of westend and westend-asset-hub, customized to be used for testing all of the above with Zombienet. It contains a lot of unrelated code as well. ## Changed > [This shows the partial diff](ankn/8127-diff-changed-base...ankn/8127-diff-changed-compare) that shows the changes to existing pallets used in prod runtimes as well as westend runtime changes. #### `Identification` This mechanism, which lives on the relay-chain, is expressed by `type FullIdentification` and `type FullIdentificationOf` in runtimes. It is a way to identify the full data needed to slash a validator. Historically, it was pointing to a validator, and their `struct Exposure`. With the move to Asset-Hub, this is no longer possible for two reasons: 1. Relay chain no longer knows the full exposures 2. Even if, the full exposures are getting bigger and bigger and relying the entirety of it is not scalable. Instead, runtimes now move to a new `type FullIdentificationOf = DefaultExposureOf`, which will identify a validator with a `Exposure::default()`. This is suboptimal, as it forces us to still store a number of bytes. Yet, it allows any old `FullIdentification`, pertaining to an old slash, to be decoded. This compromise is only needed to cater for slashes that happen around the time of AHM. #### `westend-runtime` This runtime already has the `pallet-staking-async-ah-client`, integrated into all the places such that: 1. It handles the validator reward points 2. It handles offences 5. It is the `SessionManager` Yet, it is delegating all of the above to its `type Fallback`, which is the old `pallet-staking`. This is a preparatory step for AHM, and should not be any logical change. #### `pallet-election-provider-multi-phase` This is the old single-page `ElectionProvider`. It has been updated to work with multi-page traits, yet it only supports `page-size = 1` for now. It should not have seen any logical changes. #### `pallet-bags-list` Now has two new features. 1. It can be `Locked`, in which case all updates to it fail with an `Err(_)`, even deletion of a node. This is needed because we cannot alter any nodes in this pallet during a multi-page iteration, aka. multi-page snapshot. 2. To combat this, the same `rebag` transaction can be also be used to remove a node from the list, or add a node to the list. This is done through the `score_of` api. See the file changes and tests under `./substrate/frame/bags-list` for more info. #### RuntimeDebug -> Debug To facilitate debugging, a number of types' `RuntimeDebug` impl has been changed to `Debug`. See #3107 ## Weights Below is a summary of the weights. These are generated using `staking-async/runtimes/parachain`, which assumes 22_500 nominators divided by `32` pages for Polkadot, and 12_500 nominators divided by `16` pages in Kusama, both leading to ~700 nominators snapshotted and exported per page. Doubling these parameters would easily slash the PoV weights by half, but with 10MB PoV, these numbers should be good. Also noting that with PoV clawback, we migth get even more proof_size weight back in the runtime. Although, afaik this reclaimed value does not take compression into account. ``` #### new: polkadot/pallet_election_provider_multi_block.rs old: kusama +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +======================================================================================================================+ | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_msp | 2.41MiB | 2.41MiB | -0.03 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_rest | 3.24MiB | 3.06MiB | -5.53 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed | 3.36MiB | 3.12MiB | -7.12 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_non_terminal | 2.12MiB | 1.32MiB | -37.60 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_terminal | 4.08MiB | 2.25MiB | -44.82 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_nothing | 3.53KiB | 3.53KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_unsigned | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed_validation | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | manage | 0B | 0B | Unchanged | +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_signed.rs old: kusama +------------------------------------------------+----------------------+----------+----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +===============================================================================================================+ | pallet_election_provider_multi_block_signed.rs | bail | 43.61KiB | 82.74KiB | +89.72 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_eject | 46.54KiB | 85.80KiB | +84.35 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | clear_old_round_data | 85.23KiB | 85.17KiB | -0.06 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | submit_page | 6.95KiB | 6.90KiB | -0.70 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_not_full | 6.45KiB | 6.39KiB | -1.00 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | unset_page | 20.76KiB | 18.55KiB | -10.67 | +------------------------------------------------+----------------------+----------+----------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_unsigned.rs old: kusama +--------------------------------------------------+-------------------+----------+-----------+------------------+ | File | Extrinsic | Old | New | Change [%] | +================================================================================================================+ | pallet_election_provider_multi_block_unsigned.rs | submit_unsigned | 63.56KiB | 696.00KiB | +995.01 | |--------------------------------------------------+-------------------+----------+-----------+------------------| | pallet_election_provider_multi_block_unsigned.rs | validate_unsigned | 1.81KiB | 3.66KiB | +102.65 | +--------------------------------------------------+-------------------+----------+-----------+------------------+ #### new: polkadot/pallet_election_provider_multi_block_verifier.rs old: kusama +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +=================================================================================================================================+ | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_terminal | 1.18MiB | 1.69MiB | +42.87 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_terminal | 1.18MiB | 1.69MiB | +42.71 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_non_terminal | 1.30MiB | 450.82KiB | -66.08 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_non_terminal | 279.93KiB | 62.22KiB | -77.77 | +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ ``` <summary> note for PR authors <details> <br> ## TODO - [x] Finalize weights - [x] Lock voter list when snapshot being taken - [x] push based election - [x] OffchainWorker miner can now run on multiple pages - [x] Trimming is improved, all bounds are respected. - [x] clients pallets: add ID - [x] make election prolonged - [x] bring westend-next and ah-next to staking-next - [x] Test pre-migration to post-migration state in ahm-test. - [x] Offence reporting works without exposure info on RC (done but recheck). - [ ] staking-async fix tests - [ ] root offence testing (minimally done in migration test) - [ ] Run benchmarking - [x] ~~Add custom decoder for OffenceDetails~~. ## TODO before finalizing PR - [x] Go over again and ensure no interaction with staking-classic except by AhClient (and pallets that are going away) in Westend. Make any non used apis private. - [ ] Create diff with changes from staking-classic. ## Migration Notes - At the start of the AHM migration, trigger: `RC::pallet_staking_async_ah_client::on_migration_start()` - At the start of the AHM migration, trigger the following: - definitely filter `staking::bond` - RC: set `staking::Forcing` to `ForceNone`. - At the end of the AHM migration, trigger the following - `RC::pallet_staking_async_ah_client::on_migration_end()` - Set `AH::pallet_staking_async::ForceEra` to `Forcing::NotForcing`. - Set RC staking and pool min bond to be u32::max. ## Follow-up - [ ] Offence generation e2e test (zombienet) </details> </summary> --------- Co-authored-by: kianenigma <kian@parity.io> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
github-merge-queue bot
pushed a commit
to paritytech/polkadot-sdk
that referenced
this issue
Apr 21, 2025
Moved from: #7601. Follow ups to: #7282. Closes: #8146 --- This PR is the final outcome of a multi-month development period, with a lot of background work since 2022. Its main aim is to make pallet-staking, alongside its `type ElectionProvider` compatible to be used in a parachain, and report back the validator set to a relay-chain. This setup is intended to be used for Polkadot, Kusama and Westend relay-chains, with the corresponding AssetHubs hosting the staking system. While this PR is quite big, a lot of the diffs are due to adding a relay and parachain runtime for testing. The following is a guide to help reviewers/auditors distinguish what has actually changed in this PR. > Additional reading: See polkadot-js/apps#11401, and the hackmd shared in there, which contains more in-depth explanation of how RC <> AH communicate. ## Added > [This shows the partial diff](ankn/diff-staking-async...ankn/diff-staking-async-1) introduced in pallet-staking-async and election-provider-multi-block relative to the existing (in master) pallet-staking and election-provider-multi-phase. This PR adds the following new pallets, all of which are not used anywhere yet, with the exception of one (see `westend-runtime` changes below). #### `pallet-election-provider-multi-block` This is a set of 4 pallets, capable of implementing an async, multi-page `ElectionProvider`. This pallet is not used in any real runtime yet, and is intended to be used in `AssetHub`, next to `pallet-staking-async`. #### `pallet-staking-async` A fork of the old `pallet-staking`, with a number of key differences, making it suitable to be used in a parachain: 1. It no longer has access to a secure timestamp, previously used to calculate the duration of an era. 2. It no longer has access to a `pallet-session`. 3. It no longer has access to a `pallet-authorship`. 5. It is capable of working with a multi-page `ElectionProvider`, aka. `pallet-election-provider-multi-block`. To compensate for the above, this pallet relies on XCM messages coming from the relay-chain, informing the pallet of: * When a new era should be activated, and how long its duration was * When an offence has happened on the relay relay-chain * When a session ends on the relay-chain, and how many reward points were accumulated for each validators during that period. #### `pallet-staking-async-ah-client` and `pallet-staking-async-rc-client` Are the two new pallets that facilitate the above communication. #### `pallet-ahm-test` A test-only crate that contains e2e rust-based unit test for all of the above. #### `pallet-staking-async-rc-runtime` and `pallet-staking-async-parachain-runtime` Forks of westend and westend-asset-hub, customized to be used for testing all of the above with Zombienet. It contains a lot of unrelated code as well. ## Changed > [This shows the partial diff](ankn/8127-diff-changed-base...ankn/8127-diff-changed-compare) that shows the changes to existing pallets used in prod runtimes as well as westend runtime changes. #### `Identification` This mechanism, which lives on the relay-chain, is expressed by `type FullIdentification` and `type FullIdentificationOf` in runtimes. It is a way to identify the full data needed to slash a validator. Historically, it was pointing to a validator, and their `struct Exposure`. With the move to Asset-Hub, this is no longer possible for two reasons: 1. Relay chain no longer knows the full exposures 2. Even if, the full exposures are getting bigger and bigger and relying the entirety of it is not scalable. Instead, runtimes now move to a new `type FullIdentificationOf = DefaultExposureOf`, which will identify a validator with a `Exposure::default()`. This is suboptimal, as it forces us to still store a number of bytes. Yet, it allows any old `FullIdentification`, pertaining to an old slash, to be decoded. This compromise is only needed to cater for slashes that happen around the time of AHM. #### `westend-runtime` This runtime already has the `pallet-staking-async-ah-client`, integrated into all the places such that: 1. It handles the validator reward points 2. It handles offences 5. It is the `SessionManager` Yet, it is delegating all of the above to its `type Fallback`, which is the old `pallet-staking`. This is a preparatory step for AHM, and should not be any logical change. #### `pallet-election-provider-multi-phase` This is the old single-page `ElectionProvider`. It has been updated to work with multi-page traits, yet it only supports `page-size = 1` for now. It should not have seen any logical changes. #### `pallet-bags-list` Now has two new features. 1. It can be `Locked`, in which case all updates to it fail with an `Err(_)`, even deletion of a node. This is needed because we cannot alter any nodes in this pallet during a multi-page iteration, aka. multi-page snapshot. 2. To combat this, the same `rebag` transaction can be also be used to remove a node from the list, or add a node to the list. This is done through the `score_of` api. See the file changes and tests under `./substrate/frame/bags-list` for more info. #### RuntimeDebug -> Debug To facilitate debugging, a number of types' `RuntimeDebug` impl has been changed to `Debug`. See #3107 ## Weights Below is a summary of the weights. These are generated using `staking-async/runtimes/parachain`, which assumes 22_500 nominators divided by `32` pages for Polkadot, and 12_500 nominators divided by `16` pages in Kusama, both leading to ~700 nominators snapshotted and exported per page. Doubling these parameters would easily slash the PoV weights by half, but with 10MB PoV, these numbers should be good. Also noting that with PoV clawback, we migth get even more proof_size weight back in the runtime. Although, afaik this reclaimed value does not take compression into account. ``` #### new: polkadot/pallet_election_provider_multi_block.rs old: kusama +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +======================================================================================================================+ | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_msp | 2.41MiB | 2.41MiB | -0.03 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_rest | 3.24MiB | 3.06MiB | -5.53 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed | 3.36MiB | 3.12MiB | -7.12 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_non_terminal | 2.12MiB | 1.32MiB | -37.60 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_terminal | 4.08MiB | 2.25MiB | -44.82 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_nothing | 3.53KiB | 3.53KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_unsigned | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed_validation | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | manage | 0B | 0B | Unchanged | +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_signed.rs old: kusama +------------------------------------------------+----------------------+----------+----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +===============================================================================================================+ | pallet_election_provider_multi_block_signed.rs | bail | 43.61KiB | 82.74KiB | +89.72 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_eject | 46.54KiB | 85.80KiB | +84.35 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | clear_old_round_data | 85.23KiB | 85.17KiB | -0.06 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | submit_page | 6.95KiB | 6.90KiB | -0.70 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_not_full | 6.45KiB | 6.39KiB | -1.00 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | unset_page | 20.76KiB | 18.55KiB | -10.67 | +------------------------------------------------+----------------------+----------+----------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_unsigned.rs old: kusama +--------------------------------------------------+-------------------+----------+-----------+------------------+ | File | Extrinsic | Old | New | Change [%] | +================================================================================================================+ | pallet_election_provider_multi_block_unsigned.rs | submit_unsigned | 63.56KiB | 696.00KiB | +995.01 | |--------------------------------------------------+-------------------+----------+-----------+------------------| | pallet_election_provider_multi_block_unsigned.rs | validate_unsigned | 1.81KiB | 3.66KiB | +102.65 | +--------------------------------------------------+-------------------+----------+-----------+------------------+ #### new: polkadot/pallet_election_provider_multi_block_verifier.rs old: kusama +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +=================================================================================================================================+ | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_terminal | 1.18MiB | 1.69MiB | +42.87 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_terminal | 1.18MiB | 1.69MiB | +42.71 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_non_terminal | 1.30MiB | 450.82KiB | -66.08 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_non_terminal | 279.93KiB | 62.22KiB | -77.77 | +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ ``` <summary> note for PR authors <details> <br> ## TODO - [x] Finalize weights - [x] Lock voter list when snapshot being taken - [x] push based election - [x] OffchainWorker miner can now run on multiple pages - [x] Trimming is improved, all bounds are respected. - [x] clients pallets: add ID - [x] make election prolonged - [x] bring westend-next and ah-next to staking-next - [x] Test pre-migration to post-migration state in ahm-test. - [x] Offence reporting works without exposure info on RC (done but recheck). - [ ] staking-async fix tests - [ ] root offence testing (minimally done in migration test) - [ ] Run benchmarking - [x] ~~Add custom decoder for OffenceDetails~~. ## TODO before finalizing PR - [x] Go over again and ensure no interaction with staking-classic except by AhClient (and pallets that are going away) in Westend. Make any non used apis private. - [ ] Create diff with changes from staking-classic. ## Migration Notes - At the start of the AHM migration, trigger: `RC::pallet_staking_async_ah_client::on_migration_start()` - At the start of the AHM migration, trigger the following: - definitely filter `staking::bond` - RC: set `staking::Forcing` to `ForceNone`. - At the end of the AHM migration, trigger the following - `RC::pallet_staking_async_ah_client::on_migration_end()` - Set `AH::pallet_staking_async::ForceEra` to `Forcing::NotForcing`. - Set RC staking and pool min bond to be u32::max. ## Follow-up - [ ] Offence generation e2e test (zombienet) </details> </summary> --------- Co-authored-by: kianenigma <kian@parity.io> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
github-merge-queue bot
pushed a commit
to paritytech/polkadot-sdk
that referenced
this issue
Apr 21, 2025
Moved from: #7601. Follow ups to: #7282. Closes: #8146 --- This PR is the final outcome of a multi-month development period, with a lot of background work since 2022. Its main aim is to make pallet-staking, alongside its `type ElectionProvider` compatible to be used in a parachain, and report back the validator set to a relay-chain. This setup is intended to be used for Polkadot, Kusama and Westend relay-chains, with the corresponding AssetHubs hosting the staking system. While this PR is quite big, a lot of the diffs are due to adding a relay and parachain runtime for testing. The following is a guide to help reviewers/auditors distinguish what has actually changed in this PR. > Additional reading: See polkadot-js/apps#11401, and the hackmd shared in there, which contains more in-depth explanation of how RC <> AH communicate. ## Added > [This shows the partial diff](ankn/diff-staking-async...ankn/diff-staking-async-1) introduced in pallet-staking-async and election-provider-multi-block relative to the existing (in master) pallet-staking and election-provider-multi-phase. This PR adds the following new pallets, all of which are not used anywhere yet, with the exception of one (see `westend-runtime` changes below). #### `pallet-election-provider-multi-block` This is a set of 4 pallets, capable of implementing an async, multi-page `ElectionProvider`. This pallet is not used in any real runtime yet, and is intended to be used in `AssetHub`, next to `pallet-staking-async`. #### `pallet-staking-async` A fork of the old `pallet-staking`, with a number of key differences, making it suitable to be used in a parachain: 1. It no longer has access to a secure timestamp, previously used to calculate the duration of an era. 2. It no longer has access to a `pallet-session`. 3. It no longer has access to a `pallet-authorship`. 5. It is capable of working with a multi-page `ElectionProvider`, aka. `pallet-election-provider-multi-block`. To compensate for the above, this pallet relies on XCM messages coming from the relay-chain, informing the pallet of: * When a new era should be activated, and how long its duration was * When an offence has happened on the relay relay-chain * When a session ends on the relay-chain, and how many reward points were accumulated for each validators during that period. #### `pallet-staking-async-ah-client` and `pallet-staking-async-rc-client` Are the two new pallets that facilitate the above communication. #### `pallet-ahm-test` A test-only crate that contains e2e rust-based unit test for all of the above. #### `pallet-staking-async-rc-runtime` and `pallet-staking-async-parachain-runtime` Forks of westend and westend-asset-hub, customized to be used for testing all of the above with Zombienet. It contains a lot of unrelated code as well. ## Changed > [This shows the partial diff](ankn/8127-diff-changed-base...ankn/8127-diff-changed-compare) that shows the changes to existing pallets used in prod runtimes as well as westend runtime changes. #### `Identification` This mechanism, which lives on the relay-chain, is expressed by `type FullIdentification` and `type FullIdentificationOf` in runtimes. It is a way to identify the full data needed to slash a validator. Historically, it was pointing to a validator, and their `struct Exposure`. With the move to Asset-Hub, this is no longer possible for two reasons: 1. Relay chain no longer knows the full exposures 2. Even if, the full exposures are getting bigger and bigger and relying the entirety of it is not scalable. Instead, runtimes now move to a new `type FullIdentificationOf = DefaultExposureOf`, which will identify a validator with a `Exposure::default()`. This is suboptimal, as it forces us to still store a number of bytes. Yet, it allows any old `FullIdentification`, pertaining to an old slash, to be decoded. This compromise is only needed to cater for slashes that happen around the time of AHM. #### `westend-runtime` This runtime already has the `pallet-staking-async-ah-client`, integrated into all the places such that: 1. It handles the validator reward points 2. It handles offences 5. It is the `SessionManager` Yet, it is delegating all of the above to its `type Fallback`, which is the old `pallet-staking`. This is a preparatory step for AHM, and should not be any logical change. #### `pallet-election-provider-multi-phase` This is the old single-page `ElectionProvider`. It has been updated to work with multi-page traits, yet it only supports `page-size = 1` for now. It should not have seen any logical changes. #### `pallet-bags-list` Now has two new features. 1. It can be `Locked`, in which case all updates to it fail with an `Err(_)`, even deletion of a node. This is needed because we cannot alter any nodes in this pallet during a multi-page iteration, aka. multi-page snapshot. 2. To combat this, the same `rebag` transaction can be also be used to remove a node from the list, or add a node to the list. This is done through the `score_of` api. See the file changes and tests under `./substrate/frame/bags-list` for more info. #### RuntimeDebug -> Debug To facilitate debugging, a number of types' `RuntimeDebug` impl has been changed to `Debug`. See #3107 ## Weights Below is a summary of the weights. These are generated using `staking-async/runtimes/parachain`, which assumes 22_500 nominators divided by `32` pages for Polkadot, and 12_500 nominators divided by `16` pages in Kusama, both leading to ~700 nominators snapshotted and exported per page. Doubling these parameters would easily slash the PoV weights by half, but with 10MB PoV, these numbers should be good. Also noting that with PoV clawback, we migth get even more proof_size weight back in the runtime. Although, afaik this reclaimed value does not take compression into account. ``` #### new: polkadot/pallet_election_provider_multi_block.rs old: kusama +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +======================================================================================================================+ | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_msp | 2.41MiB | 2.41MiB | -0.03 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_rest | 3.24MiB | 3.06MiB | -5.53 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed | 3.36MiB | 3.12MiB | -7.12 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_non_terminal | 2.12MiB | 1.32MiB | -37.60 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_terminal | 4.08MiB | 2.25MiB | -44.82 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_nothing | 3.53KiB | 3.53KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_unsigned | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed_validation | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | manage | 0B | 0B | Unchanged | +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_signed.rs old: kusama +------------------------------------------------+----------------------+----------+----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +===============================================================================================================+ | pallet_election_provider_multi_block_signed.rs | bail | 43.61KiB | 82.74KiB | +89.72 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_eject | 46.54KiB | 85.80KiB | +84.35 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | clear_old_round_data | 85.23KiB | 85.17KiB | -0.06 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | submit_page | 6.95KiB | 6.90KiB | -0.70 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_not_full | 6.45KiB | 6.39KiB | -1.00 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | unset_page | 20.76KiB | 18.55KiB | -10.67 | +------------------------------------------------+----------------------+----------+----------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_unsigned.rs old: kusama +--------------------------------------------------+-------------------+----------+-----------+------------------+ | File | Extrinsic | Old | New | Change [%] | +================================================================================================================+ | pallet_election_provider_multi_block_unsigned.rs | submit_unsigned | 63.56KiB | 696.00KiB | +995.01 | |--------------------------------------------------+-------------------+----------+-----------+------------------| | pallet_election_provider_multi_block_unsigned.rs | validate_unsigned | 1.81KiB | 3.66KiB | +102.65 | +--------------------------------------------------+-------------------+----------+-----------+------------------+ #### new: polkadot/pallet_election_provider_multi_block_verifier.rs old: kusama +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +=================================================================================================================================+ | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_terminal | 1.18MiB | 1.69MiB | +42.87 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_terminal | 1.18MiB | 1.69MiB | +42.71 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_non_terminal | 1.30MiB | 450.82KiB | -66.08 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_non_terminal | 279.93KiB | 62.22KiB | -77.77 | +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ ``` <summary> note for PR authors <details> <br> ## TODO - [x] Finalize weights - [x] Lock voter list when snapshot being taken - [x] push based election - [x] OffchainWorker miner can now run on multiple pages - [x] Trimming is improved, all bounds are respected. - [x] clients pallets: add ID - [x] make election prolonged - [x] bring westend-next and ah-next to staking-next - [x] Test pre-migration to post-migration state in ahm-test. - [x] Offence reporting works without exposure info on RC (done but recheck). - [ ] staking-async fix tests - [ ] root offence testing (minimally done in migration test) - [ ] Run benchmarking - [x] ~~Add custom decoder for OffenceDetails~~. ## TODO before finalizing PR - [x] Go over again and ensure no interaction with staking-classic except by AhClient (and pallets that are going away) in Westend. Make any non used apis private. - [ ] Create diff with changes from staking-classic. ## Migration Notes - At the start of the AHM migration, trigger: `RC::pallet_staking_async_ah_client::on_migration_start()` - At the start of the AHM migration, trigger the following: - definitely filter `staking::bond` - RC: set `staking::Forcing` to `ForceNone`. - At the end of the AHM migration, trigger the following - `RC::pallet_staking_async_ah_client::on_migration_end()` - Set `AH::pallet_staking_async::ForceEra` to `Forcing::NotForcing`. - Set RC staking and pool min bond to be u32::max. ## Follow-up - [ ] Offence generation e2e test (zombienet) </details> </summary> --------- Co-authored-by: kianenigma <kian@parity.io> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
github-merge-queue bot
pushed a commit
to paritytech/polkadot-sdk
that referenced
this issue
Apr 21, 2025
Moved from: #7601. Follow ups to: #7282. Closes: #8146 --- This PR is the final outcome of a multi-month development period, with a lot of background work since 2022. Its main aim is to make pallet-staking, alongside its `type ElectionProvider` compatible to be used in a parachain, and report back the validator set to a relay-chain. This setup is intended to be used for Polkadot, Kusama and Westend relay-chains, with the corresponding AssetHubs hosting the staking system. While this PR is quite big, a lot of the diffs are due to adding a relay and parachain runtime for testing. The following is a guide to help reviewers/auditors distinguish what has actually changed in this PR. > Additional reading: See polkadot-js/apps#11401, and the hackmd shared in there, which contains more in-depth explanation of how RC <> AH communicate. ## Added > [This shows the partial diff](ankn/diff-staking-async...ankn/diff-staking-async-1) introduced in pallet-staking-async and election-provider-multi-block relative to the existing (in master) pallet-staking and election-provider-multi-phase. This PR adds the following new pallets, all of which are not used anywhere yet, with the exception of one (see `westend-runtime` changes below). #### `pallet-election-provider-multi-block` This is a set of 4 pallets, capable of implementing an async, multi-page `ElectionProvider`. This pallet is not used in any real runtime yet, and is intended to be used in `AssetHub`, next to `pallet-staking-async`. #### `pallet-staking-async` A fork of the old `pallet-staking`, with a number of key differences, making it suitable to be used in a parachain: 1. It no longer has access to a secure timestamp, previously used to calculate the duration of an era. 2. It no longer has access to a `pallet-session`. 3. It no longer has access to a `pallet-authorship`. 5. It is capable of working with a multi-page `ElectionProvider`, aka. `pallet-election-provider-multi-block`. To compensate for the above, this pallet relies on XCM messages coming from the relay-chain, informing the pallet of: * When a new era should be activated, and how long its duration was * When an offence has happened on the relay relay-chain * When a session ends on the relay-chain, and how many reward points were accumulated for each validators during that period. #### `pallet-staking-async-ah-client` and `pallet-staking-async-rc-client` Are the two new pallets that facilitate the above communication. #### `pallet-ahm-test` A test-only crate that contains e2e rust-based unit test for all of the above. #### `pallet-staking-async-rc-runtime` and `pallet-staking-async-parachain-runtime` Forks of westend and westend-asset-hub, customized to be used for testing all of the above with Zombienet. It contains a lot of unrelated code as well. ## Changed > [This shows the partial diff](ankn/8127-diff-changed-base...ankn/8127-diff-changed-compare) that shows the changes to existing pallets used in prod runtimes as well as westend runtime changes. #### `Identification` This mechanism, which lives on the relay-chain, is expressed by `type FullIdentification` and `type FullIdentificationOf` in runtimes. It is a way to identify the full data needed to slash a validator. Historically, it was pointing to a validator, and their `struct Exposure`. With the move to Asset-Hub, this is no longer possible for two reasons: 1. Relay chain no longer knows the full exposures 2. Even if, the full exposures are getting bigger and bigger and relying the entirety of it is not scalable. Instead, runtimes now move to a new `type FullIdentificationOf = DefaultExposureOf`, which will identify a validator with a `Exposure::default()`. This is suboptimal, as it forces us to still store a number of bytes. Yet, it allows any old `FullIdentification`, pertaining to an old slash, to be decoded. This compromise is only needed to cater for slashes that happen around the time of AHM. #### `westend-runtime` This runtime already has the `pallet-staking-async-ah-client`, integrated into all the places such that: 1. It handles the validator reward points 2. It handles offences 5. It is the `SessionManager` Yet, it is delegating all of the above to its `type Fallback`, which is the old `pallet-staking`. This is a preparatory step for AHM, and should not be any logical change. #### `pallet-election-provider-multi-phase` This is the old single-page `ElectionProvider`. It has been updated to work with multi-page traits, yet it only supports `page-size = 1` for now. It should not have seen any logical changes. #### `pallet-bags-list` Now has two new features. 1. It can be `Locked`, in which case all updates to it fail with an `Err(_)`, even deletion of a node. This is needed because we cannot alter any nodes in this pallet during a multi-page iteration, aka. multi-page snapshot. 2. To combat this, the same `rebag` transaction can be also be used to remove a node from the list, or add a node to the list. This is done through the `score_of` api. See the file changes and tests under `./substrate/frame/bags-list` for more info. #### RuntimeDebug -> Debug To facilitate debugging, a number of types' `RuntimeDebug` impl has been changed to `Debug`. See #3107 ## Weights Below is a summary of the weights. These are generated using `staking-async/runtimes/parachain`, which assumes 22_500 nominators divided by `32` pages for Polkadot, and 12_500 nominators divided by `16` pages in Kusama, both leading to ~700 nominators snapshotted and exported per page. Doubling these parameters would easily slash the PoV weights by half, but with 10MB PoV, these numbers should be good. Also noting that with PoV clawback, we migth get even more proof_size weight back in the runtime. Although, afaik this reclaimed value does not take compression into account. ``` #### new: polkadot/pallet_election_provider_multi_block.rs old: kusama +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +======================================================================================================================+ | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_msp | 2.41MiB | 2.41MiB | -0.03 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_rest | 3.24MiB | 3.06MiB | -5.53 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed | 3.36MiB | 3.12MiB | -7.12 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_non_terminal | 2.12MiB | 1.32MiB | -37.60 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_terminal | 4.08MiB | 2.25MiB | -44.82 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_nothing | 3.53KiB | 3.53KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_unsigned | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed_validation | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | manage | 0B | 0B | Unchanged | +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_signed.rs old: kusama +------------------------------------------------+----------------------+----------+----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +===============================================================================================================+ | pallet_election_provider_multi_block_signed.rs | bail | 43.61KiB | 82.74KiB | +89.72 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_eject | 46.54KiB | 85.80KiB | +84.35 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | clear_old_round_data | 85.23KiB | 85.17KiB | -0.06 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | submit_page | 6.95KiB | 6.90KiB | -0.70 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_not_full | 6.45KiB | 6.39KiB | -1.00 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | unset_page | 20.76KiB | 18.55KiB | -10.67 | +------------------------------------------------+----------------------+----------+----------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_unsigned.rs old: kusama +--------------------------------------------------+-------------------+----------+-----------+------------------+ | File | Extrinsic | Old | New | Change [%] | +================================================================================================================+ | pallet_election_provider_multi_block_unsigned.rs | submit_unsigned | 63.56KiB | 696.00KiB | +995.01 | |--------------------------------------------------+-------------------+----------+-----------+------------------| | pallet_election_provider_multi_block_unsigned.rs | validate_unsigned | 1.81KiB | 3.66KiB | +102.65 | +--------------------------------------------------+-------------------+----------+-----------+------------------+ #### new: polkadot/pallet_election_provider_multi_block_verifier.rs old: kusama +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +=================================================================================================================================+ | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_terminal | 1.18MiB | 1.69MiB | +42.87 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_terminal | 1.18MiB | 1.69MiB | +42.71 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_non_terminal | 1.30MiB | 450.82KiB | -66.08 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_non_terminal | 279.93KiB | 62.22KiB | -77.77 | +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ ``` <summary> note for PR authors <details> <br> ## TODO - [x] Finalize weights - [x] Lock voter list when snapshot being taken - [x] push based election - [x] OffchainWorker miner can now run on multiple pages - [x] Trimming is improved, all bounds are respected. - [x] clients pallets: add ID - [x] make election prolonged - [x] bring westend-next and ah-next to staking-next - [x] Test pre-migration to post-migration state in ahm-test. - [x] Offence reporting works without exposure info on RC (done but recheck). - [ ] staking-async fix tests - [ ] root offence testing (minimally done in migration test) - [ ] Run benchmarking - [x] ~~Add custom decoder for OffenceDetails~~. ## TODO before finalizing PR - [x] Go over again and ensure no interaction with staking-classic except by AhClient (and pallets that are going away) in Westend. Make any non used apis private. - [ ] Create diff with changes from staking-classic. ## Migration Notes - At the start of the AHM migration, trigger: `RC::pallet_staking_async_ah_client::on_migration_start()` - At the start of the AHM migration, trigger the following: - definitely filter `staking::bond` - RC: set `staking::Forcing` to `ForceNone`. - At the end of the AHM migration, trigger the following - `RC::pallet_staking_async_ah_client::on_migration_end()` - Set `AH::pallet_staking_async::ForceEra` to `Forcing::NotForcing`. - Set RC staking and pool min bond to be u32::max. ## Follow-up - [ ] Offence generation e2e test (zombienet) </details> </summary> --------- Co-authored-by: kianenigma <kian@parity.io> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
github-merge-queue bot
pushed a commit
to paritytech/polkadot-sdk
that referenced
this issue
Apr 22, 2025
Moved from: #7601. Follow ups to: #7282. Closes: #8146 --- This PR is the final outcome of a multi-month development period, with a lot of background work since 2022. Its main aim is to make pallet-staking, alongside its `type ElectionProvider` compatible to be used in a parachain, and report back the validator set to a relay-chain. This setup is intended to be used for Polkadot, Kusama and Westend relay-chains, with the corresponding AssetHubs hosting the staking system. While this PR is quite big, a lot of the diffs are due to adding a relay and parachain runtime for testing. The following is a guide to help reviewers/auditors distinguish what has actually changed in this PR. > Additional reading: See polkadot-js/apps#11401, and the hackmd shared in there, which contains more in-depth explanation of how RC <> AH communicate. ## Added > [This shows the partial diff](ankn/diff-staking-async...ankn/diff-staking-async-1) introduced in pallet-staking-async and election-provider-multi-block relative to the existing (in master) pallet-staking and election-provider-multi-phase. This PR adds the following new pallets, all of which are not used anywhere yet, with the exception of one (see `westend-runtime` changes below). #### `pallet-election-provider-multi-block` This is a set of 4 pallets, capable of implementing an async, multi-page `ElectionProvider`. This pallet is not used in any real runtime yet, and is intended to be used in `AssetHub`, next to `pallet-staking-async`. #### `pallet-staking-async` A fork of the old `pallet-staking`, with a number of key differences, making it suitable to be used in a parachain: 1. It no longer has access to a secure timestamp, previously used to calculate the duration of an era. 2. It no longer has access to a `pallet-session`. 3. It no longer has access to a `pallet-authorship`. 5. It is capable of working with a multi-page `ElectionProvider`, aka. `pallet-election-provider-multi-block`. To compensate for the above, this pallet relies on XCM messages coming from the relay-chain, informing the pallet of: * When a new era should be activated, and how long its duration was * When an offence has happened on the relay relay-chain * When a session ends on the relay-chain, and how many reward points were accumulated for each validators during that period. #### `pallet-staking-async-ah-client` and `pallet-staking-async-rc-client` Are the two new pallets that facilitate the above communication. #### `pallet-ahm-test` A test-only crate that contains e2e rust-based unit test for all of the above. #### `pallet-staking-async-rc-runtime` and `pallet-staking-async-parachain-runtime` Forks of westend and westend-asset-hub, customized to be used for testing all of the above with Zombienet. It contains a lot of unrelated code as well. ## Changed > [This shows the partial diff](ankn/8127-diff-changed-base...ankn/8127-diff-changed-compare) that shows the changes to existing pallets used in prod runtimes as well as westend runtime changes. #### `Identification` This mechanism, which lives on the relay-chain, is expressed by `type FullIdentification` and `type FullIdentificationOf` in runtimes. It is a way to identify the full data needed to slash a validator. Historically, it was pointing to a validator, and their `struct Exposure`. With the move to Asset-Hub, this is no longer possible for two reasons: 1. Relay chain no longer knows the full exposures 2. Even if, the full exposures are getting bigger and bigger and relying the entirety of it is not scalable. Instead, runtimes now move to a new `type FullIdentificationOf = DefaultExposureOf`, which will identify a validator with a `Exposure::default()`. This is suboptimal, as it forces us to still store a number of bytes. Yet, it allows any old `FullIdentification`, pertaining to an old slash, to be decoded. This compromise is only needed to cater for slashes that happen around the time of AHM. #### `westend-runtime` This runtime already has the `pallet-staking-async-ah-client`, integrated into all the places such that: 1. It handles the validator reward points 2. It handles offences 5. It is the `SessionManager` Yet, it is delegating all of the above to its `type Fallback`, which is the old `pallet-staking`. This is a preparatory step for AHM, and should not be any logical change. #### `pallet-election-provider-multi-phase` This is the old single-page `ElectionProvider`. It has been updated to work with multi-page traits, yet it only supports `page-size = 1` for now. It should not have seen any logical changes. #### `pallet-bags-list` Now has two new features. 1. It can be `Locked`, in which case all updates to it fail with an `Err(_)`, even deletion of a node. This is needed because we cannot alter any nodes in this pallet during a multi-page iteration, aka. multi-page snapshot. 2. To combat this, the same `rebag` transaction can be also be used to remove a node from the list, or add a node to the list. This is done through the `score_of` api. See the file changes and tests under `./substrate/frame/bags-list` for more info. #### RuntimeDebug -> Debug To facilitate debugging, a number of types' `RuntimeDebug` impl has been changed to `Debug`. See #3107 ## Weights Below is a summary of the weights. These are generated using `staking-async/runtimes/parachain`, which assumes 22_500 nominators divided by `32` pages for Polkadot, and 12_500 nominators divided by `16` pages in Kusama, both leading to ~700 nominators snapshotted and exported per page. Doubling these parameters would easily slash the PoV weights by half, but with 10MB PoV, these numbers should be good. Also noting that with PoV clawback, we migth get even more proof_size weight back in the runtime. Although, afaik this reclaimed value does not take compression into account. ``` #### new: polkadot/pallet_election_provider_multi_block.rs old: kusama +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +======================================================================================================================+ | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_msp | 2.41MiB | 2.41MiB | -0.03 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_rest | 3.24MiB | 3.06MiB | -5.53 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed | 3.36MiB | 3.12MiB | -7.12 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_non_terminal | 2.12MiB | 1.32MiB | -37.60 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_terminal | 4.08MiB | 2.25MiB | -44.82 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_nothing | 3.53KiB | 3.53KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_unsigned | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed_validation | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | manage | 0B | 0B | Unchanged | +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_signed.rs old: kusama +------------------------------------------------+----------------------+----------+----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +===============================================================================================================+ | pallet_election_provider_multi_block_signed.rs | bail | 43.61KiB | 82.74KiB | +89.72 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_eject | 46.54KiB | 85.80KiB | +84.35 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | clear_old_round_data | 85.23KiB | 85.17KiB | -0.06 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | submit_page | 6.95KiB | 6.90KiB | -0.70 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_not_full | 6.45KiB | 6.39KiB | -1.00 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | unset_page | 20.76KiB | 18.55KiB | -10.67 | +------------------------------------------------+----------------------+----------+----------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_unsigned.rs old: kusama +--------------------------------------------------+-------------------+----------+-----------+------------------+ | File | Extrinsic | Old | New | Change [%] | +================================================================================================================+ | pallet_election_provider_multi_block_unsigned.rs | submit_unsigned | 63.56KiB | 696.00KiB | +995.01 | |--------------------------------------------------+-------------------+----------+-----------+------------------| | pallet_election_provider_multi_block_unsigned.rs | validate_unsigned | 1.81KiB | 3.66KiB | +102.65 | +--------------------------------------------------+-------------------+----------+-----------+------------------+ #### new: polkadot/pallet_election_provider_multi_block_verifier.rs old: kusama +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +=================================================================================================================================+ | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_terminal | 1.18MiB | 1.69MiB | +42.87 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_terminal | 1.18MiB | 1.69MiB | +42.71 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_non_terminal | 1.30MiB | 450.82KiB | -66.08 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_non_terminal | 279.93KiB | 62.22KiB | -77.77 | +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ ``` <summary> note for PR authors <details> <br> ## TODO - [x] Finalize weights - [x] Lock voter list when snapshot being taken - [x] push based election - [x] OffchainWorker miner can now run on multiple pages - [x] Trimming is improved, all bounds are respected. - [x] clients pallets: add ID - [x] make election prolonged - [x] bring westend-next and ah-next to staking-next - [x] Test pre-migration to post-migration state in ahm-test. - [x] Offence reporting works without exposure info on RC (done but recheck). - [ ] staking-async fix tests - [ ] root offence testing (minimally done in migration test) - [ ] Run benchmarking - [x] ~~Add custom decoder for OffenceDetails~~. ## TODO before finalizing PR - [x] Go over again and ensure no interaction with staking-classic except by AhClient (and pallets that are going away) in Westend. Make any non used apis private. - [ ] Create diff with changes from staking-classic. ## Migration Notes - At the start of the AHM migration, trigger: `RC::pallet_staking_async_ah_client::on_migration_start()` - At the start of the AHM migration, trigger the following: - definitely filter `staking::bond` - RC: set `staking::Forcing` to `ForceNone`. - At the end of the AHM migration, trigger the following - `RC::pallet_staking_async_ah_client::on_migration_end()` - Set `AH::pallet_staking_async::ForceEra` to `Forcing::NotForcing`. - Set RC staking and pool min bond to be u32::max. ## Follow-up - [ ] Offence generation e2e test (zombienet) </details> </summary> --------- Co-authored-by: kianenigma <kian@parity.io> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
castillax
pushed a commit
to paritytech/polkadot-sdk
that referenced
this issue
May 12, 2025
Moved from: #7601. Follow ups to: #7282. Closes: #8146 --- This PR is the final outcome of a multi-month development period, with a lot of background work since 2022. Its main aim is to make pallet-staking, alongside its `type ElectionProvider` compatible to be used in a parachain, and report back the validator set to a relay-chain. This setup is intended to be used for Polkadot, Kusama and Westend relay-chains, with the corresponding AssetHubs hosting the staking system. While this PR is quite big, a lot of the diffs are due to adding a relay and parachain runtime for testing. The following is a guide to help reviewers/auditors distinguish what has actually changed in this PR. > Additional reading: See polkadot-js/apps#11401, and the hackmd shared in there, which contains more in-depth explanation of how RC <> AH communicate. ## Added > [This shows the partial diff](ankn/diff-staking-async...ankn/diff-staking-async-1) introduced in pallet-staking-async and election-provider-multi-block relative to the existing (in master) pallet-staking and election-provider-multi-phase. This PR adds the following new pallets, all of which are not used anywhere yet, with the exception of one (see `westend-runtime` changes below). #### `pallet-election-provider-multi-block` This is a set of 4 pallets, capable of implementing an async, multi-page `ElectionProvider`. This pallet is not used in any real runtime yet, and is intended to be used in `AssetHub`, next to `pallet-staking-async`. #### `pallet-staking-async` A fork of the old `pallet-staking`, with a number of key differences, making it suitable to be used in a parachain: 1. It no longer has access to a secure timestamp, previously used to calculate the duration of an era. 2. It no longer has access to a `pallet-session`. 3. It no longer has access to a `pallet-authorship`. 5. It is capable of working with a multi-page `ElectionProvider`, aka. `pallet-election-provider-multi-block`. To compensate for the above, this pallet relies on XCM messages coming from the relay-chain, informing the pallet of: * When a new era should be activated, and how long its duration was * When an offence has happened on the relay relay-chain * When a session ends on the relay-chain, and how many reward points were accumulated for each validators during that period. #### `pallet-staking-async-ah-client` and `pallet-staking-async-rc-client` Are the two new pallets that facilitate the above communication. #### `pallet-ahm-test` A test-only crate that contains e2e rust-based unit test for all of the above. #### `pallet-staking-async-rc-runtime` and `pallet-staking-async-parachain-runtime` Forks of westend and westend-asset-hub, customized to be used for testing all of the above with Zombienet. It contains a lot of unrelated code as well. ## Changed > [This shows the partial diff](ankn/8127-diff-changed-base...ankn/8127-diff-changed-compare) that shows the changes to existing pallets used in prod runtimes as well as westend runtime changes. #### `Identification` This mechanism, which lives on the relay-chain, is expressed by `type FullIdentification` and `type FullIdentificationOf` in runtimes. It is a way to identify the full data needed to slash a validator. Historically, it was pointing to a validator, and their `struct Exposure`. With the move to Asset-Hub, this is no longer possible for two reasons: 1. Relay chain no longer knows the full exposures 2. Even if, the full exposures are getting bigger and bigger and relying the entirety of it is not scalable. Instead, runtimes now move to a new `type FullIdentificationOf = DefaultExposureOf`, which will identify a validator with a `Exposure::default()`. This is suboptimal, as it forces us to still store a number of bytes. Yet, it allows any old `FullIdentification`, pertaining to an old slash, to be decoded. This compromise is only needed to cater for slashes that happen around the time of AHM. #### `westend-runtime` This runtime already has the `pallet-staking-async-ah-client`, integrated into all the places such that: 1. It handles the validator reward points 2. It handles offences 5. It is the `SessionManager` Yet, it is delegating all of the above to its `type Fallback`, which is the old `pallet-staking`. This is a preparatory step for AHM, and should not be any logical change. #### `pallet-election-provider-multi-phase` This is the old single-page `ElectionProvider`. It has been updated to work with multi-page traits, yet it only supports `page-size = 1` for now. It should not have seen any logical changes. #### `pallet-bags-list` Now has two new features. 1. It can be `Locked`, in which case all updates to it fail with an `Err(_)`, even deletion of a node. This is needed because we cannot alter any nodes in this pallet during a multi-page iteration, aka. multi-page snapshot. 2. To combat this, the same `rebag` transaction can be also be used to remove a node from the list, or add a node to the list. This is done through the `score_of` api. See the file changes and tests under `./substrate/frame/bags-list` for more info. #### RuntimeDebug -> Debug To facilitate debugging, a number of types' `RuntimeDebug` impl has been changed to `Debug`. See #3107 ## Weights Below is a summary of the weights. These are generated using `staking-async/runtimes/parachain`, which assumes 22_500 nominators divided by `32` pages for Polkadot, and 12_500 nominators divided by `16` pages in Kusama, both leading to ~700 nominators snapshotted and exported per page. Doubling these parameters would easily slash the PoV weights by half, but with 10MB PoV, these numbers should be good. Also noting that with PoV clawback, we migth get even more proof_size weight back in the runtime. Although, afaik this reclaimed value does not take compression into account. ``` #### new: polkadot/pallet_election_provider_multi_block.rs old: kusama +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +======================================================================================================================+ | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_msp | 2.41MiB | 2.41MiB | -0.03 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_rest | 3.24MiB | 3.06MiB | -5.53 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed | 3.36MiB | 3.12MiB | -7.12 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_non_terminal | 2.12MiB | 1.32MiB | -37.60 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_terminal | 4.08MiB | 2.25MiB | -44.82 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_nothing | 3.53KiB | 3.53KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_unsigned | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed_validation | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | manage | 0B | 0B | Unchanged | +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_signed.rs old: kusama +------------------------------------------------+----------------------+----------+----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +===============================================================================================================+ | pallet_election_provider_multi_block_signed.rs | bail | 43.61KiB | 82.74KiB | +89.72 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_eject | 46.54KiB | 85.80KiB | +84.35 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | clear_old_round_data | 85.23KiB | 85.17KiB | -0.06 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | submit_page | 6.95KiB | 6.90KiB | -0.70 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_not_full | 6.45KiB | 6.39KiB | -1.00 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | unset_page | 20.76KiB | 18.55KiB | -10.67 | +------------------------------------------------+----------------------+----------+----------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_unsigned.rs old: kusama +--------------------------------------------------+-------------------+----------+-----------+------------------+ | File | Extrinsic | Old | New | Change [%] | +================================================================================================================+ | pallet_election_provider_multi_block_unsigned.rs | submit_unsigned | 63.56KiB | 696.00KiB | +995.01 | |--------------------------------------------------+-------------------+----------+-----------+------------------| | pallet_election_provider_multi_block_unsigned.rs | validate_unsigned | 1.81KiB | 3.66KiB | +102.65 | +--------------------------------------------------+-------------------+----------+-----------+------------------+ #### new: polkadot/pallet_election_provider_multi_block_verifier.rs old: kusama +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +=================================================================================================================================+ | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_terminal | 1.18MiB | 1.69MiB | +42.87 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_terminal | 1.18MiB | 1.69MiB | +42.71 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_non_terminal | 1.30MiB | 450.82KiB | -66.08 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_non_terminal | 279.93KiB | 62.22KiB | -77.77 | +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ ``` <summary> note for PR authors <details> <br> ## TODO - [x] Finalize weights - [x] Lock voter list when snapshot being taken - [x] push based election - [x] OffchainWorker miner can now run on multiple pages - [x] Trimming is improved, all bounds are respected. - [x] clients pallets: add ID - [x] make election prolonged - [x] bring westend-next and ah-next to staking-next - [x] Test pre-migration to post-migration state in ahm-test. - [x] Offence reporting works without exposure info on RC (done but recheck). - [ ] staking-async fix tests - [ ] root offence testing (minimally done in migration test) - [ ] Run benchmarking - [x] ~~Add custom decoder for OffenceDetails~~. ## TODO before finalizing PR - [x] Go over again and ensure no interaction with staking-classic except by AhClient (and pallets that are going away) in Westend. Make any non used apis private. - [ ] Create diff with changes from staking-classic. ## Migration Notes - At the start of the AHM migration, trigger: `RC::pallet_staking_async_ah_client::on_migration_start()` - At the start of the AHM migration, trigger the following: - definitely filter `staking::bond` - RC: set `staking::Forcing` to `ForceNone`. - At the end of the AHM migration, trigger the following - `RC::pallet_staking_async_ah_client::on_migration_end()` - Set `AH::pallet_staking_async::ForceEra` to `Forcing::NotForcing`. - Set RC staking and pool min bond to be u32::max. ## Follow-up - [ ] Offence generation e2e test (zombienet) </details> </summary> --------- Co-authored-by: kianenigma <kian@parity.io> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
|
Hi @kianenigma , As this PR is closed, can you update the document for the new branch and commit which I need to use inorder to run the relay and assethub nodes so I can test the Staking next page? |
|
Everything is now in mater, so you can build on top of that. BTW, I have made a simple version of this for myself in https://github.com/paritytech/polkadot-scripts/blob/51a99829460322be11379ea642f0642940604b1a/src/handlers.ts#L174 |
|
Yes...I took reference from that piece of code. The PR is already there, it just need to be reviewed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi there!
I am here to discuss two things:
To not pollute this issue, I have put all the docs about the details here: https://hackmd.io/7PiBrGxxRG2ib-WRZYJZhQ
I will be out for the next two weeks, but while away Tsvetomir and @Ank4n could assist with technical details.
The text was updated successfully, but these errors were encountered: