merge queue: embarking unstable (2f37bf4) and #6995 together

book/.markdownlint.yml

@@ -8,7 +8,7 @@ MD010:
 MD013: false
 
 # MD028: set to false to allow blank line between blockquote: https://github.com/DavidAnson/markdownlint/blob/main/doc/md028.md
-# This is because the blockquotes are shown separatedly (a deisred outcome) when having a blank line in between
+# This is because the blockquotes are shown separately (a desired outcome) when having a blank line in between
 MD028: false
 
 # MD024: set siblings_only to true so that same headings with different parent headings are allowed

book/src/SUMMARY.md

@@ -2,58 +2,54 @@
 
 * [Introduction](./intro.md)
 * [Installation](./installation.md)
-  * [Pre-Built Binaries](./installation-binaries.md)
-  * [Docker](./docker.md)
-  * [Build from Source](./installation-source.md)
-  * [Raspberry Pi 4](./pi.md)
-  * [Cross-Compiling](./cross-compiling.md)
-  * [Homebrew](./homebrew.md)
-  * [Update Priorities](./installation-priorities.md)
+  * [Pre-Built Binaries](./installation_binaries.md)
+  * [Docker](./installation_docker.md)
+  * [Build from Source](./installation_source.md)
+  * [Cross-Compiling](./installation_cross_compiling.md)
+  * [Homebrew](./installation_homebrew.md)
+  * [Update Priorities](./installation_priorities.md)
 * [Run a Node](./run_a_node.md)
-* [Become a Validator](./mainnet-validator.md)
-* [Validator Management](./validator-management.md)
-  * [The `validator-manager` Command](./validator-manager.md)
-    * [Creating validators](./validator-manager-create.md)
-    * [Moving validators](./validator-manager-move.md)
-    * [Managing validators](./validator-manager-api.md)
-  * [Slashing Protection](./slashing-protection.md)
-  * [Voluntary Exits](./voluntary-exit.md)
-  * [Partial Withdrawals](./partial-withdrawal.md)
-  * [Validator Monitoring](./validator-monitoring.md)
-  * [Doppelganger Protection](./validator-doppelganger.md)
-  * [Suggested Fee Recipient](./suggested-fee-recipient.md)
-  * [Validator Graffiti](./graffiti.md)
+* [Become a Validator](./mainnet_validator.md)
+* [Validator Management](./validator_management.md)
+  * [The `validator-manager` Command](./validator_manager.md)
+    * [Creating validators](./validator_manager_create.md)
+    * [Moving validators](./validator_manager_move.md)
+    * [Managing validators](./validator_manager_api.md)
+  * [Slashing Protection](./validator_slashing_protection.md)
+  * [Voluntary Exits](./validator_voluntary_exit.md)
+  * [Validator Sweep](./validator_sweep.md)
+  * [Validator Monitoring](./validator_monitoring.md)
+  * [Doppelganger Protection](./validator_doppelganger.md)
+  * [Suggested Fee Recipient](./validator_fee_recipient.md)
+  * [Validator Graffiti](./validator_graffiti.md)
 * [APIs](./api.md)
-  * [Beacon Node API](./api-bn.md)
-    * [Lighthouse API](./api-lighthouse.md)
-    * [Validator Inclusion APIs](./validator-inclusion.md)
-  * [Validator Client API](./api-vc.md)
-    * [Endpoints](./api-vc-endpoints.md)
-    * [Authorization Header](./api-vc-auth-header.md)
-    * [Signature Header](./api-vc-sig-header.md)
-  * [Prometheus Metrics](./advanced_metrics.md)
-* [Lighthouse UI (Siren)](./lighthouse-ui.md)
-  * [Configuration](./ui-configuration.md)
-  * [Authentication](./ui-authentication.md)
-  * [Usage](./ui-usage.md)
-  * [FAQs](./ui-faqs.md)
+  * [Beacon Node API](./api_bn.md)
+    * [Lighthouse API](./api_lighthouse.md)
+    * [Validator Inclusion APIs](./api_validator_inclusion.md)
+  * [Validator Client API](./api_vc.md)
+    * [Endpoints](./api_vc_endpoints.md)
+    * [Authorization Header](./api_vc_auth_header.md)
+  * [Prometheus Metrics](./api_metrics.md)
+* [Lighthouse UI (Siren)](./ui.md)
+  * [Configuration](./ui_configuration.md)
+  * [Authentication](./ui_authentication.md)
+  * [Usage](./ui_usage.md)
+  * [FAQs](./ui_faqs.md)
 * [Advanced Usage](./advanced.md)
-  * [Checkpoint Sync](./checkpoint-sync.md)
-  * [Custom Data Directories](./advanced-datadir.md)
-  * [Proposer Only Beacon Nodes](./advanced-proposer-only.md)
-  * [Remote Signing with Web3Signer](./validator-web3signer.md)
+  * [Checkpoint Sync](./advanced_checkpoint_sync.md)
+  * [Custom Data Directories](./advanced_datadir.md)
+  * [Proposer Only Beacon Nodes](./advanced_proposer_only.md)
+  * [Remote Signing with Web3Signer](./advanced_web3signer.md)
   * [Database Configuration](./advanced_database.md)
-  * [Database Migrations](./database-migrations.md)
-  * [Key Management (Deprecated)](./key-management.md)
-  * [Key Recovery](./key-recovery.md)
+  * [Database Migrations](./advanced_database_migrations.md)
+  * [Key Recovery](./advanced_key_recovery.md)
   * [Advanced Networking](./advanced_networking.md)
-  * [Running a Slasher](./slasher.md)
-  * [Redundancy](./redundancy.md)
-  * [Release Candidates](./advanced-release-candidates.md)
-  * [MEV](./builders.md)
-  * [Merge Migration](./merge-migration.md)
-  * [Late Block Re-orgs](./late-block-re-orgs.md)
-  * [Blobs](./advanced-blobs.md)
+  * [Running a Slasher](./advanced_slasher.md)
+  * [Redundancy](./advanced_redundancy.md)
+  * [Release Candidates](./advanced_release_candidates.md)
+  * [MEV](./advanced_builders.md)
+  * [Late Block Re-orgs](./advanced_re-orgs.md)
+  * [Blobs](./advanced_blobs.md)
 * [Command Line Reference (CLI)](./help_general.md)
   * [Beacon Node](./help_bn.md)
   * [Validator Client](./help_vc.md)
@@ -62,7 +58,11 @@
     * [Import](./help_vm_import.md)
     * [Move](./help_vm_move.md)
 * [Contributing](./contributing.md)
-  * [Development Environment](./setup.md)
+  * [Development Environment](./contributing_setup.md)
 * [FAQs](./faq.md)
 * [Protocol Developers](./developers.md)
 * [Security Researchers](./security.md)
+* [Archived](./archived.md)
+  * [Merge Migration](./archived_merge_migration.md)
+  * [Raspberry Pi 4](./archived_pi.md)
+  * [Key Management](./archived_key_management.md)

book/src/advanced.md

@@ -6,19 +6,17 @@ elsewhere?
 This section provides detailed information about configuring Lighthouse for specific use cases, and
 tips about how things work under the hood.
 
-* [Checkpoint Sync](./checkpoint-sync.md): quickly sync the beacon chain to perform validator duties.
-* [Custom Data Directories](./advanced-datadir.md): modify the data directory to your preferred location.
-* [Proposer Only Beacon Nodes](./advanced-proposer-only.md): beacon node only for proposer duty for increased anonymity.
-* [Remote Signing with Web3Signer](./validator-web3signer.md): don't want to store your keystore in local node? Use web3signer.
+* [Checkpoint Sync](./advanced_checkpoint_sync.md): quickly sync the beacon chain to perform validator duties.
+* [Custom Data Directories](./advanced_datadir.md): modify the data directory to your preferred location.
+* [Proposer Only Beacon Nodes](./advanced_proposer_only.md): beacon node only for proposer duty for increased anonymity.
+* [Remote Signing with Web3Signer](./advanced_web3signer.md): don't want to store your keystore in local node? Use web3signer.
 * [Database Configuration](./advanced_database.md): understanding space-time trade-offs in the database.
-* [Database Migrations](./database-migrations.md): have a look at all previous Lighthouse database scheme versions.
-* [Key Management](./key-management.md): explore how to generate wallet with Lighthouse.
-* [Key Recovery](./key-recovery.md): explore how to recover wallet and validator with Lighthouse.
+* [Database Migrations](./advanced_database_migrations.md): have a look at all previous Lighthouse database scheme versions.
+* [Key Recovery](./advanced_key_recovery.md): explore how to recover wallet and validator with Lighthouse.
 * [Advanced Networking](./advanced_networking.md): open your ports to have a diverse and healthy set of peers.
-* [Running a Slasher](./slasher.md): contribute to the health of the network by running a slasher.
-* [Redundancy](./redundancy.md): want to have more than one beacon node as backup? This is for you.
-* [Release Candidates](./advanced-release-candidates.md): latest release of Lighthouse to get feedback from users.
-* [Maximal Extractable Value](./builders.md): use external builders for a potential higher rewards during block proposals
-* [Merge Migration](./merge-migration.md): look at what you need to do during a significant network upgrade: The Merge
-* [Late Block Re-orgs](./late-block-re-orgs.md): read information about Lighthouse late block re-orgs.
-* [Blobs](./advanced-blobs.md): information about blobs in Deneb upgrade
+* [Running a Slasher](./advanced_slasher.md): contribute to the health of the network by running a slasher.
+* [Redundancy](./advanced_redundancy.md): want to have more than one beacon node as backup? This is for you.
+* [Release Candidates](./advanced_release_candidates.md): latest release of Lighthouse to get feedback from users.
+* [Maximal Extractable Value](./advanced_builders.md): use external builders for a potential higher rewards during block proposals
+* [Late Block Re-orgs](./advanced_re-orgs.md): read information about Lighthouse late block re-orgs.
+* [Blobs](./advanced_blobs.md): information about blobs in Deneb upgrade

book/src/advanced-blobs.md → book/src/advanced_blobs.md

@@ -38,4 +38,4 @@ In the Deneb network upgrade, one of the changes is the implementation of EIP-48
    curl "http://localhost:5052/lighthouse/database/info" | jq
    ```
 
-   Refer to [Lighthouse API](./api-lighthouse.md#lighthousedatabaseinfo) for an example response.
+   Refer to [Lighthouse API](./api_lighthouse.md#lighthousedatabaseinfo) for an example response.

book/src/builders.md → book/src/advanced_builders.md

@@ -83,11 +83,11 @@ is something afoot.
 
 To update gas limit per-validator you can use the [standard key manager API][gas-limit-api].
 
-Alternatively, you can use the [lighthouse API](api-vc-endpoints.md). See below for an example.
+Alternatively, you can use the [lighthouse API](api_vc_endpoints.md). See below for an example.
 
 ### Enable/Disable builder proposals via HTTP
 
-Use the [lighthouse API](api-vc-endpoints.md) to enable/disable use of the builder API on a per-validator basis.
+Use the [lighthouse API](api_vc_endpoints.md) to enable/disable use of the builder API on a per-validator basis.
 You can also update the configured gas limit with these requests.
 
 #### `PATCH /lighthouse/validators/:voting_pubkey`
@@ -98,7 +98,7 @@ You can also update the configured gas limit with these requests.
 |-------------------|--------------------------------------------|
 | Path              | `/lighthouse/validators/:voting_pubkey`    |
 | Method            | PATCH                                      |
-| Required Headers  | [`Authorization`](./api-vc-auth-header.md) |
+| Required Headers  | [`Authorization`](./api_vc_auth_header.md) |
 | Typical Responses | 200, 400                                   |
 
 #### Example Path
@@ -147,7 +147,7 @@ INFO Published validator registrations to the builder network, count: 3, service
 
 ### Fee Recipient
 
-Refer to [suggested fee recipient](suggested-fee-recipient.md) documentation.
+Refer to [suggested fee recipient](validator_fee_recipient.md) documentation.
 
 ### Validator definitions example
 
@@ -244,16 +244,9 @@ INFO Builder payload ignored
 INFO Chain is unhealthy, using local payload
 ```
 
-In case of fallback you should see a log indicating that the locally produced payload was
-used in place of one from the builder:
-
-```text
-INFO Reconstructing a full block using a local payload
-```
-
 ## Information for block builders and relays
 
-Block builders and relays can query beacon node events from the [Events API](https://ethereum.github.io/beacon-APIs/#/Events/eventstream). An example of querying the payload attributes in the Events API is outlined in [Beacon node API - Events API](./api-bn.md#events-api)
+Block builders and relays can query beacon node events from the [Events API](https://ethereum.github.io/beacon-APIs/#/Events/eventstream). An example of querying the payload attributes in the Events API is outlined in [Beacon node API - Events API](./api_bn.md#events-api)
 
 [mev-rs]: https://github.com/ralexstokes/mev-rs
 [mev-boost]: https://github.com/flashbots/mev-boost

book/src/checkpoint-sync.md → book/src/advanced_checkpoint_sync.md

@@ -134,7 +134,7 @@ Important information to be aware of:
 * It is safe to interrupt state reconstruction by gracefully terminating the node – it will pick up
   from where it left off when it restarts.
 * You can start reconstruction from the HTTP API, and view its progress. See the
-  [`/lighthouse/database`](./api-lighthouse.md) APIs.
+  [`/lighthouse/database`](./api_lighthouse.md) APIs.
 
 For more information on historic state storage see the
 [Database Configuration](./advanced_database.md) page.

book/src/advanced_database.md

@@ -61,6 +61,26 @@ that we have observed are:
   to apply. We observed no significant performance benefit from `--hierarchy-exponents 5,7,11`, and
   a substantial increase in space consumed.
 
+The following table lists the data for different configurations. Note that the disk space requirement is for the `chain_db` and `freezer_db`, excluding the `blobs_db`.
+
+| Hierarchy Exponents | Storage Requirement | Sequential Slot Query | Uncached Query | Time to Sync |
+|---|---|---|---|---|
+| 5,9,11,13,16,18,21 (default) | 418 GiB | 250-700 ms | up to 10 s | 1 week |
+| 5,7,11 (frequent snapshots) | 589 GiB | 250-700 ms | up to 6 s | 1 week |
+| 0,5,7,11 (per-slot diffs) | 2500 GiB | 250-700 ms | up to 4 s | 7 weeks |
+
+[Jim](https://github.com/mcdee) has done some experiments to study the response time of querying random slots (uncached query) for `--hierarchy-exponents 0,5,7,11` (per-slot diffs) and `--hierarchy-exponents 5,9,11,13,17,21` (per-epoch diffs), as show in the figures below. From the figures, two points can be concluded:
+
+- response time (y-axis) increases with slot number (x-axis) due to state growth.
+- response time for per-slot configuration in general is 2x faster than that of per-epoch.
+
+In short, setting different configurations is a trade-off between disk space requirement, sync time and response time. The data presented here is useful to help users choosing the configuration that suit their needs.
+
+_We acknowledge the data provided by [Jim](https://github.com/mcdee) and his consent for us to share it here._
+
+![Response time for per-epoch archive](./imgs/per-epoch.png)
+![Response time for per-slot archive](./imgs/per-slot.png)
+
 If in doubt, we recommend running with the default configuration! It takes a long time to
 reconstruct states in any given configuration, so it might be some time before the optimal
 configuration is determined.

book/src/advanced-datadir.md → book/src/advanced_datadir.md

@@ -12,7 +12,7 @@ lighthouse --network mainnet --datadir /var/lib/my-custom-dir bn --staking
 lighthouse --network mainnet --datadir /var/lib/my-custom-dir vc
 ```
 
-The first step creates a `validators` directory under `/var/lib/my-custom-dir` which contains the imported keys and [`validator_definitions.yml`](./validator-management.md).
+The first step creates a `validators` directory under `/var/lib/my-custom-dir` which contains the imported keys and [`validator_definitions.yml`](./validator_management.md).
 After that, we simply run the beacon chain and validator client with the custom dir path.
 
 ## Relative Paths

book/src/advanced_networking.md

@@ -123,8 +123,12 @@ Lighthouse listens for connections, and the parameters used to tell other peers
 how to connect to your node. This distinction is relevant and applies to most
 nodes that do not run directly on a public network.
 
+Since Lighthouse v7.0.0, Lighthouse listens to both IPv4 and IPv6 by default if it detects a globally routable IPv6 address. This means that dual-stack is enabled by default.
+
 ### Configuring Lighthouse to listen over IPv4/IPv6/Dual stack
 
+To listen over only IPv4 and not IPv6, use the flag `--listen-address 0.0.0.0`.
+
 To listen over only IPv6 use the same parameters as done when listening over
 IPv4 only:
 
@@ -136,7 +140,7 @@ TCP and UDP.
   If the specified port is 9909, QUIC will use port 9910 for IPv6 UDP connections.
   This can be configured with `--quic-port`.
 
-To listen over both IPv4 and IPv6:
+To listen over both IPv4 and IPv6 and using a different port for IPv6::
 
 - Set two listening addresses using the `--listen-address` flag twice ensuring
   the two addresses are one IPv4, and the other IPv6. When doing so, the
@@ -165,7 +169,7 @@ To listen over both IPv4 and IPv6:
 > It listens on the default value of --port6 (`9000`) for both UDP and TCP.
 > QUIC will use port `9001` for UDP, which is the default `--port6` value (`9000`) + 1.
 
-> When using `--listen-address :: --listen-address --port 9909 --discovery-port6 9999`, listening will be set up as follows:
+> When using `--listen-address :: --listen-address 0.0.0.0 --port 9909 --discovery-port6 9999`, listening will be set up as follows:
 >
 > **IPv4**:
 >

book/src/advanced-proposer-only.md → book/src/advanced_proposer_only.md

@@ -56,7 +56,7 @@ these nodes for added security).
 
 The intended set-up to take advantage of this mechanism is to run one (or more)
 normal beacon nodes in conjunction with one (or more) proposer-only beacon
-nodes. See the [Redundancy](./redundancy.md) section for more information about
+nodes. See the [Redundancy](./advanced_redundancy.md) section for more information about
 setting up redundant beacon nodes. The proposer-only beacon nodes should be
 setup to use a different IP address than the primary (non proposer-only) nodes.
 For added security, the IP addresses of the proposer-only nodes should be

book/src/redundancy.md → book/src/advanced_redundancy.md

@@ -9,7 +9,7 @@ There are three places in Lighthouse where redundancy is notable:
 We mention (3) since it is unsafe and should not be confused with the other two
 uses of redundancy. **Running the same validator keypair in more than one
 validator client (Lighthouse, or otherwise) will eventually lead to slashing.**
-See [Slashing Protection](./slashing-protection.md) for more information.
+See [Slashing Protection](./validator_slashing_protection.md) for more information.
 
 From this paragraph, this document will *only* refer to the first two items (1, 2). We
 *never* recommend that users implement redundancy for validator keypairs.
@@ -58,8 +58,8 @@ following flags:
 
  > Note: You could also use `--http-address 0.0.0.0`, but this allows *any* external IP address to access the HTTP server. As such, a firewall should be configured to deny unauthorized access to port `5052`.
 
-- `--execution-endpoint`: see [Merge Migration](./merge-migration.md).
-- `--execution-jwt`: see [Merge Migration](./merge-migration.md).
+- `--execution-endpoint`: see [Merge Migration](./archived_merge_migration.md).
+- `--execution-jwt`: see [Merge Migration](./archived_merge_migration.md).
 
 For example one could use the following command to provide a backup beacon node:
 
@@ -107,7 +107,7 @@ The default is `--broadcast subscriptions`. To also broadcast blocks for example
 Lighthouse previously supported redundant execution nodes for fetching data from the deposit
 contract. On merged networks *this is no longer supported*. Each Lighthouse beacon node must be
 configured in a 1:1 relationship with an execution node. For more information on the rationale
-behind this decision please see the [Merge Migration](./merge-migration.md) documentation.
+behind this decision please see the [Merge Migration](./archived_merge_migration.md) documentation.
 
 To achieve redundancy we recommend configuring [Redundant beacon nodes](#redundant-beacon-nodes)
 where each has its own execution engine.

book/src/slasher.md → book/src/advanced_slasher.md

@@ -81,7 +81,7 @@ WARN Slasher backend override failed    advice: delete old MDBX database or enab
 
 In this case you should either obtain a Lighthouse binary with the MDBX backend enabled, or delete
 the files for the old backend. The pre-built Lighthouse binaries and Docker images have MDBX enabled,
-or if you're [building from source](./installation-source.md) you can enable the `slasher-mdbx` feature.
+or if you're [building from source](./installation_source.md) you can enable the `slasher-mdbx` feature.
 
 To delete the files, use the `path` from the `WARN` log, and then delete the `mbdx.dat` and
 `mdbx.lck` files.