[ZK Stack] - Syncing ZKSync Era external node from genesis but got root hash mismatched error

[cifer76]

Team or Project

No response

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.82.0-nightly

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

I wanted sync an new Era external node from genesis, I have followed the external node docker example: https://github.com/matter-labs/zksync-era/blob/main/docs/src/guides/external-node/docker-compose-examples/mainnet-external-node-docker-compose.yml, kept most of the configs in the example, except disabling snapshot recovery (I need to get all of the traces from the genesis).

But the external node just refuses to run, with a "Root hash mismatched" error (see below)

Expected Behavior

The external node should run smoothly and sync from the genesis

Current Behavior

geth-1      | 2025-03-24T15:46:10.803662Z  INFO NamedFuture{name=node_storage_initializer}: zksync_node_storage_init: Node has neither genesis L1 batch, nor snapshot recovery info
geth-1      | 2025-03-24T15:46:10.803677Z  INFO NamedFuture{name=node_storage_initializer}: zksync_node_storage_init: Performing genesis initialization
geth-1      | 2025-03-24T15:46:10.808706Z  INFO NamedFuture{name=l1_batch_commitment_mode_validation}: zksync_commitment_generator::validation_task: Checked that the configured L1 batch commitment mode (Rollup) matches the commitment mode used on L1 contract 0x32400084c286cf3e17e7b677ea9583e60a000324
geth-1      | 2025-03-24T15:46:10.810728Z  INFO NamedFuture{name=validate_chain_ids}: zksync_node_sync::validate_chain_ids_task: Checked that chain ID 1 is returned by client
geth-1      | 2025-03-24T15:46:11.012215Z  INFO NamedFuture{name=validate_chain_ids}: zksync_node_sync::validate_chain_ids_task: Checked that L1 chain ID 1 is returned by main node client
geth-1      | 2025-03-24T15:46:11.016551Z  INFO NamedFuture{name=validate_chain_ids}: zksync_node_sync::validate_chain_ids_task: Checked that L2 chain ID L2ChainId(324) is returned by main node client
geth-1      | 2025-03-24T15:46:22.175289Z  INFO NamedFuture{name=node_storage_initializer}: zksync_node_genesis: running regenesis
geth-1      | 2025-03-24T15:46:22.190474Z  INFO NamedFuture{name=node_storage_initializer}: zksync_node_genesis: chain_schema_genesis is complete
geth-1      | 2025-03-24T15:46:22.190516Z  INFO NamedFuture{name=node_storage_initializer}: zksync_merkle_tree::domain: Creating Merkle tree for genesis batch with 35  writes
geth-1      | 2025-03-24T15:46:22.191786Z  INFO NamedFuture{name=node_storage_initializer}: zksync_merkle_tree::domain: Processed genesis batch; root hash is 0x79b3…ffe3, 35 leaves in total
geth-1      | 2025-03-24T15:46:22.193299Z  INFO NamedFuture{name=node_storage_initializer}: zksync_db_connection::connection: Long-living connection for `en` created at node/node_storage_init/src/external_node/genesis.rs:24: 11.387190193s
geth-1      | 2025-03-24T15:46:22.193413Z ERROR zksync_node_framework::service: Task oneshot_runner failed: Oneshot task node_storage_initializer failed
geth-1      |
geth-1      | Caused by:
geth-1      |     0: performing genesis failed
geth-1      |     1: ensure_genesis_state
geth-1      |     2: Root hash mismatched: From config: 0xbc59c242d551e3939b9b2939b8b686efa77ba3833183045d548aa5f53357ba95, Calculated 0x79b383138554ae78daa0cd096ca63d7e69dc7e1868daf3c9a86d01918df1ffe3
geth-1      | 2025-03-24T15:46:22.193436Z  INFO zksync_node_framework::service: One of the task has exited, shutting down the node
geth-1      | 2025-03-24T15:46:22.193472Z  INFO zksync_node_api_server::healthcheck: Stop signal received, healthcheck server is shutting down
geth-1      | 2025-03-24T15:46:22.193498Z  INFO zksync_node_framework::implementations::layers::postgres: Stop signal received, Postgres metrics scraping is shutting down
geth-1      | 2025-03-24T15:46:22.193612Z  INFO zksync_node_api_server::healthcheck: Healthcheck server shut down

Repository Link (if applicable)

No response

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[sarahschwartz]

Hi @windycrypto! This likely caused by a mismatched dependency. Can you try these steps and try again?

• make sure you are using the same version of forge as here
• check that a node_modules folder is not present in l1-contracts and then delete its artifacts and recompile

[cifer76]

Yes we do need all of the historical traces and state diffs, which is not in the recovery snapshot, syncing from the genesis is now the only way to do that. So it's really appreciated if it can be supported.
We can accept the months-length syncing time, it's a long-term project we are working so months is acceptable for us

[sarahschwartz]

@windycrypto you should be able to use this repo along with the full mainnet archive EN dump available at https://en-backups.matterlabs.dev/ to get this data.

We prefer to keep technical discussions on this forum so they can be referenced by others in the future, but if there is something else you need to discuss that could work.

[cifer76]

Hi @sarahschwartz I have downloaded the repo but it doesn't well suit our needs, we do need sync from the genesis, to derive some customized data, which do not exist in the raw database. Could you help proceed this again?

[popzxc]

Sorry for the late reply.
Right now, the genesis sync is not a recommended way to run the external node, as the synchronization process was never optimized for that use case. As a result, we don't perform continuous testing for genesis sync, and cannot guarantee that there won't be any issues (the EN is guaranteed to never serve incorrect state, but there might be issues related to syncrhonization, especially on the older blocks).

However, regarding getting customized data that doesn't exist in the raw database: the database has all the data the node was capable of producing when it executed each corresponding block. I might try to guess that you might want to access traces for very old blocks; unfortunately, I don't think it would work. In the first few protocol versions, the VM was unable to produce traces for transactions, so re-running these blocks will likely not produce traces, unless we make changes to the old VM versions.

[cifer76]

@popzxc thank you, we are making changes to the old VM versions, we have already done that, just need further polish, so it will be really appreciated if we can make genesis sync possible. Thank you!

[popzxc]

Unfortunately, right now we don't have engineering resources to help with that.
However, given that you're modifying the code already, I can give a few ideas of how I would approach it:

1. The error you're discovering seems to be originating from the genesis initalization. Likely, it fetches the genesis root hash from the main node and compares it to the one in config. Check the RPC request it does to fetch the genesis root hash and put it into your config, so that node can launch.
2. Once syncrhonization starts, watch for consistency_checker error: it's a component that makes sure that the state is the same as on the main node. Note that this component runs after the batches are executed, so e.g. state keeper can execute batch X, but consistency checker would fail at batch X - 2.
3. If syncrhonization fails at a certain batch, compare the execution data (e.g. data in the database) against the data in the dump to try to identify the transaction that causes issues with the syncrhonization. From there, you will need to go down into the corresponding VM logic and find the divergence.
4. VM is developed in a way that there are many VM versions for different protocol versions (something like "hardforks" for ZKsync); however, some versions have sub-versions if the changes were made only on the bootloader level (e.g. memory layout changed, but the VM is the same). If any errors happen, most likely the reason is that wrong VM version is chosen by the state keeper for the batch. In the database, we have the data for all the protocol upgrades that tells the state keeper which VM version to choose. For newer protocol versions, this information is passed through L1, but for older ones I'm not sure if EN can sync. What you can do here is to copy protocol versions data from the dump to the EN.

From here, it'll be mostly be about observing the behavior and fixing issues one by one as (if) they occur. Hope that helps.

[popzxc]

Unfortunately, right now we don't have engineering resources to help with that.
However, given that you're modifying the code already, I can give a few ideas of how I would approach it:

1. The error you're discovering seems to be originating from the genesis initalization. Likely, it fetches the genesis root hash from the main node and compares it to the one in config. Check the RPC request it does to fetch the genesis root hash and put it into your config, so that node can launch.
2. Once syncrhonization starts, watch for consistency_checker error: it's a component that makes sure that the state is the same as on the main node. Note that this component runs after the batches are executed, so e.g. state keeper can execute batch X, but consistency checker would fail at batch X - 2.
3. If syncrhonization fails at a certain batch, compare the execution data (e.g. data in the database) against the data in the dump to try to identify the transaction that causes issues with the syncrhonization. From there, you will need to go down into the corresponding VM logic and find the divergence.
4. VM is developed in a way that there are many VM versions for different protocol versions (something like "hardforks" for ZKsync); however, some versions have sub-versions if the changes were made only on the bootloader level (e.g. memory layout changed, but the VM is the same). If any errors happen, most likely the reason is that wrong VM version is chosen by the state keeper for the batch. In the database, we have the data for all the protocol upgrades that tells the state keeper which VM version to choose. For newer protocol versions, this information is passed through L1, but for older ones I'm not sure if EN can sync. What you can do here is to copy protocol versions data from the dump to the EN.

From here, it'll be mostly be about observing the behavior and fixing issues one by one as (if) they occur. Hope that helps.Unfortunately, right now we don't have engineering resources to help with that.
However, given that you're modifying the code already, I can give a few ideas of how I would approach it:

1. The error you're discovering seems to be originating from the genesis initalization. Likely, it fetches the genesis root hash from the main node and compares it to the one in config. Check the RPC request it does to fetch the genesis root hash and put it into your config, so that node can launch.
2. Once syncrhonization starts, watch for consistency_checker error: it's a component that makes sure that the state is the same as on the main node. Note that this component runs after the batches are executed, so e.g. state keeper can execute batch X, but consistency checker would fail at batch X - 2.
3. If syncrhonization fails at a certain batch, compare the execution data (e.g. data in the database) against the data in the dump to try to identify the transaction that causes issues with the syncrhonization. From there, you will need to go down into the corresponding VM logic and find the divergence.
4. VM is developed in a way that there are many VM versions for different protocol versions (something like "hardforks" for ZKsync); however, some versions have sub-versions if the changes were made only on the bootloader level (e.g. memory layout changed, but the VM is the same). If any errors happen, most likely the reason is that wrong VM version is chosen by the state keeper for the batch. In the database, we have the data for all the protocol upgrades that tells the state keeper which VM version to choose. For newer protocol versions, this information is passed through L1, but for older ones I'm not sure if EN can sync. What you can do here is to copy protocol versions data from the dump to the EN.

From here, it'll be mostly be about observing the behavior and fixing issues one by one as (if) they occur. Hope that helps.

[cifer76]

Thank you @popzxc so much for the detailed information. We didn't change the original VM logic just add some new fields to the Call structure to includes more context about the call in the generated call_traces.

That's why we need genesis sync, because we need the historical call traces to have the new fields too.

Currently we're trying to re-creating the historical call traces by modifying the debug_* calls, instead of read traces directly from the call_traces table, we re-executing the transactions using the execute_in_sandbox, but it seems the resulting call traces are different with the call traces stored in the postgres call_traces table.

I find in the postgres miniblocks table older l2 blocks(blocks before 13083680) are without a protocol id, and during debug_* call the blocks without protocol id are default to Version9, which may be one reason causes the divergence. But there're divergences in debug_* calls generated traces and syncing generated traces on new blocks too. It seems not easy to fix the divergences in the debug_* calls.

So we are still resorting to genesis sync as a preferred approach