[ZK Stack] - System has failed L1 transaction

[sachanayush47]

Team or Project

Validium Network

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.82.0

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

We have self deployed a zksync hyperchain running from last 15 days and then suddenly server stops with this error ERROR zksync_node_framework::service: Task circuit_breaker_checker failed: Circuit breaker error. Reason: System has failed L1 transaction.
We found out that we accidentally made a transaction from chain's blob operator wallet.

We tried using the block_reverter -- clear-failed-transactions but it doesnt seems to fix the thing properly.

After successfully running the block_reverter -- clear-failed-transactions, a new error comes i.e Error: One or more tasks failed: ["Task eth_tx_manager panicked: We can't operate after tx fail", "Shutdown for task web3_http_server timed out", "Shutdown for task api_task_garbage_collector timed out"] and server shutdown and then if we again try to restart the chain old error comes back; System has failed L1 transaction,

Failed Transaction on Blob operator address: https://holesky.etherscan.io/tx/0x7ffce6e26073cef66b6c7769827fa852dc9235e3f6f12abcb07c7474064d8cdb

Old discussion on the same issue: #764

Expected Behavior

Server should be running normally and submitting transactions to L1 holesky

Current Behavior

Server failed to start

Repository Link (if applicable)

No response

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[alhenaai[bot]]

It appears that the issue you're encountering with the zksync_node_framework::service error and subsequent failures even after using the block_reverter -- clear-failed-transactions tool is quite complex and might be due to deeper systemic issues within your zkSync hyperchain setup.

Given the recurring nature of the error and the failure of the block reverter tool to resolve the issue permanently, I recommend the following steps:

1. Review the Entire System Setup: Ensure that all configurations and deployments are according to the zkSync standards and that there are no underlying issues with the system contracts or the server setup itself.

2. Consult the zkSync Developer Community: Since this issue seems to be quite specific and technical, engaging with the zkSync developer community might provide more targeted advice or solutions that have worked for others in similar situations. You can post your detailed issue here: zkSync Developers Discussions.

3. Check for Updates or Patches: Sometimes, issues like these are resolved in newer updates or patches. Make sure your zkSync components are up to date.

4. Detailed Logging and Monitoring: If not already in place, enhance logging and monitoring around the transaction submissions and server operations to capture more detailed error information when the server fails. This might give more insight into what exactly is failing and why.

For more in-depth technical support, you might also consider directly reaching out to the zkSync technical team if the problem persists and is critical to your operations.

Your feedback helps us serve you better. If this fully addresses your question, please give it a thumbs up. If it doesn't, please give it a thumbs down. This will inform the community and team so we can improve our support.

[bxpana]

I'll see if we can get someone to look into this and why this keeps reoccurring for you

[sachanayush47]

Sure. This happened to us around 3 times. 2 times when we use the same sets of wallets for multiple chains under an ecosystem and 3rd time when we bridged some eth from the blob operator to our chain.

[sanekmelnikov]

Hi! Let me share my understanding, and @bxpana will add more, if someone corrects me.

For each batch we must do a triplet of [commit, prove, execute] txs.
commit is sent by blob_operator
prove, execute - by operator
To maintain the chain state valid, each operator can't send the next transaction if the previous one failed. That's why the server is crashing.

I think, usually, after inspecting what's going on, you can just clean failed transaction with block_reverter, and try again.

But in this case:
Imagine blob_operator is at nonce 2.
Server adds two new batches (2, 3) into database and tries to send commit txs with nonces 2 and 3.
Meanwhile someone sends another transaction from blob_operator with nonce 2 and it's included into the blockchain.
Server still tries to send tx with nonce 2 - but keeps getting RPC error "nonce too low".
I guess it's safe to fail at this point, but we are not good at parsing RPC errors :)
Server also tries to send tx with nonce 3 and, despite of having correct nonce, execution reverts (since previous batch was not committed), and tx fails.
And you end up in a situation like this:

# SELECT id, confirmed_eth_tx_history_id, nonce, tx_type, has_failed FROM eth_txs;

 id  | confirmed_eth_tx_history_id | nonce |          tx_type          | has_failed 
-----+-----------------------------+-------+---------------------------+------------
   1 |                           1 |     1 | CommitBlocks              | f
   2 |                           2 |     1 | PublishProofBlocksOnchain | f
   3 |                           3 |     2 | ExecuteBlocks             | f
   4 |                             |     2 | CommitBlocks              | f
   5 |                             |     3 | CommitBlocks              | t

So blob_operator is at nonce 3 and cleaning failed tx won't help, because server will try to resend tx with nonce 2, will get "nonce too low", will send another tx with nonce 4 and will fail.

I guess the recommendation is to check that's what happened and clean all unconfirmed txs from blob_operator:
DELETE FROM eth_txs WHERE confirmed_eth_tx_history_id IS NULL AND type = 'CommitBlocks';
(And have dedicated operators for each chain - don't share them across the chains, don't use them elsewhere)

[sachanayush47]

Hi, thanks for the detailed explanation. We have redeployed everything again so unable to try the suggest solution but Im planning to reproduce the issue to try out the suggested fix. Will update very soon on this.

[bxpana]

Going to mark this as answered for now, but if it comes back just unmark it as answered and respond. We should get a notification to work on this again.

[sachanayush47]

Hi @sanekmelnikov @bxpana
We encountered the issue again, and this time we didn't make any transactions from protected wallets. We attempted the suggested solution, but it didn't seem to work.

After running the block-reverter tool, the transactions were cleared. However, the following SQL query:
DELETE FROM eth_txs WHERE confirmed_eth_tx_history_id IS NULL AND tx_type = 'CommitBlocks';
does not delete any records, as they were already cleared by the block-reverter.

[sachanayush47]

Block reverter is throwing error zksync_protobuf_config: Failed to parse config: universal_setup_path: missing after updating zksync-era to the latest version. Til now, we were slightly older version released 2 months back.

[sanekmelnikov]

zksync_protobuf_config error refers to the changes in configs.
There are few things to be careful about when updating zksync-era if you intend to run a new components with an existing chain:

1. There might be changes in configs

• You can check https://github.com/matter-labs/zksync-era/tree/main/etc/env/file_based to see what's changed

2. There might be changes to DB schema:

• zkstack dev database migrate --core [DB_NAME] --core-url [DB_URL] --prover ... applies new migrations from the repo

3. New code might only be compatible with newer protocol version (no easy way around it unfortunately, but shouldn't affect block-reverter)

I recommend using zkstack update for updating repository since it will do steps 1 and 2 for you.

[sanekmelnikov]

On your original issue:

From tx history of blob operator, it seems that it's trying to send the right batch, but tx is still reverted.
I've started the anvil fork locally
anvil --fork-url [HOLESKY_RPC_URL] --auto-impersonate --port 9000
And replayed tx to get more details (commitBatchesSharedBridge):
cast run 0x2e0b031804e31ffa88cc9b124893258b966012697260ae737e9ec0137f2d1038 --rpc-url localhost:9000
I got [Revert] custom error 0x08753982
Here I checked that this error corresponds to TimeNotReached error.
Which I think happens here.

I don't know whether it's because I'm replaying it 3 days after it happened or whether this was an initial error, but at this point we will still have to run block-reverter so that batch will be created with the new timestamp to avoid TimeNotReached error. Once we do that, you can try to run everything above again and see if you'll get another error.

On a second note, the frequency of sending batches from operators is way too high. I think this might contribute to an error. This is era blob operator for comparison - sends batches every 2 hours.
I'd suggest overriding your general.yaml config with these parameters.
zkstack dev config-writer --path $ZKSYNC_HOME/etc/env/file_based/overrides/testnet.yaml

[sanekmelnikov]

Ok, actually it's a pretty tricky one and tx actually fails with TimeNotReached error.
I think, because of an aggressive default config settings (which are suitable for localhost setup), the gap between batch timestamp and commitBatchTx timestamp is growing over time:

batch# batch_ts   commit_ts    gap
24740 1736227348 1736486496 259148
24739 1736227343 1736486460 259117
...
23000 1736219656 1736413032 193376
...

It grows until it hits 3 days = 259,200s (maximum allowed gap) and then your commit tx fails.
The proper fix is reverting the latest batch and restarting the server with configs suggestions in my previous comment.

[sachanayush47]

On the another server where I reproduced the same issue, running these 2 commands fixed the issue.

./block_reverter clear-failed-transactions --config-path $ZK_CHAIN_CONFIG_ROOT/configs/general.yaml --contracts-config-path $ZK_CHAIN_CONFIG_ROOT/configs/contracts.yaml  --secrets-path $ZK_CHAIN_CONFIG_ROOT/configs/secrets.yaml  --wallets-path $ZK_CHAIN_CONFIG_ROOT/configs/wallets.yaml  --genesis-path $ZK_CHAIN_CONFIG_ROOT/configs/genesis.yaml

./block_reverter rollback-db --config-path $ZK_CHAIN_CONFIG_ROOT/configs/general.yaml --contracts-config-path $ZK_CHAIN_CONFIG_ROOT/configs/contracts.yaml  --secrets-path $ZK_CHAIN_CONFIG_ROOT/configs/secrets.yaml  --wallets-path $ZK_CHAIN_CONFIG_ROOT/configs/wallets.yaml  --genesis-path $ZK_CHAIN_CONFIG_ROOT/configs/genesis.yaml --l1-batch-number 0 --rollback-postgres --rollback-tree --rollback-sk-cache --rollback-vm-runners-cache

For main server, --l1-batch-number is 24730

[sachanayush47]

I have replicated the state of main server to another dev environment and tried to run the block-reverter clear-failed-transactions and rollback-db command, around 2515 Tx permanently wiped out. Can this be done without wiping out any Tx?

[sachanayush47]

Update: Tried the block-reverter rollback-db and 2516 permanently wiped out and started the server. Server is running without issue until I send a transaction, after sending a transaction server crashed with System has failed L1 transaction within 2 minutes.

Error before server crash:
ERROR EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Eth tx failed EthTx { id: 58158, nonce: Nonce(41423), contract_address: 0x3bcb4356a6a6c841e53573f72b1e7f6ae68dbcfe, tx_type: Commit, created_at_timestamp: 1737032846, predicted_gas_cost: 294528, chain_id: Some(SLChainId(17000)) }, TransactionReceipt { transaction_hash: 0x988758928d000822d990f6429c71986db754f05f3f1e9581b3e3d151d46c57a4, transaction_index: 0, block_hash: Some(0x0e141549a30cfeec769fd9d0e395686ccb531e2d1b9f85e87412668c7782223b), block_number: Some(3157138), from: 0xb2c2c467e0ef64070b56eec0ae499174912de565, to: Some(0x3bcb4356a6a6c841e53573f72b1e7f6ae68dbcfe), cumulative_gas_used: 91033, gas_used: Some(91033), contract_address: None, logs: [], status: Some(0), root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, transaction_type: Some(2), effective_gas_price: Some(3260253883) }, failure reason Some(FailureInfo { revert_code: 3, revert_reason: "", gas_used: Some(91033), gas_limit: 15000000 })

Edit:
This error is also continuously being logged:
Jan 16 13:20:41 validium-testnet start.sh[1900816]: 2025-01-16T13:20:41.205461Z ERROR EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Possible block reorgs: finalized nonce increase detected, but no tx receipt found for tx EthTx { id: 44957, nonce: Nonce(51042), contract_address: 0x3bcb4356a6a6c841e53573f72b1e7f6ae68dbcfe, tx_type: PublishProofOnchain, created_at_timestamp: 1736323177, predicted_gas_cost: 1007000, chain_id: Some(SLChainId(17000)) }

[sachanayush47]

Update: Chain has been recovered.
Series of block-reverter command to be executed:

1. clear-failed-transaction
2. rollback-db
3. print-suggested-values (This is required for send-eth-transaction to get params value)
4. send-eth-transaction

Everything is working now.
The only concern that is left is this error is still unresolved: eth_tx_manager: Possible block reorgs: finalized nonce increase detected, but no tx receipt found for tx EthTx

[popzxc]

Hey! Sorry for the delay, and glad that you figured it out.

Regarding the eth sender issue -- such an error might happen if there are old entries in eth sender tables related to the blocks that were reverted. Probably what you can do is to connect to the database with psql and check if there are any entries in eth_txs table related to the blocks that were reverted.