[ZK Stack] - System has failed L1 transaction

[sachanayush47]

Team or Project

No response

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.80.1

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

We have self deployed a zksync hyperchain running from last 10 days and then suddenly server stops with this error ERROR zksync_node_framework::service: Task circuit_breaker_checker failed: Circuit breaker error. Reason: System has failed L1 transaction.
So when we checked the postgres DB, there was failed transaction so we decided to delete it and ran the server again. Server failed to start with this error

Eth tx failed EthTx { id: 6781, nonce: Nonce(11530), contra
ct_address: 0xc3ef0f406b22a6d514d63f229af1bb31530715b2, tx_type: Commit, created_at_timestamp: 1730974553, predicted_gas_cost: 294528, chain_id: Some(SLChainI
d(17000)) }, TransactionReceipt { transaction_hash: 0x89ed61dfeb1ddc7a9f1cca34fb5a12310bb9447adda77592009c90e58c1401c9, transaction_index: 2, block_hash: Some
(0x6fd0dc55500ec684e8330e7934c5d9488f33fa8b1ebfa26e186b861d68cfac75), block_number: Some(2690734), from: 0xb2c2c467e0ef64070b56eec0ae499174912de565, to: Some(
0xc3ef0f406b22a6d514d63f229af1bb31530715b2), cumulative_gas_used: 225488, gas_used: Some(90504), contract_address: None, logs: [], status: Some(0), root: None
, logs_bloom: 0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000, transaction_type: Some(2), effective_gas_price: Some(2000000008) }, failure reason Some(FailureInfo { 
revert_code: 3, revert_reason: "i", gas_used: Some(90504), gas_limit: 15000000 })

Then again we tried to run server (3rd attempt) got again ERROR zksync_node_framework::service: Task circuit_breaker_checker failed: Circuit breaker error. Reason: System has failed L1 transaction

On the block explorer, there are 2 transactions which are stuck at Sending step.

Expected Behavior

Server should be running normally and submitting transactions to L1 holesky

Current Behavior

Server failed to start

Repository Link (if applicable)

No response

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[alhenaai[bot]]

It appears that your ZKsync hyperchain is experiencing issues with L1 transactions, which is causing the server to stop and fail to restart. The error messages indicate a problem with the transaction processing on L1, specifically related to a failed transaction that was attempted to be deleted from the Postgres database.

Here are some steps you can consider to troubleshoot and potentially resolve the issue:

1. Check the Transaction Details: Ensure that the transaction details such as nonce, gas limit, and gas price are correctly set. The error logs show a transaction with a specific nonce and gas settings that might not be compatible or sufficient for the transaction to be processed successfully on L1.

2. Review the Circuit Breaker Settings: The circuit_breaker_checker task failure suggests that there might be safety mechanisms in place that stop the server when certain conditions are not met. Review these settings to ensure they are not too restrictive or incorrectly configured.

3. Transaction Recovery: Instead of deleting the failed transaction from the database, consider using a recovery or rollback mechanism to handle failed transactions more gracefully. This might involve re-submitting the transaction with adjusted parameters or rolling back to a previous stable state.

4. Logs and Monitoring: Increase the logging level to capture more detailed information about the transactions and system behavior. This can provide more insights into why the transactions are failing.

5. Consult Documentation and Community: Review the ZKsync documentation and community discussions for similar issues and recommended practices. Engaging with the community can also provide insights and potential solutions from other users who might have faced similar problems.

For more detailed assistance and to discuss this issue with other developers, you can visit the ZKsync developers discussions on Github at ZKsync Developers Discussions.

If the problem persists, consider reaching out to a ZKsync support specialist or a blockchain consultant who can provide more personalized assistance based on your specific setup and requirements.

Your feedback helps us serve you better. If this fully addresses your question, please give it a thumbs up. If it doesn't, please give it a thumbs down. This will inform the community and team so we can improve our support.

[dutterbutter]

Have you tried looking at using: block_reverter -- clear-failed-transactions. It seems that clears failed L1 transactions.

[sachanayush47]

Have you tried looking at using: block_reverter -- clear-failed-transactions. It seems that clears failed L1 transactions.

Thanks. I was not aware of this utility, so trying to recover the chain somehow data got corrupt. So we had new deployment. Will try this in future if required.

[sachanayush47]

Have you tried looking at using: block_reverter -- clear-failed-transactions. It seems that clears failed L1 transactions.

Hi, we again ran into the same issue and tried using the method you have suggested. We were getting this error.

root@validium-testnet:/home/zksync/zksync-era/target/debug# ./block_reverter clear-failed-transactions
Error: ProtectiveReadsWriterConfig::from_env()

Caused by:
    0: Cannot load config <vm_runner.protective_reads>
    1: missing value for field window_size

[MexicanAce]

Hi @sachanayush47 , you need to include the various path parameters when using block_reverter. Example:

 ./target/release/block_reverter clear-failed-transactions \
    --config-path $ZK_CHAIN_CONFIG_ROOT/configs/general.yaml \
    --contracts-config-path $ZK_CHAIN_CONFIG_ROOT/configs/contracts.yaml  \
    --secrets-path $ZK_CHAIN_CONFIG_ROOT/configs/secrets.yaml  \
    --wallets-path $ZK_CHAIN_CONFIG_ROOT/configs/wallets.yaml  \
    --genesis-path $ZK_CHAIN_CONFIG_ROOT/configs/genesis.yaml

[sachanayush47]

Thanks this worked but the problem still kinda persists. It clears the failed transaction but it comes within a minute causing the L2 server to crash.

[abbasfvi]

Hi @sachanayush47, were you able to solve L2 server crash issue?

[sachanayush47]

Yes, i was able to solve

[abbasfvi]

can you please provide solution, I'm facing the same issue.