[ZK Stack] - Server fails to send batch proof to l1 in custom testnet setup

[thecodingshrimp]

Team or Project

No response

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.82.0

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

when trying to setup a testnet with custom l1, l2, and prover the network fails to send the compressed proof to the l1 with the following error:

2025-06-19T13:10:37.236415Z  INFO EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Checking tx id: 3, operator_nonce: OperatorNonce { finalized: Nonce(2), latest: Nonce(2) }, tx nonce: 1                                                                                                                                  
2025-06-19T13:10:37.236433Z  INFO EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Updating status of tx 3 of type PublishProofBlocksOnchain with nonc
e 1                                                                                                                                                                   
2025-06-19T13:10:37.237301Z ERROR EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Possible block reorgs: finalized nonce increase detected, but no tx
 receipt found for tx EthTx { id: 3, nonce: Nonce(1), contract_address: 0x691de3abe6480facb799b3b0a38ba0c6b72ccfab, tx_type: PublishProofOnchain, created_at_timestamp
: 1750338247, predicted_gas_cost: Some(800000), chain_id: Some(SLChainId(9)) }                                                                                        
2025-06-19T13:10:38.247583Z  INFO EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Sending 4 NonBlob new transactions                                 
2025-06-19T13:10:38.247819Z  INFO EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Sending NonBlob tx 3 (nonce 1) at block 16205 with base_fee_per_gas
 10, priority_fee_per_gas 1000000000, blob_fee_per_gas None,max_gas_per_pubdata_price None,gas_limit 800000,                                                          
2025-06-19T13:10:38.249712Z  WARN EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Error Sending NonBlob tx 3 (nonce 1) at block 16205 with base_fee_p
er_gas 10, priority_fee_per_gas 1000000000, blob_fee_per_gas None,gas_limit 800000,                                                                                   
                error Ethereum gateway error: JSON-RPC request send_raw_transaction() failed: ErrorObject { code: ServerError(-32001), message: "Nonce too low", data:
 None }

Expected Behavior

after executing

$ zkstack ecosystem create --l1-network localhost --link-to-code /path/to/zksync-era --ecosystem-name eco-zksync --chain-name testchain --chain-id 271 --wallet-creation localhost --prover-mode gpu --l1-batch-commit-data-generator-mode validium --evm-emulator false --update-submodules false --start-containers true

$ zkstack ecosystem init --update-submodules false --validium-type no-da --deploy-erc20 true --deploy-ecosystem true --ecosystem-contracts-path /path/to/era/contracts --l1-rpc-url http://localhost:8545 --deploy-paymaster true

$ zkstack server --components=api,tree,eth,state_keeper,housekeeper,commitment_generator,da_dispatcher,proof_data_handler,vm_runner_protective_reads,vm_runner_bwip,external_proof_integration_api

$ zksync-cli bridge deposit --rpc http://localhost:3050 --l1-rpc http://localhost:8545 (just to have a transaction to prove)

$ zkstack prover init --setup-keys true --mode generate --setup-database true --use-default true --dont-drop false --bellman-cuda true --setup-compressor-key true

$ zkstack prover run --component=gateway --docker false

$ zkstack prover run --component=witness-generator --round=all-rounds --docker false

$ zkstack prover run --component=circuit-prover -l 5 -h 1 --docker false --max-allocation 32212254720

$ zkstack prover run --component=prover-job-monitor --docker false

$ prover run --component=compressor --mode plonk --docker false --max-allocation 48000000000

And starting the explorer.
It should begin to validate the transaction batches and send batch proofs to the l1. It should validate each new block that is generated on l2.

Current Behavior

It fails to send the second batch proof to l1. The server fails with the message:

2025-06-19T13:10:37.236415Z  INFO EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Checking tx id: 3, operator_nonce: OperatorNonce { finalized: Nonce(2), latest: Nonce(2) }, tx nonce: 1                                                                                                                                  
2025-06-19T13:10:37.236433Z  INFO EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Updating status of tx 3 of type PublishProofBlocksOnchain with nonc
e 1                                                                                                                                                                   
2025-06-19T13:10:37.237301Z ERROR EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Possible block reorgs: finalized nonce increase detected, but no tx
 receipt found for tx EthTx { id: 3, nonce: Nonce(1), contract_address: 0x691de3abe6480facb799b3b0a38ba0c6b72ccfab, tx_type: PublishProofOnchain, created_at_timestamp
: 1750338247, predicted_gas_cost: Some(800000), chain_id: Some(SLChainId(9)) }                                                                                        
2025-06-19T13:10:38.247583Z  INFO EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Sending 4 NonBlob new transactions                                 
2025-06-19T13:10:38.247819Z  INFO EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Sending NonBlob tx 3 (nonce 1) at block 16205 with base_fee_per_gas
 10, priority_fee_per_gas 1000000000, blob_fee_per_gas None,max_gas_per_pubdata_price None,gas_limit 800000,                                                          
2025-06-19T13:10:38.249712Z  WARN EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Error Sending NonBlob tx 3 (nonce 1) at block 16205 with base_fee_p
er_gas 10, priority_fee_per_gas 1000000000, blob_fee_per_gas None,gas_limit 800000,                                                                                   
                error Ethereum gateway error: JSON-RPC request send_raw_transaction() failed: ErrorObject { code: ServerError(-32001), message: "Nonce too low", data:
 None }

From the explorer it seems like the initial deployments are separated into several different proof batches, but the whole l2 ecosystem jsut recognizes one of them.

Repository Link (if applicable)

No response

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[JackHamer09]

Hey, I forwarded your report to the team and we'll get back to you!

[JackHamer09]

@thecodingshrimp someone in the team is suggesting that nonce too low error shouldn't be critical, unless it's consistent.
The reason might be something like using non-unique operator address (basically someone/something else might be sending transaction from same account resulting in nonce missmatch).

[thecodingshrimp]

Thanks for the clarification! Let me double check with my setup and get back to you over the weekend if that helped 💪🏽

[thecodingshrimp]

@JackHamer09 your suggestion works! I used the operator pk to send funds to l2, but I guess the operator should only have tx that are relevant to the zksync contracts on l1?

When using an unrelated wallet the transaction goes through :)