[ZK Stack] - Transactions stuck at "SENDING" status

[Teamvalid]

Team or Project

Validium

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.82.0

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

Our chain is using holesky network as a L1, in the recent update holesky had some issues due to fix transaction on our chain were not been not sent to L1. Transactions were stuck on "sending" status for couple of days. So we decided to rollback to last executed stage.

Upon using block reverters print-suggested-values, recommneded values are

Suggested values for reversion: SuggestedRevertValues {
    last_executed_l1_batch_number: L1BatchNumber(
        26678,
    ),
    nonce: 61636,
    priority_fee: 1000000000,
}

But last executed batch is 26685

We used these 2 commands clear-failed-transactions and rollback-db to fix and 26685 for --l1-batch-number flag. Chain state rolled back but it didnt fixed the problem. Block reverter has removed the transactions but the problem still persists. New transactions are still stuck at "sending" status.

This is our blob operator, it has some failed transactions https://holesky.etherscan.io/address/0xb2c2c467e0ef64070b56eec0ae499174912de565

Please help to fix it.

Expected Behavior

Transactions should settle on L1

Current Behavior

Transactions are stuck on "sending" status.

Repository Link (if applicable)

No response

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[cytadela8]

Hi, we have a document about batch reversion here though it seems to me you were using it correctly. Your situation is very tricky to solve due to the reorg on Holesky. The recently failing L1 transactions from blob operator seem to commit some state transitions with mismatching state hashes. Please:

• check the nonces in the eth_txs table. It may be possible that the node has or is generating transactions with too high or overlapping nonces. Please compare what you see there against blockchain.
• check the l1_bacthes table for the batches from 26678 to 26685. (Please share the rows if you can). Please confirm if the batches have the prove, commit and execute transaction ids there. Explorer might have got desynced with the node.
• run print-suggested-values with RUST_LOG=info you will get more information then.
• provide server logs. It might shed light on what transactions are generated and whats wrong with them.

[Teamvalid]

QuickNode rpc seems to work. The batch 26686 seems to have some problem. Tx has been sent to the L1 but validation and execution step is still pending.

I have checked in DB, batch 26686 has prove and commit id.

[Teamvalid]

Mar 13 16:29:49 validium-testnet start.sh[118101]: 2025-03-13T16:29:49.659157Z  WARN EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Error Sending NonBlob tx 64140 (nonce 61666) at block 3484439 with base_fee_per_gas 130666559, priority_fee_per_gas 2539765277962, blob_fee_per_gas None,error Ethereum gateway error: JSON-RPC request send_raw_transaction() failed: ErrorObject { code: ServerError(-32000), message: "insufficient funds for gas * price + value: balance 37644730108383264017, tx cost 38098439167815000000, overshot 453709059431735983", data: None }

I'm also getting this error. The more ETH I transfer to the operator address, the higher the transaction cost becomes

[ly0va]

Hi @Teamvalid. It looks like your server database state matches the L1 blockchain state (block 26686 was executed on L1), but does not match your explorer UI -- which suggests that your explorer DB is out of sync with your server database -- can you check that?

Since the issue occured because of a reorg on your settlement layer (holesky), there might have been issues with registering the settlement transactions your operator sent, by the EthWatcher service. Which would affect the transactions being suggested to be sent by your operator.

Can you please confirm that the settlement transactions recorded in your database match the ones sent on L1 by your operator (including their status)?

[popzxc]

@Teamvalid have you been able to resolve the issue?

[popzxc]

@Teamvalid I'll close this as stale; if you will need further assistance -- feel free to start a new discussion.