[ZK Stack] - Eth sender is behind - block commited is smaller than most recent state keeper batch

[wjw12]

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.74.0-nightly (5c6a7e71c 2023-08-20)

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

I followed the prover_fri guide trying to enable prover by GPU. I've set dummy_verifier to false. All the server, gateway, witness generator, prover, witness vector generator run successfully without a problem.
However, zk status prover never shows a block proving is complete.

My hardware: NVIDIA A4000
Driver: NVIDIA-SMI 545.23.08 Driver Version: 545.23.08 CUDA Version: 12.3

Expected Behavior

I expect to see at least 1 block verified from zk status prover result

Current Behavior

zk status prover result

==== FRI Prover status ====
State keeper: First batch: 0, recent batch: 461
L1 state: block verified: 0, block committed: 460
Eth sender is behind - block commited 460 is smaller than most recent state keeper batch 461.

Repository Link (if applicable)

No response

Additional Details

zksync_proof_fri_compressor occasionally prints ERROR vise_exporter::exporter: Error submitting metrics to Prometheus push gateway err=error trying to connect: tcp connect error: Connection refused (os error 111) endpoint=http://127.0.0.1:9091/ but I don't think this is important.

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[shahar4]

Thanks for reaching out with this! Eth Sender being behind isn't necessarily an issue. Your proof system night still be chugging along. Some questions:

1. When did you start the system? How long has it been running?
2. Have you played with batch sealing time or at all with how frequently batches are created, or did you leave the defaults?
3. Does the server send batches (commit blocks) as soon as they are created, or does the lag ever exceed one batch?

[shahar4]

@wjw12 if you merge main, run zk, and then run the zk status prover command again, you will see more details about the status of the prover