[ZK Stack] - Compressor giving up after 5 attempts

[eierina]

Team or Project

No response

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.82.0-nightly (28a58f2fa 2024-07-31)

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

Using core-v25.4.0, I set up a custom ZKsync deployment to mainnet on the following hardware:

• GPU: 1x A100 (40 GB PCIe)
• CPU: 30 vCPUs
• RAM: 200 GiB

Almost everything works fine, except the compressor does not finalize, as shown in the status output below (more than 48 hours passed, and the prover took about 20 minutes per batch to complete):

prover_cli status batch -n 1 2 3 4
== Batch 1 Status ==

-- Aggregation Round 0 --
Basic Witness Generator: Successful ✅
> Prover Jobs: Successful ✅

-- Aggregation Round 1 --
Leaf Witness Generator: Successful ✅
> Prover Jobs: Successful ✅

-- Aggregation Round 2 --
Node Witness Generator: Successful ✅
> Prover Jobs: Successful ✅

-- Aggregation Round 3 --
Recursion Tip: Successful ✅

-- Aggregation Round 4 --
Scheduler: Successful ✅

-- Proof Compression --
Compressor: In Progress ⌛️
...
[Same status repeated for batches 2, 3, and 4]

I suspect the prover and the compressor are unable to share the GPU. I'm asking for direction because I see multiple, widely different hardware requirements for GPU and RAM across documentation.

Looking at the compressor logs, I observed:

1. The compressor launches with GPU support enabled:

cargo run --features=gpu --release --bin zksync_proof_fri_compressor

2. Each batch fails consistently with this pattern:

Start allocating new prover
transforming CRS
[/home/ubuntu/.cargo/registry/src/index.crates.io-6f17d22bba15001f/zksync-gpu-prover-0.152.9/src/memory_manager/mod.rs:37:9] device_ids = [0]
thread 'tokio-runtime-worker' panicked at crates/bin/proof_fri_compressor/src/compressor.rs:76:92: called `Result::unwrap()` on an `Err` value: Gpu(SetDeviceErr(2))

3. After 5 attempts, it gives up:

2025-03-24T10:18:47.102667Z  INFO zksync_proof_fri_compressor::compressor: Started proof compression for L1 batch: L1BatchNumber(4)
2025-03-24T10:18:47.104911Z ERROR zksync_queued_job_processor: Max attempts (5) reached for ProofCompressor job L1BatchNumber(4)

The error code SetDeviceErr(2) in CUDA indicates "invalid device ordinal," suggesting the system is trying to use GPU device 0, but it's either unavailable or improperly configured. Possible reasons include:

• The GPU lacks sufficient memory to handle both prover and compressor simultaneously (where can I find definitive memory requirements for each?).
• The GPU cannot be concurrently accessed by both processes regardless of memory availability.

Can anyone provide clarification on these issues?
Additionally, how can I instruct the prover to retry compression for batches it previously abandoned?

Expected Behavior

Prover and compressor works together on the same GPU instance.

Current Behavior

Prover works fine, compressor fails (probably because it starts last).

Repository Link (if applicable)

No response

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[EmilLuta]

Hey @eierina. Thanks for reaching out. Let me try to clarify, I'll go top down and answer each question:

1. Both processes can share the GPU, but they have to have GPU available. Prover at start time will allocate ALL VRAM available. You can always limit it with --max-allocation (comes from here).
2. About the exact error, I'm not sure, as it comes from CUDA. It's very likely that your assessment is correct, but if not, please circle back to this issue and I'll assist further.
3. About resource usage, circuit prover can work with as low as 8 GB of VRAM (albeit, it'll be super slow) & works best at around 20GB. Compressor works with 20GB of VRAM (IIRC), but works best with 24GB of VRAM.
4. About restarting the jobs, I'd suggest connecting to prover database and run something like:

UPDATE proof_compression_jobs_fri SET status='queued', attempts=0 WHERE l1_batch_number IN (<BATCHES>, <YOU>, <WANT>, <TO>, <RESTART>);

Hope this helps and adds more clarity to the setup. Keep us in the loop if you still struggle!

[eierina]

Hey @EmilLuta, thanks for the detailed clarification — your suggestions solved my issue!