[ZK Stack] - compressor in custom testnet fails with fflonk flag

[thecodingshrimp]

Team or Project

No response

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.82.0

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

2025-06-17T12:02:34.264969Z  INFO zksync_proof_fri_compressor::compressor: Started proof compression for L1 batch: L1BatchId
        (chain_id: 271, batch_number: 2)

thread 'tokio-runtime-worker' panicked at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/fflonk-cuda-0.154.2/src/primitives/mem.rs:80:13:
Couln't allocate host buffer 
stack backtrace:
   0:     0x5af6b4237ac2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h2137b6342f1d7514
   1:     0x5af6b42673a3 - core::fmt::write::hc09fda18d9ba7e13
   2:     0x5af6b4232c53 - std::io::Write::write_fmt::hc5178d2f7c502de3
   3:     0x5af6b4237912 - std::sys::backtrace::BacktraceLock::print::hd3aa20f1625542d3
   4:     0x5af6b4238d4c - std::panicking::default_hook::{{closure}}::h27a178e455c58e1c
   5:     0x5af6b4238b4f - std::panicking::default_hook::hf315b1d4878133c6
   6:     0x5af6b4239772 - std::panicking::rust_panic_with_hook::hab1c27c56ca0369e
   7:     0x5af6b42394fa - std::panicking::begin_panic_handler::{{closure}}::h22745f64f0ff8046
   8:     0x5af6b4237fc9 - std::sys::backtrace::__rust_end_short_backtrace::he285cf1e051c592f
   9:     0x5af6b423918d - __rustc[1b65734bf76d56e4]::rust_begin_unwind
  10:     0x5af6b4264300 - core::panicking::panic_fmt::h7c7d882b734bf2b1
  11:     0x5af6b307cba7 - fflonk_cuda::allocator::pinned::init_static_host_alloc::he0fe5dc76c9daf5c
  12:     0x5af6b2cc5510 - proof_compression::chain::run_proof_chain::h1f2b4bd09a2a7f85
  13:     0x5af6b2e50257 - tokio::runtime::task::raw::poll::h63ebab9477cc7b77
  14:     0x5af6b418a117 - tokio::runtime::blocking::pool::Inner::run::h6eb6a449ec1b3a31
  15:     0x5af6b418e470 - std::sys::backtrace::__rust_begin_short_backtrace::hb7a89879d8cf7050
  16:     0x5af6b418eca8 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hc2436d489bb03b63
  17:     0x5af6b423d74b - std::sys::pal::unix::thread::Thread::new::thread_start::h2bd0c059cc6cccbb
  18:     0x7240b149caa4 - <unknown>
  19:     0x7240b1529c3c - <unknown>
  20:                0x0 - <unknown>

Expected Behavior

This is a follow up for #1037. I was told to use fflonk for compressor and not plonk.
after executing the following commands, the compressor should generate a compressed proof and send it back to the server:
I have a 80GB A100 GPU and set max-allocation so that prover and compressor do not collide. I also tested killing the prover after it was done and only executing the compressor but it still fails with the same error.

Note: I am using zksync-era version 28.2.0.

$ zkstack ecosystem create --l1-network localhost --link-to-code /path/to/zksync-era --ecosystem-name eco-zksync --chain-name testchain --chain-id 271 --wallet-creation localhost --prover-mode gpu --l1-batch-commit-data-generator-mode validium --evm-emulator false --update-submodules false --start-containers true

$ zkstack ecosystem init --update-submodules false --validium-type no-da --deploy-erc2 true --deploy-ecosystem true --ecosystem-contracts-path /path/to/era/contracts --l1-rpc-url http://localhost:8545 --deploy-paymaster true

$ zkstack server --components=api,tree,eth,state_keeper,housekeeper,commitment_generator,da_dispatcher,proof_data_handler,vm_runner_protective_reads,vm_runner_bwip,external_proof_integration_api

$ zksync-cli bridge deposit --rpc http://localhost:3050 --l1-rpc http://localhost:8545 (just to have a transaction to prove)

$ zkstack prover init --setup-keys true --mode generate --setup-database true --use-default true --dont-drop false --bellman-cuda true --setup-compressor-key true

$ zkstack prover run --component=gateway --docker false

$ zkstack prover run --component=witness-generator --round=all-rounds --docker false

$ zkstack prover run --component=circuit-prover -l 5 -h 1 --docker false --max-allocation 32212254720

$ zkstack prover run --component=prover-job-monitor --docker false

$ RUST_BACKTRACE=full zkstack prover run --component=compressor --mode fflonk --docker false --max-allocation 48000000000

Current Behavior

the compressor fails to allocate memory on the gpu, even though the prover was able to access it just fine and use it for the proof.

Repository Link (if applicable)

https://github.com/matter-labs/zksync-era/tree/core-v28.2.0

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[JackHamer09]

Hey, thanks for report, I forwarded it to the team!

[Artemka374]

Hey, thanks for reaching out!
First of all, while circuit-prover's memory is flexible(you can decrease max allocation and it will still work), compressor's memory isn't. Both of them try to allocate full free memory, although for compressor's work it needs around ~20GB - so you limit prover's memory and allow compressor to get the remaining part. This part was mostly FYI.

On the other hand - are you sure that nothing uses the GPU memory when you stop prover and start compressor? Did you run nvidia-smi to check that?

[thecodingshrimp]

@JackHamer09 told me to use fflonk, maybe you know what the issue could be?

[EmilLuta]

That all makes sense. FFLONK is what needs to be used, as it's both faster & cheaper to verify. I'll defer this further to our GPU team. The problem seems GPU related.

[robik75]

This is not a GPU issue, the error is caused by the failure to allocate pinned host memory:
Couln't allocate host buffer 7247757312 1
11: 0x6408d8283ba7 - fflonk_cuda::allocator::pinned::init_static_host_alloc::he0fe5dc76c9daf5c
https://github.com/matter-labs/zksync-crypto-gpu/blob/main/crates/fflonk/src/allocator/pinned.rs#L32

I suppose the number 7247757312 means 7247757312 bytes, that would be a 6.75 GB chunk, that would be a lot for a single pinned allocation, certainly something that can fail.

Not sure who the right person is to look into the details and maybe come up with a better way. @Artemka374 or @AlexOnTheStorm ?

[Romsters]

Hi @thecodingshrimp , we were able to reproduce the issue and the team is looking into it, we will provide an update once we know more.

[antonbaliasnikov]

Hey @thecodingshrimp ,
Just a quick update that we are still working on it. Hopefully, we will have a fix soon.