[General] <"Couln't allocate host buffer" when compressor component of prover is trying to process L1Batch>

[sugarbody]

Team or Project

No response

ZK chain

Revo

Environment

Testnet

L2 block number

No response

Provide a brief description of the functionality you're trying to implement and the issue you are running into.

When running compressor in fflonk mode, I have "Couln't allocate host buffer" many times. How to solve this problem?

2025-03-12T17:58:46.309783Z INFO zksync_db_connection::connection_pool: Created DB pool with parameters ConnectionPoolBuilder { database_url: "postgres://:@45.33.90.86:31028/prover_devd", max_size: 1, acquire_timeout: 30s, statement_timeout: None, db: "zksync_prover_dal::Prover" }
2025-03-12T17:58:46.567179Z INFO download_initial_setup_keys_if_not_present: zksync_proof_fri_compressor::initial_setup_keys: Initial setup already present at "keys/setup/setup_compact.key"
2025-03-12T17:58:46.567273Z INFO zksync_proof_fri_compressor: Starting proof compressor
2025-03-12T17:58:46.670335Z ERROR vise_exporter::exporter: Error submitting metrics to Prometheus push gateway err=error trying to connect: tcp connect error: Connection refused (os error 111) endpoint=http://127.0.0.1:9091/
2025-03-12T17:58:46.743277Z INFO zksync_proof_fri_compressor::compressor: Started proof compression for L1 batch: L1BatchNumber(1)
thread 'tokio-runtime-worker' panicked at /home/revolution/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fflonk-cuda-0.153.1/src/primitives/mem.rs:80:13:
Couln't allocate host buffer
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
2025-03-12T17:58:48.948811Z ERROR zksync_queued_job_processor: Error occurred while processing ProofCompressor job L1BatchNumber(1): "Couln't allocate host buffer"

Repo Link (Optional)

No response

Additional Details

No response

[Artemka374]

Hi, I have a few questions:

• Do you encounter the issue only with FFLONK or PLONK/default mode is also not working?
• How much GPU memory is there on machine are you using?(The requirement for compressor is ~20GB, so something like L4 GPU should succeed)
• Did you stop the prover when running compressor? By default, prover allocates all free memory(although can be limited), so if you did not limit the allocation or the machine simply doesn't have enough memory for both, it won't work.

[sugarbody]

I compiled this code in 64 bit. nvcc -m64 test.cu -o test

revolution@revolution1:~/revo_dev/revo_dev$ ./test
Free: 23746.6 MB, Total: 24202.1 MB
CUDA Memory Allocation Error: invalid argument

However, I have still same error. The system is 64 bit. What is the problem?

$ file test
test: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=8473e16b67210a33c68287d2bd0d4bef310f3983, for GNU/Linux 3.2.0, not stripped

#include <cuda_runtime.h>
#include <iostream>

int main() {
    void* ptr1;
    size_t size = 23000000000; // Try 2.3GB allocation
    size_t free_mem, total_mem;
    cudaMemGetInfo(&free_mem, &total_mem);
    std::cout << "Free: " << free_mem / (1024.0 * 1024.0) << " MB, Total: " << total_mem / (1024.0 * 1024.0) << " MB\n";
    
    cudaError_t err = cudaMallocHost(&ptr1, size);

    if (err != cudaSuccess) {
        std::cerr << "CUDA Memory Allocation Error: " << cudaGetErrorString(err) << std::endl;
        return 1;
    }

    std::cout << "Successfully allocated " << size << " bytes in pinned memory!" << std::endl;
    cudaFreeHost(ptr1);
    return 0;
}

[vasyl-ivanchuk]

Since a separate script fails as well, it seems that the problem above is not related to ZKsync and is specific to CPU memory allocation. I'd suggest fixing the CPU memory allocation first and then trying again to see if the compressor works. We will be happy to assist if the allocation problem is resolved but the compressor still doesn't work.

[sugarbody]

Thank you for your help. Do you think where I can get compressor setup keys including fflonk_setup_snark_data.bin, setup_compression_1_data.bin and so on?

[vasyl-ivanchuk]

As for the memory allocation, feel free to check a workaround in this GH discussion.

Do you think where I can get compressor setup keys including fflonk_setup_snark_data.bin, setup_compression_1_data.bin and so on?

There is a dedicated GH discussion for this. Please check the provided answer here.