[ZK Stack] - Multi-prover generates proof problems

[Marius-BC-Dev]

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.73.0-nightly

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

1. Can multiple prover clients be used to generate proofs per circuit?For a circuit of type 0, when I am generating proofs with two prover clients, one prover exits abnormally after accepting the data and the other one works fine.
2. What machine configuration does prover need to generate proofs quickly? I currently take 40-50 minutes to generate proofs .
   Machine Configuration:
   CPU: 48C
   GPU: 3090
   GPU RAM: 20G

Expected Behavior

Jobs for a class of circuits can be produced proof with multiple prover.

Current Behavior

Jobs for a class of circuits can only be generated proof with one prover.

Repository Link (if applicable)

No response

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[robik75]

Hello,
running multiple prover-instances in parallel on the same GPU is not supported.
When a prover instance starts, it allocates as much GPU RAM as it can so if you try to run a second one in parallel, it will fail because all the memory was already taken by the first one.

[Marius-BC-Dev]

I am using multiple GPU servers to run prover, different circuits are fine.

[Marius-BC-Dev]

Thank you! Do you have any suggestions? Or relevant configuration documentation?

[Marius-BC-Dev]

Thank you! Do you have a suggested configuration? Or is there a relevant config file I can refer to?

[Marius-BC-Dev]

Give me an example? I can't get it to work on my own.