[ZK Stack] - <ways to deploy multi-provers in different machines>

[qiwenzhou]

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.74.0

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

What is the proper way to deploy multi-prover for my server. I ran a server and a prover on one machine A , worked fine.
After that, I tried to add one more prover by: export DATABASE_PROVER_URL=postgres://postgres:notsecurepassword@A_IP/prover_local on machine B and ran zk f cargo run --features "gpu" --release --bin zksync_prover_fri.
At the beginning, the two prover worked fine.Both prover could get jobs. After certain time, there always be a problem to come up.

It seems like zksync_witness_generator failed to process "aggregation type" jobs due to lack of files. More than that, zksync_witness_generator stopped to produce prove jobs for new batches.

Expected Behavior

Run multi-provers on multi-machines with one prover on one machine . How to achieve that?

Current Behavior

will fail.

Repository Link (if applicable)

No response

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[bxpana]

Not sure if we have documentation about this quite yet, but will check on this for you.

[qiwenzhou]

Thank you!

[qiwenzhou]

UPDATE:
I turned down postgres in machine B,pointing prover database to machine A and prover on machine B failed to verify proofs after some successful jobs.

[qiwenzhou]

FIXED：It turned out that provers in two machine store the data in defferernt positions. After making prover in machine B to fetch and store data in machine A, it worked.