[ZK Stack] - Starting a Prover When Deploying a Hyperchain

[nhtransis]

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.75.0

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

I'm following a tutorial to deploy a hyperchain using the Sepolia testnet and have run into a challenge. Although I have successfully enabled the prover according to the steps provided, I'm unable to observe the expected rollup transactions in the testnet explorer. Can someone guide me on what might be preventing the rollup transactions from appearing after the prover is enabled

Expected Behavior

Prover will generate the proof, then that will be verified on Sepolia

Current Behavior

Prover does not work

Repository Link (if applicable)

No response

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[zk-Lumi]

Hi @nhtransis,

Could you please send the log for the prover so we can try to observe?

[nhtransis]

Hello @zk-Lumi ,

I am trying to run Prover with GPU. But I get the issue

2024-03-29T06:20:11.154161Z INFO zksync_prover_fri: Starting FRI proof generation for group: 100 with circuits: []
allocated 262144 bytes on device
allocated 15527313408 bytes on device
allocated 65536 bytes on host
allocated 2147483648 bytes on host
thread 'main' panicked at prover_fri/src/gpu_prover_job_processor.rs:89:22:
failed initializing gpu prover context: ErrorUnsupportedLimit
stack backtrace:
0: 0x5d1e0e9c06ec - std::backtrace_rs::backtrace::libunwind::trace::h68b3f03db065435f
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
1: 0x5d1e0e9c06ec - std::backtrace_rs::backtrace::trace_unsynchronized::hcc29d957a519c467
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x5d1e0e9c06ec - std::sys_common::backtrace::_print_fmt::h51e5567c7446b80e
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/sys_common/backtrace.rs:67:5
3: 0x5d1e0e9c06ec - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h54890bb4d0b3c330
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/sys_common/backtrace.rs:44:22
4: 0x5d1e0e9edf6c - core::fmt::rt::Argument::fmt::heae7cd8ccc900aeb
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/core/src/fmt/rt.rs:138:9
5: 0x5d1e0e9edf6c - core::fmt::write::hc761729f1163b685
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/core/src/fmt/mod.rs:1094:21
6: 0x5d1e0e9bcbce - std::io::Write::write_fmt::hd8de790e313b58ee
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/io/mod.rs:1714:15
7: 0x5d1e0e9c04d4 - std::sys_common::backtrace::_print::h1b2adaff5c947b3b
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/sys_common/backtrace.rs:47:5
8: 0x5d1e0e9c04d4 - std::sys_common::backtrace::print::hf6a9b81f93e62890
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/sys_common/backtrace.rs:34:9
9: 0x5d1e0e9c230a - std::panicking::panic_hook_with_disk_dump::{{closure}}::h663629ab1a73f13e
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/panicking.rs:278:22
10: 0x5d1e0e9c1ff7 - std::panicking::panic_hook_with_disk_dump::h7dc7628c0cba33b1
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/panicking.rs:312:9
11: 0x5d1e0e9c290b - std::panicking::default_hook::h4cb3ec769d1db09c
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/panicking.rs:239:5
12: 0x5d1e0e9c290b - std::panicking::rust_panic_with_hook::hdd1521f6ef2543b7
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/panicking.rs:729:13
13: 0x5d1e0e9c2807 - std::panicking::begin_panic_handler::{{closure}}::h7c4ede115dee30b0
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/panicking.rs:621:13
14: 0x5d1e0e9c0c16 - std::sys_common::backtrace::__rust_end_short_backtrace::habd7cadfbd319a50
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/sys_common/backtrace.rs:170:18
15: 0x5d1e0e9c2552 - rust_begin_unwind
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/panicking.rs:617:5
16: 0x5d1e0dde8ae3 - core::panicking::panic_fmt::hb85e049a4301ecd3
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/core/src/panicking.rs:67:14
17: 0x5d1e0dde8eea - core::result::unwrap_failed::h2a76005869fd26f6
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/core/src/result.rs:1652:5
18: 0x5d1e0df8f7f6 - zksync_prover_fri::gpu_prover_job_processor::gpu_prover::Prover::new::h9895d333a188e3f0
19: 0x5d1e0df0cb3a - zksync_prover_fri::main::{{closure}}::h89655e97b96cb68f
20: 0x5d1e0df06ef0 - tokio::runtime::park::CachedParkThread::block_on::h99ef7d45f4902a68
21: 0x5d1e0e03d0d9 - zksync_prover_fri::main::hc05473bec95ea4d9
22: 0x5d1e0dfcf4e3 - std::sys_common::backtrace::__rust_begin_short_backtrace::h73e4db1a5dc94c18
23: 0x5d1e0deb178d - std::rt::lang_start::{{closure}}::hd99a0f673477333b
24: 0x5d1e0e9b4d7b - core::ops::function::impls::<impl core::ops::function::FnOnce for &F>::call_once::h15d49bc2cec08bca
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/core/src/ops/function.rs:284:13
25: 0x5d1e0e9b4d7b - std::panicking::try::do_call::h7e6046cce1796a4c
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/panicking.rs:524:40
26: 0x5d1e0e9b4d7b - std::panicking::try::h7a03f577141ecafa
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/panicking.rs:488:19
27: 0x5d1e0e9b4d7b - std::panic::catch_unwind::hf8b5488fd0ea6e9f
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/panic.rs:142:14
28: 0x5d1e0e9b4d7b - std::rt::lang_start_internal::{{closure}}::h6830acc6e473b9c7
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/rt.rs:148:48
29: 0x5d1e0e9b4d7b - std::panicking::try::do_call::h4a5f68ee94ccb8f2
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/panicking.rs:524:40
30: 0x5d1e0e9b4d7b - std::panicking::try::hf5a7a97c92e8e8b4
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/panicking.rs:488:19
31: 0x5d1e0e9b4d7b - std::panic::catch_unwind::h0d98fd51ab78927d
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/panic.rs:142:14
32: 0x5d1e0e9b4d7b - std::rt::lang_start_internal::he20da38a2539432c
at /rustc/5c6a7e71cd66705c31c9af94077901a220f0870c/library/std/src/rt.rs:148:20
33: 0x5d1e0e03e145 - main
34: 0x70d8a2829d90 -
35: 0x70d8a2829e40 - __libc_start_main
36: 0x5d1e0dde95e5 - _start
37: 0x0 -

Here is the info

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.07 Driver Version: 535.161.07 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 43C P8 10W / 70W | 2MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

Thanks for supporting.

[robik75]

This is a bug in the GPU prover, concretely in the shivini dependency which lives in a separate repository.
PR for the fix: matter-labs/era-shivini#42
Once the PR is merged, the dependency in the zksync-era repository needs to be updated to use this fixed version.

[robik75]

zksync-era has been updated: matter-labs/zksync-era#1588
This should resolve the issue.
Please give it a try.

[nhtransis]

Thanks. I passed this issue already.