lotus v1.5.0 pc2 computation failure

[yann1989]

2021-03-03T15:41:27.081 INFO filcrypto::proofs::api > seal_pre_commit_phase2: start
2021-03-03T15:41:27.081 INFO filecoin_proofs::api > validate_cache_for_precommit_phase2:start
2021-03-03T15:41:27.081 INFO filecoin_proofs::api > validate_cache_for_precommit_phase2:finish
2021-03-03T15:41:27.081 INFO filecoin_proofs::api::seal > seal_pre_commit_phase2:start
2021-03-03T15:41:27.081 INFO storage_proofs_porep::stacked::vanilla::proof > replicate_phase2
2021-03-03T15:41:27.081 INFO storage_proofs_porep::stacked::vanilla::proof > generating tree c using the GPU
2021-03-03T15:41:27.081 INFO storage_proofs_porep::stacked::vanilla::proof > Building column hashes
2021-03-03T15:41:27.081 INFO neptune::triton::cl > getting default futhark context
2021-03-03T15:41:27.081 INFO neptune::triton::cl > getting context for ~Index(0)
thread '' panicked at 'failed to create ColumnTreeBuilder: ClError(BusIdNotAvailable)', /root/.cargo/registry/src/mirrors.sjtug.sjtu.edu.cn-7a04d2510079875b/storage-proofs-porep-5.4.0/src/stacked/vanilla/proof.rs:516:23
stack backtrace:
0: 0x1f7ada0 - std::backtrace_rs::backtrace::libunwind::trace::he85dfb3ae4206056
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/../../backtrace/src/backtrace/libunwind.rs:96
1: 0x1f7ada0 - std::backtrace_rs::backtrace::trace_unsynchronized::h1ad28094d7b00c21
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/../../backtrace/src/backtrace/mod.rs:66
2: 0x1f7ada0 - std::sys_common::backtrace::_print_fmt::h901b54610713cd21
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/sys_common/backtrace.rs:79
3: 0x1f7ada0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hb0ad78ee1571f7e0
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/sys_common/backtrace.rs:58
4: 0x1fe955c - core::fmt::write::h1857a60b204f1b6a
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/core/src/fmt/mod.rs:1080
5: 0x1f6ca52 - std::io::Write::write_fmt::hf7b7d7b243f84a36
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/io/mod.rs:1516
6: 0x1f7fb7d - std::sys_common::backtrace::_print::hd093978a5287b8ff
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/sys_common/backtrace.rs:61
7: 0x1f7fb7d - std::sys_common::backtrace::print::h20f46787581d56d7
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/sys_common/backtrace.rs:48
8: 0x1f7fb7d - std::panicking::default_hook::{{closure}}::h486cbb4b82ffc357
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/panicking.rs:208
9: 0x1f7f828 - std::panicking::default_hook::h4190c9e3edd4d591
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/panicking.rs:227
10: 0x1f803a1 - std::panicking::rust_panic_with_hook::h72e78719cdda225c
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/panicking.rs:577
11: 0x1f7ff49 - std::panicking::begin_panic_handler::{{closure}}::h8bd07dbd34150a96
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/panicking.rs:484
12: 0x1f7b22c - std::sys_common::backtrace::__rust_end_short_backtrace::hdb6b3066ad29028a
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/sys_common/backtrace.rs:153
13: 0x1f7ff09 - rust_begin_unwind
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/panicking.rs:483
14: 0x1fe5881 - core::panicking::panic_fmt::hb15d6f55e8472f62
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/core/src/panicking.rs:85
15: 0x1fe54f3 - core::option::expect_none_failed::he492a18657d97593
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/core/src/option.rs:1234
16: 0x167e5f7 - <std::panic::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once::h57c264de9ebec2a6
17: 0x201b52e - std::panicking::try::h1274a0b2593158ac
18: 0x17f2a19 - <rayon_core::job::HeapJob as rayon_core::job::Job>::execute::h334622116c8971dc
19: 0x1f09401 - rayon_core::registry::WorkerThread::wait_until_cold::h37ed1352194a903d
20: 0x1f07642 - rayon_core::registry::ThreadBuilder::run::hd50e617b60902b7c
21: 0x1f0d6d3 - std::sys_common::backtrace::__rust_begin_short_backtrace::hd96649c4a7c9e5d9
22: 0x1f0ebf4 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hb79e3c0d98138dc0
23: 0x1f8ecda - <alloc::boxed::Box as core::ops::function::FnOnce>::call_once::h1080dfe0ef616bdf
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/alloc/src/boxed.rs:1042
24: 0x1f8ecda - <alloc::boxed::Box as core::ops::function::FnOnce>::call_once::hd2747e1f2d5cec32
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/alloc/src/boxed.rs:1042
25: 0x1f8ecda - std::sys::unix::thread::Thread::new::thread_start::hd0f336b4ef6808a7
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/sys/unix/thread.rs:87
26: 0x7ff1bfda06db - start_thread
27: 0x7ff1bf46971f - __clone
28: 0x0 -
thread '' panicked at 'failed to send columns: "SendError(..)"', /root/.cargo/registry/src/mirrors.sjtug.sjtu.edu.cn-7a04d2510079875b/storage-proofs-porep-5.4.0/src/stacked/vanilla/proof.rs:502:34
stack backtrace:
0: 0x1f7ada0 - std::backtrace_rs::backtrace::libunwind::trace::he85dfb3ae4206056
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/../../backtrace/src/backtrace/libunwind.rs:96
1: 0x1f7ada0 - std::backtrace_rs::backtrace::trace_unsynchronized::h1ad28094d7b00c21
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/../../backtrace/src/backtrace/mod.rs:66
2: 0x1f7ada0 - std::sys_common::backtrace::_print_fmt::h901b54610713cd21
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/sys_common/backtrace.rs:79
3: 0x1f7ada0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hb0ad78ee1571f7e0
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/sys_common/backtrace.rs:58
4: 0x1fe955c - core::fmt::write::h1857a60b204f1b6a
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/core/src/fmt/mod.rs:1080
5: 0x1f6ca52 - std::io::Write::write_fmt::hf7b7d7b243f84a36
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/io/mod.rs:1516
6: 0x1f7fb7d - std::sys_common::backtrace::_print::hd093978a5287b8ff
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/sys_common/backtrace.rs:61
7: 0x1f7fb7d - std::sys_common::backtrace::print::h20f46787581d56d7
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/sys_common/backtrace.rs:48
8: 0x1f7fb7d - std::panicking::default_hook::{{closure}}::h486cbb4b82ffc357
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/panicking.rs:208
9: 0x1f7f828 - std::panicking::default_hook::h4190c9e3edd4d591
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/panicking.rs:227
10: 0x1f803a1 - std::panicking::rust_panic_with_hook::h72e78719cdda225c
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/panicking.rs:577
11: 0x1f7ff49 - std::panicking::begin_panic_handler::{{closure}}::h8bd07dbd34150a96
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/panicking.rs:484
12: 0x1f7b22c - std::sys_common::backtrace::__rust_end_short_backtrace::hdb6b3066ad29028a
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/sys_common/backtrace.rs:153
13: 0x1f7ff09 - rust_begin_unwind
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/panicking.rs:483
14: 0x1fe5881 - core::panicking::panic_fmt::hb15d6f55e8472f62
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/core/src/panicking.rs:85
15: 0x1fe54f3 - core::option::expect_none_failed::he492a18657d97593
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/core/src/option.rs:1234
16: 0x1692fd7 - <std::panic::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once::ha1afc35a71a03729
17: 0x20283be - std::panicking::try::haaf7ef456514cf16
18: 0x17f4d80 - <rayon_core::job::HeapJob as rayon_core::job::Job>::execute::he6dcde350a8059c3
19: 0x1f09401 - rayon_core::registry::WorkerThread::wait_until_cold::h37ed1352194a903d
20: 0x1f07642 - rayon_core::registry::ThreadBuilder::run::hd50e617b60902b7c
21: 0x1f0d6d3 - std::sys_common::backtrace::__rust_begin_short_backtrace::hd96649c4a7c9e5d9
22: 0x1f0ebf4 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hb79e3c0d98138dc0
23: 0x1f8ecda - <alloc::boxed::Box as core::ops::function::FnOnce>::call_once::h1080dfe0ef616bdf
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/alloc/src/boxed.rs:1042
24: 0x1f8ecda - <alloc::boxed::Box as core::ops::function::FnOnce>::call_once::hd2747e1f2d5cec32
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/alloc/src/boxed.rs:1042
25: 0x1f8ecda - std::sys::unix::thread::Thread::new::thread_start::hd0f336b4ef6808a7
at /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/library/std/src/sys/unix/thread.rs:87
26: 0x7ff1bfda06db - start_thread
27: 0x7ff1bf46971f - __clone
28: 0x0 -

[jburnett]

We're experiencing this panic as well. This miner is not able to proceed to PC2 at all.
Ubuntu 20.04 srv; NVidia Quadro M5000 using 460.39 drivers (2/21/2021).

Key time frame is 03-02T16:09 in attached files:

1. systemd-miner-20210302-edited.log: includes Rust full backtraces
2. miner-20210302-2056.log: notice "caller":"storage-sealing/states_failed.go:25","msg":"SealPreCommit2Failed(3)..."
3. daemon-20210302-partial.log: included in case relevant to others

Most other miners are using older version of NVidia drivers. We'll install 450.66 and see if that changes our results.

[cryptonemo]

Please follow-up on the re-install of the drivers, as that has fixed this issue for others in the past. Also, make sure that nvidia-smi is returning proper GPU information.

[jburnett]

Miner is no longer panicking at PC2 after reverting to Nvidia 450.66. Thanks to those who assisted.

[yann1989]

NVIDIA-SMI 460.39 Driver Version: 460.39 CUDA Version: 11.2
Driver Version: 460.56
run smoothly.
It has nothing to do with the graphics card driver version

[yann1989]

恢复到Nvidia 450.66之后，Miner不再对PC2感到惊慌。感谢那些协助。

I have 450 460 435 Driver version. run smoothly.
But the same driver version, some machines will have the above problems.

[cryptonemo]

@yann1989 Yes, I agree. We have seen a case where re-installing even the same driver version 'fixes' the issue.

[bugxzhu]

same problem to me。

4. 2021-03-04 08:40:09 +0800 CST: [event;sealing.SectorSealPreCommit2Failed] {"User":{}}
   seal pre commit(2) failed: storage call error 0: presealing sector 2269 (): Rust panic: no unwind information
5. 2021-03-04 08:41:09 +0800 CST: [event;sealing.SectorRetrySealPreCommit2] {"User":{}}
6. 2021-03-04 08:41:10 +0800 CST: [event;sealing.SectorSealPreCommit2Failed] {"User":{}}
   seal pre commit(2) failed: storage call error 0: presealing sector 2269 (): Rust panic: no unwind information
7. 2021-03-04 08:42:10 +0800 CST: [event;sealing.SectorRetrySealPreCommit2] {"User":{}}
8. 2021-03-04 08:42:10 +0800 CST: [event;sealing.SectorSealPreCommit2Failed] {"User":{}}
   seal pre commit(2) failed: storage call error 0: presealing sector 2269 (): Rust panic: no unwind information
9. 2021-03-04 08:43:10 +0800 CST: [event;sealing.SectorRetrySealPreCommit2] {"User":{}}
10. 2021-03-04 08:43:10 +0800 CST: [event;sealing.SectorSealPreCommit2Failed] {"User":{}}
    seal pre commit(2) failed: storage call error 0: presealing sector 2269 (): Rust panic: no unwind information
11. 2021-03-04 08:44:10 +0800 CST: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}