Block proof is invalid: Invalid Vrf Proof value

**Describe the bug**
For given network:

![image](https://user-images.githubusercontent.com/20424186/85852258-81524100-b7b0-11ea-9da0-1c24c6dde360.png)

Where Leader1 is a legacy node, i can observe panic:

```
Jun 26 10:58:32.089 ERRO task process leadership block finished with error, error: Error(BlockHeaderVerificationFailed("Block proof is invalid: Invalid Vrf Proof value in 41584339-0000006e-1.56, poolId: 9aecced24ff0be4c18f7aa0d4ef9f6f9ae58c1b18a27e451742ed038dcb03f0e"), State { next_error: None, backtrace: InternalBacktrace { backtrace: Some(   0: backtrace::backtrace::trace_unsynchronized<closure-0>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\backtrace-0.3.48\src\backtrace\mod.rs:66
   1: backtrace::backtrace::trace<closure-0>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\backtrace-0.3.48\src\backtrace\mod.rs:53
   2: backtrace::capture::Backtrace::create
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\backtrace-0.3.48\src\capture.rs:164
   3: backtrace::capture::Backtrace::new_unresolved
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\backtrace-0.3.48\src\capture.rs:158
   4: error_chain::backtrace::imp::InternalBacktrace::new
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\error-chain-0.12.2\src\backtrace.rs:56
   5: error_chain::{{impl}}::default
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\error-chain-0.12.2\src\lib.rs:687
   6: jormungandr::blockchain::chain::Error::from_kind
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\error-chain-0.12.2\src\error_chain.rs:269
   7: jormungandr::blockchain::chain::{{impl}}::from
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\error-chain-0.12.2\src\error_chain.rs:370
   8: jormungandr::blockchain::chain::{{impl}}::post_check_header::{{closure}}
             at C:\Work\iohk\jormungandr\jormungandr\src\blockchain\chain.rs:444
   9: core::future::from_generator::{{impl}}::poll<generator-0>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libcore\future\mod.rs:73
  10: jormungandr::blockchain::process::process_leadership_block_inner::{{closure}}
             at C:\Work\iohk\jormungandr\jormungandr\src\blockchain\process.rs:423
  11: core::future::from_generator::{{impl}}::poll<generator-0>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libcore\future\mod.rs:73
  12: jormungandr::blockchain::process::process_leadership_block::{{closure}}
             at C:\Work\iohk\jormungandr\jormungandr\src\blockchain\process.rs:381
  13: core::future::from_generator::{{impl}}::poll<generator-0>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libcore\future\mod.rs:73
  14: tokio::time::timeout::{{impl}}::poll<core::future::from_generator::GenFuture<generator-0>>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\time\timeout.rs:155
  15: jormungandr::utils::task::{{impl}}::timeout_spawn_fallible::{{closure}}<core::future::from_generator::GenFuture<generator-0>,jormungandr::blockchain::chain::Error>
             at C:\Work\iohk\jormungandr\jormungandr\src\utils\task.rs:268
  16: core::future::from_generator::{{impl}}::poll<generator-0>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libcore\future\mod.rs:73
  17: tokio::runtime::task::core::{{impl}}::poll::{{closure}}<core::future::from_generator::GenFuture<generator-0>,alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\core.rs:173
  18: tokio::loom::std::unsafe_cell::UnsafeCell<tokio::runtime::task::core::Stage<core::future::from_generator::GenFuture<generator-0>>>::with_mut<tokio::runtime::task::core::Stage<core::future::from_generator::GenFuture<generator-0>>,core::task::poll::Poll<()>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\loom\std\unsafe_cell.rs:14
  19: tokio::runtime::task::core::Core<core::future::from_generator::GenFuture<generator-0>, alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>::poll<core::future::from_generator::GenFuture<generator-0>,alloc::sync::Arc<tokio::runtime::thread_pool::
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\core.rs:158
  20: tokio::runtime::task::harness::{{impl}}::poll::{{closure}}<core::future::from_generator::GenFuture<generator-0>,alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\harness.rs:107
  21: core::ops::function::FnOnce::call_once<closure-0,()>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libcore\ops\function.rs:232
  22: std::panic::{{impl}}::call_once<core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>,closure-0>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panic.rs:318
  23: std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure-0>,core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panicking.rs:297
  24: std::panicking::try::do_catch<std::panic::AssertUnwindSafe<closure-0>,()>
  25: std::panicking::try<core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>,std::panic::AssertUnwindSafe<closure-0>>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panicking.rs:274
  26: std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure-0>,core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panic.rs:394
  27: tokio::runtime::task::harness::Harness<core::future::from_generator::GenFuture<generator-0>, alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>::poll<core::future::from_generator::GenFuture<generator-0>,alloc::sync::Arc<tokio::runtime::thread_
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\harness.rs:89
  28: tokio::runtime::task::raw::poll<core::future::from_generator::GenFuture<generator-0>,alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\raw.rs:104
  29: tokio::runtime::task::raw::RawTask::poll
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\raw.rs:66
  30: tokio::runtime::task::Notified<alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>::run<alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\mod.rs:169
  31: tokio::runtime::thread_pool::worker::{{impl}}::run_task::{{closure}}
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\thread_pool\worker.rs:349
  32: tokio::coop::with_budget::{{closure}}<core::result::Result<alloc::boxed::Box<tokio::runtime::thread_pool::worker::Core>, ()>,closure-0>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\coop.rs:127
  33: std::thread::local::LocalKey<core::cell::Cell<tokio::coop::Budget>>::try_with<core::cell::Cell<tokio::coop::Budget>,closure-0,core::result::Result<alloc::boxed::Box<tokio::runtime::thread_pool::worker::Core>, ()>>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\thread\local.rs:263
  34: std::thread::local::LocalKey<core::cell::Cell<tokio::coop::Budget>>::with<core::cell::Cell<tokio::coop::Budget>,closure-0,core::result::Result<alloc::boxed::Box<tokio::runtime::thread_pool::worker::Core>, ()>>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\thread\local.rs:239
  35: tokio::coop::with_budget
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\coop.rs:120
  36: tokio::coop::budget
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\coop.rs:96
  37: tokio::runtime::thread_pool::worker::Context::run_task
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\thread_pool\worker.rs:348
  38: tokio::runtime::thread_pool::worker::Context::run
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\thread_pool\worker.rs:320
  39: tokio::runtime::thread_pool::worker::run::{{closure}}
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\thread_pool\worker.rs:305
  40: tokio::macros::scoped_tls::ScopedKey<tokio::runtime::thread_pool::worker::Context>::set<tokio::runtime::thread_pool::worker::Context,closure-0,()>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\macros\scoped_tls.rs:63
  41: tokio::runtime::thread_pool::worker::run
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\thread_pool\worker.rs:302
  42: tokio::runtime::thread_pool::worker::{{impl}}::launch::{{closure}}
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\thread_pool\worker.rs:281
  43: tokio::runtime::blocking::task::{{impl}}::poll<closure-0,()>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\blocking\task.rs:38
  44: tokio::runtime::task::core::{{impl}}::poll::{{closure}}<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedule>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\core.rs:173
  45: tokio::loom::std::unsafe_cell::UnsafeCell<tokio::runtime::task::core::Stage<tokio::runtime::blocking::task::BlockingTask<closure-0>>>::with_mut<tokio::runtime::task::core::Stage<tokio::runtime::blocking::task::BlockingTask<closure-0>>,core::task::poll::Po
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\loom\std\unsafe_cell.rs:14
  46: tokio::runtime::task::core::Core<tokio::runtime::blocking::task::BlockingTask<closure-0>, tokio::runtime::blocking::schedule::NoopSchedule>::poll<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedule>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\core.rs:158
  47: tokio::runtime::task::harness::{{impl}}::poll::{{closure}}<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedule>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\harness.rs:107
  48: core::ops::function::FnOnce::call_once<closure-0,()>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libcore\ops\function.rs:232
  49: std::panic::{{impl}}::call_once<core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>,closure-0>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panic.rs:318
  50: std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure-0>,core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panicking.rs:297
  51: tokio::time::instant::{{impl}}::gt
  52: std::panicking::try<core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>,std::panic::AssertUnwindSafe<closure-0>>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panicking.rs:274
  53: std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure-0>,core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panic.rs:394
  54: tokio::runtime::task::harness::Harness<tokio::runtime::blocking::task::BlockingTask<closure-0>, tokio::runtime::blocking::schedule::NoopSchedule>::poll<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedul
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\harness.rs:89
  55: tokio::runtime::task::raw::poll<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedule>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\raw.rs:104
  56: tokio::runtime::task::raw::RawTask::poll
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\raw.rs:66
  57: tokio::runtime::task::Notified<tokio::runtime::blocking::schedule::NoopSchedule>::run<tokio::runtime::blocking::schedule::NoopSchedule>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\mod.rs:169
  58: tokio::runtime::blocking::pool::Inner::run
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\blocking\pool.rs:230
  59: tokio::runtime::blocking::pool::{{impl}}::spawn_thread::{{closure}}::{{closure}}
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\blocking\pool.rs:210
  60: tokio::runtime::context::enter<closure-0,()>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\context.rs:72
  61: tokio::runtime::handle::Handle::enter<closure-0,()>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\handle.rs:76
  62: tokio::runtime::blocking::pool::{{impl}}::spawn_thread::{{closure}}
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\blocking\pool.rs:209
  63: std::sys_common::backtrace::__rust_begin_short_backtrace<closure-0,()>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\sys_common\backtrace.rs:130
  64: std::thread::{{impl}}::spawn_unchecked::{{closure}}::{{closure}}<closure-0,()>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\thread\mod.rs:475
  65: std::panic::{{impl}}::call_once<(),closure-0>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panic.rs:318
  66: std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure-0>,()>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panicking.rs:297
  67: tokio::time::instant::{{impl}}::gt
  68: std::panicking::try<(),std::panic::AssertUnwindSafe<closure-0>>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panicking.rs:274
  69: std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure-0>,()>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panic.rs:394
  70: std::thread::{{impl}}::spawn_unchecked::{{closure}}<closure-0,()>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\thread\mod.rs:474
  71: core::ops::function::FnOnce::call_once<closure-0,()>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libcore\ops\function.rs:232
  72: alloc::boxed::{{impl}}::call_once
             at /rustc/b7dc83a3f6ca9746fb3d121761c3605477b77d90\src\liballoc\boxed.rs:1076
  73: alloc::boxed::{{impl}}::call_once
             at /rustc/b7dc83a3f6ca9746fb3d121761c3605477b77d90\src\liballoc\boxed.rs:1076
  74: std::sys::windows::thread::{{impl}}::new::thread_start
             at /rustc/b7dc83a3f6ca9746fb3d121761c3605477b77d90\/src\libstd\sys\windows\thread.rs:56
  75: BaseThreadInitThunk
  76: RtlUserThreadStart 
```
in leaders: 1,2,3,4 which i believe is causing lack of sync after leader1 restart

**Mandatory Information**
1. `jcli --full-version` newest master
2. `jormungandr --full-version` newest master

**To Reproduce**
Steps to reproduce the behavior:
1. Start Leader1
2. Start Leader2,3,4
3. Wait and send some tx's (attached fragments)
4. Restart Leader4.
5. Everything should get back to normal (nodes are in sync)
6. Restart Leader1 (legacy one)
7. In logs Invalid Vrf Proof errors start appearing, despite Leader1

**Expected behavior**
After Leader1 restart all nodes can sync

**Additional context**
run `legacy_disruption_last_4th_release` scenario (80% reproduction).
Or use below pack to bootrstap network:


[leader2_node.log](https://github.com/input-output-hk/jormungandr/files/4836656/leader2_node.log)
[legacy_disruption_last_4th_release.zip](https://github.com/input-output-hk/jormungandr/files/4836660/legacy_disruption_last_4th_release.zip)



Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Block proof is invalid: Invalid Vrf Proof value #2428

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Block proof is invalid: Invalid Vrf Proof value #2428

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions