Test flake - TestEpochLeaderFailoverWithEmptyNotarization-with-crash #108
Description
Seems like the test sometimes makes the node propose a block with a different digest than the votes that are prepared for it.
The node proposes the following block:
2025-02-27T17:07:07.4128104Z [02-27|16:57:06.213] DEBUG Simplex/epoch.go:1619 Proposal broadcast {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "round": 4, "size": 125, "digest": "1b0aab43dbc712af854e..."}
However the test creates the following votes:
2025-02-27T17:07:07.4119905Z [02-27|16:57:06.212] DEBUG Simplex/epoch.go:651 Received vote message {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "from": "03", "round": 4, "digest": "fad5b74a829a60610bff..."}
Log output:
2025-02-27T17:07:07.3980815Z [02-27|16:57:06.200] DEBUG Simplex/sched.go:95 No ready tasks, going to sleep {"test": "TestAsyncScheduler/Executes_several_pending_tasks_concurrently_in_arbitrary_order"}
2025-02-27T17:07:07.3981314Z [02-27|16:57:06.200] DEBUG Simplex/sched.go:97 Woken up from sleep {"test": "TestAsyncScheduler/Executes_several_pending_tasks_concurrently_in_arbitrary_order", "ready tasks": 0}
2025-02-27T17:07:07.3981779Z [02-27|16:57:06.201] DEBUG Simplex/epoch.go:1564 Scheduling block building {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 0}
2025-02-27T17:07:07.3982174Z [02-27|16:57:06.201] DEBUG Simplex/sched.go:95 No ready tasks, going to sleep {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1}
2025-02-27T17:07:07.3982706Z [02-27|16:57:06.201] DEBUG Simplex/sched.go:137 Scheduling new ready task {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "dependency": "00000000000000000000..."}
2025-02-27T17:07:07.3983241Z [02-27|16:57:06.201] DEBUG Simplex/sched.go:97 Woken up from sleep {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "ready tasks": 1}
2025-02-27T17:07:07.3983700Z [02-27|16:57:06.201] DEBUG Simplex/sched.go:107 Running task {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "remaining ready tasks": 0}
2025-02-27T17:07:07.3984338Z [02-27|16:57:06.201] DEBUG Simplex/epoch.go:1597 Wrote block to WAL {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 0, "size": 125, "digest": "f83a59cbcffde4ef1665..."}
2025-02-27T17:07:07.3984905Z [02-27|16:57:06.201] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "01", "round": 0}
2025-02-27T17:07:07.3985528Z [02-27|16:57:06.201] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 0}
2025-02-27T17:07:07.3986085Z [02-27|16:57:06.201] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 0}
2025-02-27T17:07:07.3986635Z [02-27|16:57:06.201] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "04", "round": 0}
2025-02-27T17:07:07.3987350Z [02-27|16:57:06.201] DEBUG Simplex/epoch.go:1619 Proposal broadcast {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 0, "size": 125, "digest": "f83a59cbcffde4ef1665..."}
2025-02-27T17:07:07.3987945Z [02-27|16:57:06.201] DEBUG Simplex/epoch.go:651 Received vote message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "01", "round": 0, "digest": "f83a59cbcffde4ef1665..."}
2025-02-27T17:07:07.3988464Z [02-27|16:57:06.201] DEBUG Simplex/epoch.go:1015 Counting votes {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 0, "votes": 1, "from": "[01]"}
2025-02-27T17:07:07.3993505Z [02-27|16:57:06.202] DEBUG Simplex/sched.go:109 Task finished execution {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "taskID": "f83a59cbcffde4ef1665..."}
2025-02-27T17:07:07.3994018Z [02-27|16:57:06.202] DEBUG Simplex/sched.go:114 Enqueued newly ready tasks {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "number of ready tasks": 0}
2025-02-27T17:07:07.3994410Z [02-27|16:57:06.202] DEBUG Simplex/sched.go:95 No ready tasks, going to sleep {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1}
2025-02-27T17:07:07.3994978Z [02-27|16:57:06.202] DEBUG Simplex/epoch.go:651 Received vote message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 0, "digest": "f83a59cbcffde4ef1665..."}
2025-02-27T17:07:07.3995549Z [02-27|16:57:06.202] DEBUG Simplex/epoch.go:1015 Counting votes {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 0, "votes": 2, "from": "[02 01]"}
2025-02-27T17:07:07.3998390Z [02-27|16:57:06.202] DEBUG Simplex/epoch.go:651 Received vote message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 0, "digest": "f83a59cbcffde4ef1665..."}
2025-02-27T17:07:07.3998892Z [02-27|16:57:06.202] INFO Simplex/notarization.go:46 Collected Quorum of votes {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 0, "votes": 3}
2025-02-27T17:07:07.3999361Z [02-27|16:57:06.202] DEBUG Simplex/notarization.go:50 Collected vote from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "01"}
2025-02-27T17:07:07.3999832Z [02-27|16:57:06.202] DEBUG Simplex/notarization.go:50 Collected vote from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "02"}
2025-02-27T17:07:07.4000296Z [02-27|16:57:06.202] DEBUG Simplex/notarization.go:50 Collected vote from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "03"}
2025-02-27T17:07:07.4000899Z [02-27|16:57:06.202] DEBUG Simplex/epoch.go:1061 Persisted notarization to WAL {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "size": 129, "round": 0, "digest": "f83a59cbcffde4ef1665..."}
2025-02-27T17:07:07.4001444Z [02-27|16:57:06.202] DEBUG Simplex/epoch.go:1074 Broadcast notarization {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 0, "digest": "f83a59cbcffde4ef1665..."}
2025-02-27T17:07:07.4001962Z [02-27|16:57:06.202] INFO Simplex/epoch.go:1825 Moving to a new round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "old round": 0, "new round": 1, "leader": "01"}
2025-02-27T17:07:07.4002388Z [02-27|16:57:06.202] DEBUG Simplex/epoch.go:1703 Monitoring progress {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1}
2025-02-27T17:07:07.4002904Z [02-27|16:57:06.202] DEBUG Simplex/epoch.go:518 Received finalization message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "01", "round": 0}
2025-02-27T17:07:07.4003412Z [02-27|16:57:06.202] DEBUG Simplex/monitor.go:79 Executing f {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "taskID": 0}
2025-02-27T17:07:07.4003888Z [02-27|16:57:06.203] DEBUG Simplex/epoch.go:789 Counting finalizations {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1, "votes": 1}
d finalization message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 0}
2025-02-27T17:07:07.4007101Z [02-27|16:57:06.203] DEBUG Simplex/epoch.go:789 Counting finalizations {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1, "votes": 2}
2025-02-27T17:07:07.4007596Z [02-27|16:57:06.203] DEBUG Simplex/epoch.go:518 Received finalization message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 0}
2025-02-27T17:07:07.4008249Z [02-27|16:57:06.203] DEBUG Simplex/notarization.go:87 Collected finalization from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "02", "round": 0}
2025-02-27T17:07:07.4008785Z [02-27|16:57:06.203] DEBUG Simplex/notarization.go:87 Collected finalization from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "03", "round": 0}
2025-02-27T17:07:07.4009324Z [02-27|16:57:06.203] DEBUG Simplex/notarization.go:87 Collected finalization from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "01", "round": 0}
2025-02-27T17:07:07.4009842Z [02-27|16:57:06.203] DEBUG Simplex/epoch.go:836 Received enough finalizations to finalize a block {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 0}
2025-02-27T17:07:07.4010397Z [02-27|16:57:06.203] INFO Simplex/epoch.go:913 Committed block {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 0, "sequence": 0, "digest": "f83a59cbcffde4ef1665..."}
2025-02-27T17:07:07.4010857Z [02-27|16:57:06.203] DEBUG Simplex/epoch.go:829 Progressing rounds due to commit {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1}
2025-02-27T17:07:07.4011448Z [02-27|16:57:06.203] DEBUG Simplex/epoch.go:864 Broadcast finalization certificate {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 0, "digest": "f83a59cbcffde4ef1665..."}
2025-02-27T17:07:07.4011927Z [02-27|16:57:06.203] DEBUG Simplex/epoch.go:1086 Received empty notarization message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 3}
2025-02-27T17:07:07.4012498Z [02-27|16:57:06.203] DEBUG Simplex/epoch.go:1117 Received empty notarization for a future round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 3, "our round": 1}
2025-02-27T17:07:07.4013066Z [02-27|16:57:06.203] DEBUG Simplex/epoch.go:1177 Received block message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 1}
2025-02-27T17:07:07.4013613Z [02-27|16:57:06.203] DEBUG Simplex/epoch.go:1191 Handling block message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "digest": "68d45ca370007f9a0aa3...", "round": 1}
2025-02-27T17:07:07.4014068Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:1280 Scheduling block verification {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1}
2025-02-27T17:07:07.4014598Z [02-27|16:57:06.204] DEBUG Simplex/sched.go:137 Scheduling new ready task {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "dependency": "f83a59cbcffde4ef1665..."}
2025-02-27T17:07:07.4015021Z [02-27|16:57:06.204] DEBUG Simplex/sched.go:97 Woken up from sleep {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "ready tasks": 1}
2025-02-27T17:07:07.4015469Z [02-27|16:57:06.204] DEBUG Simplex/sched.go:107 Running task {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "remaining ready tasks": 0}
2025-02-27T17:07:07.4015919Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:1331 Block verification started {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1}
2025-02-27T17:07:07.4016541Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:651 Received vote message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 1, "digest": "68d45ca370007f9a0aa3..."}
2025-02-27T17:07:07.4017106Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:673 Received a vote for the current round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1, "from": "03"}
2025-02-27T17:07:07.4017609Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:518 Received finalization message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 1}
2025-02-27T17:07:07.4018158Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:537 Received a finalization for the current round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1, "from": "02"}
2025-02-27T17:07:07.4018777Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:518 Received finalization message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 1}
2025-02-27T17:07:07.4019335Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:537 Received a finalization for the current round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1, "from": "03"}
2025-02-27T17:07:07.4019870Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:1353 Persisted block to WAL {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1, "digest": "68d45ca370007f9a0aa3..."}
2025-02-27T17:07:07.4021411Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:1177 Received block message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 1}
2025-02-27T17:07:07.4021944Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:1191 Handling block message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "digest": "68d45ca370007f9a0aa3...", "round": 1}
2025-02-27T17:07:07.4022408Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:1280 Scheduling block verification {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1}
2025-02-27T17:07:07.4022905Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:518 Received finalization message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 1}
2025-02-27T17:07:07.4023476Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:789 Counting finalizations {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1, "votes": 1}
2025-02-27T17:07:07.4024059Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:651 Received vote message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 1, "digest": "68d45ca370007f9a0aa3..."}
2025-02-27T17:07:07.4024550Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:1015 Counting votes {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1, "votes": 1, "from": "[03]"}
2025-02-27T17:07:07.4025049Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:518 Received finalization message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 1}
2025-02-27T17:07:07.4025522Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:789 Counting finalizations {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1, "votes": 2}
2025-02-27T17:07:07.4026050Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "04", "round": 1}
2025-02-27T17:07:07.4026664Z [02-27|16:57:06.204] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "01", "round": 1}
2025-02-27T17:07:07.4027141Z [02-27|16:57:06.205] DEBUG Simplex/epoch.go:1177 Received block message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 1}
2025-02-27T17:07:07.4027730Z [02-27|16:57:06.205] DEBUG Simplex/epoch.go:1191 Handling block message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "digest": "68d45ca370007f9a0aa3...", "round": 1}
2025-02-27T17:07:07.4028197Z [02-27|16:57:06.205] DEBUG Simplex/epoch.go:1280 Scheduling block verification {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1}
2025-02-27T17:07:07.4028725Z [02-27|16:57:06.205] DEBUG Simplex/epoch.go:1793 Broadcasting vote {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1, "digest": "68d45ca370007f9a0aa3..."}
2025-02-27T17:07:07.4029381Z [02-27|16:57:06.205] DEBUG Simplex/epoch.go:651 Received vote message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "01", "round": 1, "digest": "68d45ca370007f9a0aa3..."}
2025-02-27T17:07:07.4029889Z [02-27|16:57:06.205] INFO Simplex/notarization.go:46 Collected Quorum of votes {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1, "votes": 3}
2025-02-27T17:07:07.4030353Z [02-27|16:57:06.205] DEBUG Simplex/notarization.go:50 Collected vote from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "03"}
2025-02-27T17:07:07.4030819Z [02-27|16:57:06.205] DEBUG Simplex/notarization.go:50 Collected vote from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "02"}
2025-02-27T17:07:07.4031276Z [02-27|16:57:06.205] DEBUG Simplex/notarization.go:50 Collected vote from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "01"}
2025-02-27T17:07:07.4031873Z [02-27|16:57:06.205] DEBUG Simplex/epoch.go:1061 Persisted notarization to WAL {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "size": 129, "round": 1, "digest": "68d45ca370007f9a0aa3..."}
2025-02-27T17:07:07.4032413Z [02-27|16:57:06.205] DEBUG Simplex/epoch.go:1074 Broadcast notarization {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1, "digest": "68d45ca370007f9a0aa3..."}
2025-02-27T17:07:07.4033021Z [02-27|16:57:06.205] INFO Simplex/epoch.go:1825 Moving to a new round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "old round": 1, "new round": 2, "leader": "02"}
2025-02-27T17:07:07.4033432Z [02-27|16:57:06.205] DEBUG Simplex/monitor.go:81 Task executed {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "taskID": 0}
2025-02-27T17:07:07.4033863Z [02-27|16:57:06.205] DEBUG Simplex/epoch.go:1703 Monitoring progress {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 2}
2025-02-27T17:07:07.4034364Z [02-27|16:57:06.205] DEBUG Simplex/epoch.go:518 Received finalization message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "01", "round": 1}
2025-02-27T17:07:07.4034765Z [02-27|16:57:06.205] DEBUG Simplex/monitor.go:79 Executing f {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "taskID": 1}
2025-02-27T17:07:07.4035303Z [02-27|16:57:06.205] DEBUG Simplex/notarization.go:87 Collected finalization from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "02", "round": 1}
2025-02-27T17:07:07.4035839Z [02-27|16:57:06.205] DEBUG Simplex/notarization.go:87 Collected finalization from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "03", "round": 1}
2025-02-27T17:07:07.4036374Z [02-27|16:57:06.205] DEBUG Simplex/notarization.go:87 Collected finalization from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "01", "round": 1}
2025-02-27T17:07:07.4036952Z [02-27|16:57:06.205] DEBUG Simplex/epoch.go:836 Received enough finalizations to finalize a block {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1}
2025-02-27T17:07:07.4037505Z [02-27|16:57:06.205] INFO Simplex/epoch.go:913 Committed block {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1, "sequence": 1, "digest": "68d45ca370007f9a0aa3..."}
2025-02-27T17:07:07.4038021Z [02-27|16:57:06.205] DEBUG Simplex/epoch.go:829 Progressing rounds due to commit {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 2}
2025-02-27T17:07:07.4038605Z [02-27|16:57:06.205] DEBUG Simplex/epoch.go:864 Broadcast finalization certificate {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1, "digest": "68d45ca370007f9a0aa3..."}
2025-02-27T17:07:07.4039134Z [02-27|16:57:06.205] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "04", "round": 2}
2025-02-27T17:07:07.4039767Z [02-27|16:57:06.205] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "01", "round": 2}
2025-02-27T17:07:07.4040292Z [02-27|16:57:06.205] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 2}
2025-02-27T17:07:07.4040818Z [02-27|16:57:06.205] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 2}
2025-02-27T17:07:07.4041332Z [02-27|16:57:06.205] DEBUG Simplex/epoch.go:1335 Block verification ended {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 1, "elapsed": "1.68151ms"}
2025-02-27T17:07:07.4041845Z [02-27|16:57:06.205] DEBUG Simplex/sched.go:109 Task finished execution {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "taskID": "68d45ca370007f9a0aa3..."}
2025-02-27T17:07:07.4042334Z [02-27|16:57:06.206] DEBUG Simplex/sched.go:114 Enqueued newly ready tasks {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "number of ready tasks": 0}
2025-02-27T17:07:07.4042725Z [02-27|16:57:06.206] DEBUG Simplex/sched.go:95 No ready tasks, going to sleep {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1}
2025-02-27T17:07:07.4043296Z [02-27|16:57:06.206] DEBUG Simplex/epoch.go:1177 Received block message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 2}
2025-02-27T17:07:07.4043833Z [02-27|16:57:06.206] DEBUG Simplex/epoch.go:1191 Handling block message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "digest": "39428085912723bd2fe2...", "round": 2}
2025-02-27T17:07:07.4044290Z [02-27|16:57:06.206] DEBUG Simplex/epoch.go:1280 Scheduling block verification {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 2}
2025-02-27T17:07:07.4044820Z [02-27|16:57:06.206] DEBUG Simplex/sched.go:137 Scheduling new ready task {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "dependency": "68d45ca370007f9a0aa3..."}
2025-02-27T17:07:07.4045372Z [02-27|16:57:06.206] DEBUG Simplex/epoch.go:651 Received vote message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 2, "digest": "39428085912723bd2fe2..."}
2025-02-27T17:07:07.4045800Z [02-27|16:57:06.206] DEBUG Simplex/sched.go:97 Woken up from sleep {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "ready tasks": 1}
2025-02-27T17:07:07.4046245Z [02-27|16:57:06.206] DEBUG Simplex/sched.go:107 Running task {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "remaining ready tasks": 0}
2025-02-27T17:07:07.4046693Z [02-27|16:57:06.206] DEBUG Simplex/epoch.go:1331 Block verification started {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 2}
2025-02-27T17:07:07.4047267Z [02-27|16:57:06.206] DEBUG Simplex/epoch.go:673 Received a vote for the current round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 2, "from": "02"}
2025-02-27T17:07:07.4047800Z [02-27|16:57:06.206] DEBUG Simplex/epoch.go:1353 Persisted block to WAL {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 2, "digest": "39428085912723bd2fe2..."}
2025-02-27T17:07:07.4048381Z [02-27|16:57:06.206] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "01", "round": 2}
2025-02-27T17:07:07.4048941Z [02-27|16:57:06.206] DEBUG Simplex/epoch.go:651 Received vote message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 2, "digest": "39428085912723bd2fe2..."}
2025-02-27T17:07:07.4049427Z [02-27|16:57:06.206] DEBUG Simplex/epoch.go:1015 Counting votes {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 2, "votes": 1, "from": "[02]"}
2025-02-27T17:07:07.4050090Z [02-27|16:57:06.206] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "01", "round": 2}
2025-02-27T17:07:07.4050591Z [02-27|16:57:06.206] DEBUG Simplex/epoch.go:1177 Received block message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 2}
2025-02-27T17:07:07.4051147Z [02-27|16:57:06.206] DEBUG Simplex/epoch.go:1191 Handling block message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "digest": "39428085912723bd2fe2...", "round": 2}
2025-02-27T17:07:07.4051623Z [02-27|16:57:06.206] DEBUG Simplex/epoch.go:1280 Scheduling block verification {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 2}
2025-02-27T17:07:07.4052169Z [02-27|16:57:06.206] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "04", "round": 2}
2025-02-27T17:07:07.4052643Z [02-27|16:57:06.206] DEBUG Simplex/epoch.go:1177 Received block message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 2}
2025-02-27T17:07:07.4053264Z [02-27|16:57:06.206] DEBUG Simplex/epoch.go:1191 Handling block message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "digest": "39428085912723bd2fe2...", "round": 2}
2025-02-27T17:07:07.4053721Z [02-27|16:57:06.207] DEBUG Simplex/epoch.go:1280 Scheduling block verification {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 2}
2025-02-27T17:07:07.4054252Z [02-27|16:57:06.207] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "04", "round": 2}
2025-02-27T17:07:07.4054769Z [02-27|16:57:06.207] DEBUG Simplex/epoch.go:1793 Broadcasting vote {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 2, "digest": "39428085912723bd2fe2..."}
2025-02-27T17:07:07.4055331Z [02-27|16:57:06.207] DEBUG Simplex/epoch.go:651 Received vote message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "01", "round": 2, "digest": "39428085912723bd2fe2..."}
2025-02-27T17:07:07.4055832Z [02-27|16:57:06.207] INFO Simplex/notarization.go:46 Collected Quorum of votes {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 2, "votes": 3}
2025-02-27T17:07:07.4056296Z [02-27|16:57:06.207] DEBUG Simplex/notarization.go:50 Collected vote from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "02"}
2025-02-27T17:07:07.4056760Z [02-27|16:57:06.207] DEBUG Simplex/notarization.go:50 Collected vote from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "03"}
2025-02-27T17:07:07.4057221Z [02-27|16:57:06.207] DEBUG Simplex/notarization.go:50 Collected vote from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "01"}
2025-02-27T17:07:07.4057878Z [02-27|16:57:06.207] DEBUG Simplex/epoch.go:1061 Persisted notarization to WAL {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "size": 129, "round": 2, "digest": "39428085912723bd2fe2..."}
2025-02-27T17:07:07.4058468Z [02-27|16:57:06.207] DEBUG Simplex/epoch.go:1074 Broadcast notarization {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 2, "digest": "39428085912723bd2fe2..."}
2025-02-27T17:07:07.4058986Z [02-27|16:57:06.207] INFO Simplex/epoch.go:1825 Moving to a new round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "old round": 2, "new round": 3, "leader": "03"}
2025-02-27T17:07:07.4059410Z [02-27|16:57:06.207] DEBUG Simplex/epoch.go:1703 Monitoring progress {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 3}
2025-02-27T17:07:07.4059873Z [02-27|16:57:06.207] DEBUG Simplex/monitor.go:81 Task executed {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "taskID": 1}
2025-02-27T17:07:07.4060457Z [02-27|16:57:06.207] DEBUG Simplex/epoch.go:518 Received finalization message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "01", "round": 2}
2025-02-27T17:07:07.4060856Z [02-27|16:57:06.207] DEBUG Simplex/monitor.go:79 Executing f {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "taskID": 2}
2025-02-27T17:07:07.4061326Z [02-27|16:57:06.207] DEBUG Simplex/epoch.go:789 Counting finalizations {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 3, "votes": 1}
2025-02-27T17:07:07.4061860Z [02-27|16:57:06.207] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 3}
2025-02-27T17:07:07.4062391Z [02-27|16:57:06.207] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 3}
2025-02-27T17:07:07.4063015Z [02-27|16:57:06.207] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "04", "round": 3}
2025-02-27T17:07:07.4063539Z [02-27|16:57:06.207] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "01", "round": 3}
2025-02-27T17:07:07.4064018Z [02-27|16:57:06.207] DEBUG Simplex/epoch.go:1086 Received empty notarization message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 3}
2025-02-27T17:07:07.4064493Z [02-27|16:57:06.208] DEBUG Simplex/epoch.go:988 Persisted empty block to WAL {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "size": 91, "round": 3}
2025-02-27T17:07:07.4065011Z [02-27|16:57:06.208] INFO Simplex/epoch.go:1825 Moving to a new round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "old round": 3, "new round": 4, "leader": "04"}
2025-02-27T17:07:07.4065417Z [02-27|16:57:06.208] DEBUG Simplex/monitor.go:81 Task executed {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "taskID": 2}
2025-02-27T17:07:07.4065869Z [02-27|16:57:06.208] DEBUG Simplex/epoch.go:1564 Scheduling block building {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 4}
2025-02-27T17:07:07.4066387Z [02-27|16:57:06.208] DEBUG Simplex/sched.go:137 Scheduling new ready task {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "dependency": "39428085912723bd2fe2..."}
2025-02-27T17:07:07.4071364Z [02-27|16:57:06.208] DEBUG Simplex/epoch.go:1335 Block verification ended {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 2, "elapsed": "2.152071ms"}
2025-02-27T17:07:07.4071869Z [02-27|16:57:06.208] DEBUG Simplex/sched.go:109 Task finished execution {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "taskID": "39428085912723bd2fe2..."}
2025-02-27T17:07:07.4072370Z [02-27|16:57:06.208] DEBUG Simplex/epoch.go:518 Received finalization message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 2}
2025-02-27T17:07:07.4072861Z [02-27|16:57:06.208] DEBUG Simplex/sched.go:114 Enqueued newly ready tasks {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "number of ready tasks": 0}
2025-02-27T17:07:07.4073409Z [02-27|16:57:06.208] DEBUG Simplex/sched.go:107 Running task {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "remaining ready tasks": 0}
2025-02-27T17:07:07.4073884Z [02-27|16:57:06.208] DEBUG Simplex/epoch.go:789 Counting finalizations {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 4, "votes": 2}
2025-02-27T17:07:07.4074382Z [02-27|16:57:06.208] DEBUG Simplex/epoch.go:518 Received finalization message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 2}
2025-02-27T17:07:07.4074923Z [02-27|16:57:06.208] DEBUG Simplex/notarization.go:87 Collected finalization from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "01", "round": 2}
2025-02-27T17:07:07.4075453Z [02-27|16:57:06.208] DEBUG Simplex/notarization.go:87 Collected finalization from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "02", "round": 2}
2025-02-27T17:07:07.4075994Z [02-27|16:57:06.208] DEBUG Simplex/notarization.go:87 Collected finalization from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "03", "round": 2}
2025-02-27T17:07:07.4076508Z [02-27|16:57:06.209] DEBUG Simplex/epoch.go:836 Received enough finalizations to finalize a block {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 2}
2025-02-27T17:07:07.4077056Z [02-27|16:57:06.209] INFO Simplex/epoch.go:913 Committed block {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 2, "sequence": 2, "digest": "39428085912723bd2fe2..."}
2025-02-27T17:07:07.4077511Z [02-27|16:57:06.209] DEBUG Simplex/epoch.go:829 Progressing rounds due to commit {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 3}
2025-02-27T17:07:07.4078105Z [02-27|16:57:06.209] DEBUG Simplex/epoch.go:864 Broadcast finalization certificate {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 2, "digest": "39428085912723bd2fe2..."}
2025-02-27T17:07:07.4078709Z [02-27|16:57:06.209] DEBUG Simplex/epoch.go:1597 Wrote block to WAL {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 4, "size": 125, "digest": "fad5b74a829a60610bff..."}
2025-02-27T17:07:07.4079305Z [02-27|16:57:06.209] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 4}
2025-02-27T17:07:07.4079833Z [02-27|16:57:06.209] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "04", "round": 4}
2025-02-27T17:07:07.4080362Z [02-27|16:57:06.209] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "01", "round": 4}
2025-02-27T17:07:07.4080989Z [02-27|16:57:06.209] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 4}
2025-02-27T17:07:07.4081550Z [02-27|16:57:06.209] DEBUG Simplex/epoch.go:1619 Proposal broadcast {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 4, "size": 125, "digest": "fad5b74a829a60610bff..."}
2025-02-27T17:07:07.4082107Z [02-27|16:57:06.209] DEBUG Simplex/epoch.go:651 Received vote message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "01", "round": 4, "digest": "fad5b74a829a60610bff..."}
2025-02-27T17:07:07.4082599Z [02-27|16:57:06.209] DEBUG Simplex/epoch.go:1015 Counting votes {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 4, "votes": 1, "from": "[01]"}
2025-02-27T17:07:07.4083221Z [02-27|16:57:06.209] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 4}
2025-02-27T17:07:07.4083753Z [02-27|16:57:06.209] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 4}
2025-02-27T17:07:07.4084274Z [02-27|16:57:06.209] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "04", "round": 4}
2025-02-27T17:07:07.4084803Z [02-27|16:57:06.209] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "01", "round": 4}
2025-02-27T17:07:07.4085324Z [02-27|16:57:06.209] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "04", "round": 4}
2025-02-27T17:07:07.4085850Z [02-27|16:57:06.209] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "01", "round": 4}
2025-02-27T17:07:07.4086377Z [02-27|16:57:06.209] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 4}
2025-02-27T17:07:07.4086908Z [02-27|16:57:06.210] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 4}
2025-02-27T17:07:07.4087417Z [02-27|16:57:06.210] DEBUG Simplex/sched.go:109 Task finished execution {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "taskID": "fad5b74a829a60610bff..."}
2025-02-27T17:07:07.4087909Z [02-27|16:57:06.210] DEBUG Simplex/sched.go:114 Enqueued newly ready tasks {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "number of ready tasks": 0}
2025-02-27T17:07:07.4088298Z [02-27|16:57:06.210] DEBUG Simplex/sched.go:95 No ready tasks, going to sleep {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1}
2025-02-27T17:07:07.4088921Z [02-27|16:57:06.210] DEBUG Simplex/epoch.go:651 Received vote message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 4, "digest": "fad5b74a829a60610bff..."}
2025-02-27T17:07:07.4089416Z [02-27|16:57:06.210] DEBUG Simplex/epoch.go:1015 Counting votes {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 4, "votes": 2, "from": "[01 02]"}
2025-02-27T17:07:07.4090004Z [02-27|16:57:06.210] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "01", "round": 4}
2025-02-27T17:07:07.4090529Z [02-27|16:57:06.210] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 4}
2025-02-27T17:07:07.4091111Z [02-27|16:57:06.210] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 4}
2025-02-27T17:07:07.4091703Z [02-27|16:57:06.210] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "04", "round": 4}
2025-02-27T17:07:07.4092262Z [02-27|16:57:06.210] DEBUG Simplex/epoch.go:651 Received vote message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 4, "digest": "fad5b74a829a60610bff..."}
2025-02-27T17:07:07.4092758Z [02-27|16:57:06.210] INFO Simplex/notarization.go:46 Collected Quorum of votes {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 4, "votes": 3}
2025-02-27T17:07:07.4093523Z [02-27|16:57:06.210] DEBUG Simplex/notarization.go:50 Collected vote from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "01"}
2025-02-27T17:07:07.4094001Z [02-27|16:57:06.210] DEBUG Simplex/notarization.go:50 Collected vote from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "02"}
2025-02-27T17:07:07.4094470Z [02-27|16:57:06.210] DEBUG Simplex/notarization.go:50 Collected vote from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "03"}
2025-02-27T17:07:07.4095062Z [02-27|16:57:06.210] DEBUG Simplex/epoch.go:1061 Persisted notarization to WAL {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "size": 129, "round": 4, "digest": "fad5b74a829a60610bff..."}
2025-02-27T17:07:07.4095609Z [02-27|16:57:06.210] DEBUG Simplex/epoch.go:1074 Broadcast notarization {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 4, "digest": "fad5b74a829a60610bff..."}
2025-02-27T17:07:07.4096118Z [02-27|16:57:06.210] INFO Simplex/epoch.go:1825 Moving to a new round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "old round": 4, "new round": 5, "leader": "01"}
2025-02-27T17:07:07.4096557Z [02-27|16:57:06.210] DEBUG Simplex/epoch.go:1703 Monitoring progress {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 5}
2025-02-27T17:07:07.4097056Z [02-27|16:57:06.210] DEBUG Simplex/epoch.go:518 Received finalization message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "01", "round": 4}
2025-02-27T17:07:07.4097456Z [02-27|16:57:06.210] DEBUG Simplex/monitor.go:79 Executing f {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "taskID": 3}
2025-02-27T17:07:07.4097878Z [02-27|16:57:06.211] INFO Simplex/epoch.go:1727 It is time to build a block {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 5}
2025-02-27T17:07:07.4098348Z [02-27|16:57:06.211] DEBUG Simplex/epoch.go:789 Counting finalizations {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 5, "votes": 1}
2025-02-27T17:07:07.4098877Z [02-27|16:57:06.211] DEBUG Simplex/monitor.go:122 Scheduling task {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "timeout": "5s", "deadline": "[02-27|16:57:06.200]"}
2025-02-27T17:07:07.4099483Z [02-27|16:57:06.211] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "01", "round": 5}
2025-02-27T17:07:07.4100069Z [02-27|16:57:06.211] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 5}
2025-02-27T17:07:07.4100593Z [02-27|16:57:06.211] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 5}
2025-02-27T17:07:07.4101119Z [02-27|16:57:06.211] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "04", "round": 5}
2025-02-27T17:07:07.4101740Z [02-27|16:57:06.211] DEBUG Simplex/epoch.go:518 Received finalization message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "02", "round": 4}
2025-02-27T17:07:07.4102214Z [02-27|16:57:06.211] DEBUG Simplex/epoch.go:789 Counting finalizations {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 5, "votes": 2}
2025-02-27T17:07:07.4102711Z [02-27|16:57:06.211] DEBUG Simplex/epoch.go:518 Received finalization message {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "from": "03", "round": 4}
2025-02-27T17:07:07.4103308Z [02-27|16:57:06.211] DEBUG Simplex/monitor.go:81 Task executed {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "taskID": 3}
2025-02-27T17:07:07.4103854Z [02-27|16:57:06.211] DEBUG Simplex/notarization.go:87 Collected finalization from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "01", "round": 4}
2025-02-27T17:07:07.4104392Z [02-27|16:57:06.211] DEBUG Simplex/notarization.go:87 Collected finalization from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "02", "round": 4}
2025-02-27T17:07:07.4104928Z [02-27|16:57:06.211] DEBUG Simplex/notarization.go:87 Collected finalization from node {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "NodeID": "03", "round": 4}
2025-02-27T17:07:07.4105448Z [02-27|16:57:06.211] DEBUG Simplex/epoch.go:836 Received enough finalizations to finalize a block {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 4}
2025-02-27T17:07:07.4105992Z [02-27|16:57:06.211] INFO Simplex/epoch.go:913 Committed block {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 4, "sequence": 3, "digest": "fad5b74a829a60610bff..."}
2025-02-27T17:07:07.4106449Z [02-27|16:57:06.211] DEBUG Simplex/epoch.go:829 Progressing rounds due to commit {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 5}
2025-02-27T17:07:07.4107032Z [02-27|16:57:06.211] DEBUG Simplex/epoch.go:864 Broadcast finalization certificate {"test": "TestEpochLeaderFailoverWithEmptyNotarization", "node": 1, "round": 4, "digest": "fad5b74a829a60610bff..."}
2025-02-27T17:07:07.4107756Z [02-27|16:57:06.212] INFO Simplex/epoch.go:207 Block Proposal Recovered From WAL {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "Round": 0}
2025-02-27T17:07:07.4108396Z [02-27|16:57:06.212] DEBUG Simplex/sched.go:95 No ready tasks, going to sleep {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1}
2025-02-27T17:07:07.4109103Z [02-27|16:57:06.212] INFO Simplex/epoch.go:221 Notarization Recovered From WAL {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "Round": 0}
2025-02-27T17:07:07.4109804Z [02-27|16:57:06.212] INFO Simplex/epoch.go:207 Block Proposal Recovered From WAL {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "Round": 1}
2025-02-27T17:07:07.4110586Z [02-27|16:57:06.212] INFO Simplex/epoch.go:221 Notarization Recovered From WAL {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "Round": 1}
2025-02-27T17:07:07.4111341Z [02-27|16:57:06.212] INFO Simplex/epoch.go:207 Block Proposal Recovered From WAL {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "Round": 2}
2025-02-27T17:07:07.4112044Z [02-27|16:57:06.212] INFO Simplex/epoch.go:221 Notarization Recovered From WAL {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "Round": 2}
2025-02-27T17:07:07.4112734Z [02-27|16:57:06.212] DEBUG Simplex/epoch.go:1564 Scheduling block building {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "round": 4}
2025-02-27T17:07:07.4113723Z [02-27|16:57:06.212] DEBUG Simplex/sched.go:137 Scheduling new ready task {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "dependency": "39428085912723bd2fe2..."}
2025-02-27T17:07:07.4114541Z [02-27|16:57:06.212] DEBUG Simplex/epoch.go:651 Received vote message {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "from": "02", "round": 4, "digest": "fad5b74a829a60610bff..."}
2025-02-27T17:07:07.4115314Z [02-27|16:57:06.212] DEBUG Simplex/epoch.go:673 Received a vote for the current round {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "round": 4, "from": "02"}
2025-02-27T17:07:07.4115985Z [02-27|16:57:06.212] DEBUG Simplex/sched.go:97 Woken up from sleep {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "ready tasks": 1}
2025-02-27T17:07:07.4116695Z [02-27|16:57:06.212] DEBUG Simplex/sched.go:107 Running task {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "remaining ready tasks": 0}
2025-02-27T17:07:07.4117510Z [02-27|16:57:06.212] DEBUG Simplex/epoch.go:651 Received vote message {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "from": "03", "round": 4, "digest": "fad5b74a829a60610bff..."}
2025-02-27T17:07:07.4118285Z [02-27|16:57:06.212] DEBUG Simplex/epoch.go:673 Received a vote for the current round {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "round": 4, "from": "03"}
2025-02-27T17:07:07.4119081Z [02-27|16:57:06.212] DEBUG Simplex/epoch.go:1597 Wrote block to WAL {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "round": 4, "size": 125, "digest": "1b0aab43dbc712af854e..."}
2025-02-27T17:07:07.4119905Z [02-27|16:57:06.212] DEBUG Simplex/epoch.go:651 Received vote message {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "from": "03", "round": 4, "digest": "fad5b74a829a60610bff..."}
2025-02-27T17:07:07.4120651Z [02-27|16:57:06.212] DEBUG Simplex/epoch.go:1015 Counting votes {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "round": 4, "votes": 1, "from": "[03]"}
2025-02-27T17:07:07.4121452Z [02-27|16:57:06.213] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "from": "01", "round": 4}
2025-02-27T17:07:07.4122266Z [02-27|16:57:06.213] DEBUG Simplex/epoch.go:651 Received vote message {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "from": "02", "round": 4, "digest": "fad5b74a829a60610bff..."}
2025-02-27T17:07:07.4123163Z [02-27|16:57:06.213] DEBUG Simplex/epoch.go:1015 Counting votes {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "round": 4, "votes": 2, "from": "[03 02]"}
2025-02-27T17:07:07.4124039Z [02-27|16:57:06.213] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "from": "01", "round": 4}
2025-02-27T17:07:07.4124824Z [02-27|16:57:06.213] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "from": "04", "round": 4}
2025-02-27T17:07:07.4125719Z [02-27|16:57:06.213] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "from": "04", "round": 4}
2025-02-27T17:07:07.4126502Z [02-27|16:57:06.213] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "from": "04", "round": 4}
2025-02-27T17:07:07.4127289Z [02-27|16:57:06.213] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "from": "01", "round": 4}
2025-02-27T17:07:07.4128104Z [02-27|16:57:06.213] DEBUG Simplex/epoch.go:1619 Proposal broadcast {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "round": 4, "size": 125, "digest": "1b0aab43dbc712af854e..."}
2025-02-27T17:07:07.4128927Z [02-27|16:57:06.213] DEBUG Simplex/epoch.go:651 Received vote message {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "from": "01", "round": 4, "digest": "1b0aab43dbc712af854e..."}
2025-02-27T17:07:07.4129836Z [02-27|16:57:06.213] WARN Simplex/epoch.go:1036 Counting votes for the digest we received from the leader {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "round": 4, "voteForOurDigests": 1, "total votes": 3}
2025-02-27T17:07:07.4130622Z [02-27|16:57:06.213] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "from": "01", "round": 4}
2025-02-27T17:07:07.4131398Z [02-27|16:57:06.213] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "from": "04", "round": 4}
2025-02-27T17:07:07.4132182Z [02-27|16:57:06.213] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "from": "04", "round": 4}
2025-02-27T17:07:07.4133056Z [02-27|16:57:06.213] DEBUG Simplex/epoch.go:1920 No future messages received for this round {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "from": "01", "round": 4}
2025-02-27T17:07:07.4133833Z [02-27|16:57:06.213] DEBUG Simplex/sched.go:109 Task finished execution {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "taskID": "1b0aab43dbc712af854e..."}
2025-02-27T17:07:07.4134590Z [02-27|16:57:06.213] DEBUG Simplex/epoch.go:518 Received finalization message {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "from": "02", "round": 4}
2025-02-27T17:07:07.4135405Z [02-27|16:57:06.213] DEBUG Simplex/sched.go:114 Enqueued newly ready tasks {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "number of ready tasks": 0}
2025-02-27T17:07:07.4136099Z [02-27|16:57:06.213] DEBUG Simplex/sched.go:95 No ready tasks, going to sleep {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1}
2025-02-27T17:07:07.4136835Z [02-27|16:57:06.213] DEBUG Simplex/epoch.go:789 Counting finalizations {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "round": 4, "votes": 1}
2025-02-27T17:07:07.4137649Z [02-27|16:57:06.213] DEBUG Simplex/epoch.go:518 Received finalization message {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "from": "03", "round": 4}
2025-02-27T17:07:07.4138435Z [02-27|16:57:06.213] DEBUG Simplex/epoch.go:789 Counting finalizations {"test": "TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash", "node": 1, "round": 4, "votes": 2}
2025-02-27T17:07:07.4138544Z panic: test timed out after 10m0s
2025-02-27T17:07:07.4138639Z running tests:
2025-02-27T17:07:07.4138840Z TestEpochLeaderFailoverWithEmptyNotarization (9m59s)
2025-02-27T17:07:07.4139307Z TestEpochLeaderFailoverWithEmptyNotarization/TestEpochLeaderFailoverWithEmptyNotarization-with-crash (9m59s)
2025-02-27T17:07:07.4139314Z
2025-02-27T17:07:07.4139405Z goroutine 62 [running]:
2025-02-27T17:07:07.4139506Z testing.(*M).startAlarm.func1()
2025-02-27T17:07:07.4139741Z /opt/hostedtoolcache/go/1.23.6/x64/src/testing/testing.go:2373 +0x265
2025-02-27T17:07:07.4139837Z created by time.goFunc
2025-02-27T17:07:07.4140038Z /opt/hostedtoolcache/go/1.23.6/x64/src/time/sleep.go:215 +0x45
2025-02-27T17:07:07.4140043Z
2025-02-27T17:07:07.4140149Z goroutine 1 [chan receive, 9 minutes]:
2025-02-27T17:07:07.4140305Z testing.(*T).Run(0xc0000de9c0, {0x949171, 0x2c}, 0x969ff8)
2025-02-27T17:07:07.4140523Z /opt/hostedtoolcache/go/1.23.6/x64/src/testing/testing.go:1751 +0x851
2025-02-27T17:07:07.4140631Z testing.runTests.func1(0xc0000de9c0)
2025-02-27T17:07:07.4140848Z /opt/hostedtoolcache/go/1.23.6/x64/src/testing/testing.go:2168 +0x86
2025-02-27T17:07:07.4140961Z testing.tRunner(0xc0000de9c0, 0xc0000edae0)
2025-02-27T17:07:07.4141176Z /opt/hostedtoolcache/go/1.23.6/x64/src/testing/testing.go:1690 +0x227
2025-02-27T17:07:07.4141427Z testing.runTests(0xc0000b81b0, {0xc09d00, 0x30, 0x30}, {0x7f75b59b05b8?, 0x40?, 0xc0fd40?})
2025-02-27T17:07:07.4141639Z /opt/hostedtoolcache/go/1.23.6/x64/src/testing/testing.go:2166 +0x8bf
2025-02-27T17:07:07.4141738Z testing.(*M).Run(0xc0000aee60)
2025-02-27T17:07:07.4141946Z /opt/hostedtoolcache/go/1.23.6/x64/src/testing/testing.go:2034 +0xf18
2025-02-27T17:07:07.4142025Z main.main()
2025-02-27T17:07:07.4142126Z _testmain.go:149 +0x165
2025-02-27T17:07:07.4142130Z
2025-02-27T17:07:07.4142233Z goroutine 26 [select, 9 minutes]:
2025-02-27T17:07:07.4142338Z simplex.(*Monitor).run(0xc0000da2d0)
2025-02-27T17:07:07.4142514Z /home/runner/work/Simplex/Simplex/monitor.go:75 +0x169
2025-02-27T17:07:07.4142639Z created by simplex.NewMonitor in goroutine 25
2025-02-27T17:07:07.4142804Z /home/runner/work/Simplex/Simplex/monitor.go:38 +0x25d
2025-02-27T17:07:07.4142814Z
2025-02-27T17:07:07.4143011Z goroutine 55 [chan receive, 9 minutes]:
2025-02-27T17:07:07.4143179Z testing.(*T).Run(0xc00007e000, {0xc00001a6c0, 0x37}, 0xc000562700)
2025-02-27T17:07:07.4143398Z /opt/hostedtoolcache/go/1.23.6/x64/src/testing/testing.go:1751 +0x851
2025-02-27T17:07:07.4143780Z simplex_test.runCrashAndRestartExecution(0xc00007e000, 0xc000104380, 0xc000013038, 0xc00027e1e0, 0xc00028c050, 0xc000485c80)
2025-02-27T17:07:07.4144004Z /home/runner/work/Simplex/Simplex/epoch_failover_test.go:785 +0x905
2025-02-27T17:07:07.4144235Z simplex_test.TestEpochLeaderFailoverWithEmptyNotarization(0xc00007e000)
2025-02-27T17:07:07.4144520Z /home/runner/work/Simplex/Simplex/epoch_failover_test.go:121 +0x172d
2025-02-27T17:07:07.4144626Z testing.tRunner(0xc00007e000, 0x969ff8)
2025-02-27T17:07:07.4144834Z /opt/hostedtoolcache/go/1.23.6/x64/src/testing/testing.go:1690 +0x227
2025-02-27T17:07:07.4145023Z created by testing.(*T).Run in goroutine 1
2025-02-27T17:07:07.4145236Z /opt/hostedtoolcache/go/1.23.6/x64/src/testing/testing.go:1743 +0x826
2025-02-27T17:07:07.4145241Z
2025-02-27T17:07:07.4145344Z goroutine 30 [select, 9 minutes]:
2025-02-27T17:07:07.4145444Z simplex.(*Monitor).run(0xc0000da320)
2025-02-27T17:07:07.4145614Z /home/runner/work/Simplex/Simplex/monitor.go:75 +0x169
2025-02-27T17:07:07.4145739Z created by simplex.NewMonitor in goroutine 29
2025-02-27T17:07:07.4146018Z /home/runner/work/Simplex/Simplex/monitor.go:38 +0x25d
2025-02-27T17:07:07.4146028Z
2025-02-27T17:07:07.4146130Z goroutine 35 [select, 9 minutes]:
2025-02-27T17:07:07.4146230Z simplex.(*Monitor).run(0xc0001aa000)
2025-02-27T17:07:07.4146404Z /home/runner/work/Simplex/Simplex/monitor.go:75 +0x169
2025-02-27T17:07:07.4146533Z created by simplex.NewMonitor in goroutine 34
2025-02-27T17:07:07.4146702Z /home/runner/work/Simplex/Simplex/monitor.go:38 +0x25d
2025-02-27T17:07:07.4146707Z
2025-02-27T17:07:07.4146806Z goroutine 6 [select, 9 minutes]:
2025-02-27T17:07:07.4146908Z simplex.(*Monitor).run(0xc00028c000)
2025-02-27T17:07:07.4147067Z /home/runner/work/Simplex/Simplex/monitor.go:75 +0x169
2025-02-27T17:07:07.4147225Z created by simplex.NewMonitor in goroutine 5
2025-02-27T17:07:07.4147517Z /home/runner/work/Simplex/Simplex/monitor.go:38 +0x25d
2025-02-27T17:07:07.4147527Z
2025-02-27T17:07:07.4147698Z goroutine 56 [sync.Cond.Wait, 9 minutes]:
2025-02-27T17:07:07.4147839Z sync.runtime_notifyListWait(0xc000290028, 0x3)
2025-02-27T17:07:07.4148054Z /opt/hostedtoolcache/go/1.23.6/x64/src/runtime/sema.go:587 +0x159
2025-02-27T17:07:07.4148155Z sync.(*Cond).Wait(0xc000290018)
2025-02-27T17:07:07.4148337Z /opt/hostedtoolcache/go/1.23.6/x64/src/sync/cond.go:71 +0x75
2025-02-27T17:07:07.4148445Z simplex.(*scheduler).run(0xc000290000)
2025-02-27T17:07:07.4148607Z /home/runner/work/Simplex/Simplex/sched.go:96 +0x159
2025-02-27T17:07:07.4148739Z created by simplex.NewScheduler in goroutine 55
2025-02-27T17:07:07.4148902Z /home/runner/work/Simplex/Simplex/sched.go:28 +0x225
2025-02-27T17:07:07.4148907Z
2025-02-27T17:07:07.4149004Z goroutine 57 [select, 9 minutes]:
2025-02-27T17:07:07.4149110Z simplex.(*Monitor).run(0xc00028c0a0)
2025-02-27T17:07:07.4149272Z /home/runner/work/Simplex/Simplex/monitor.go:75 +0x169
2025-02-27T17:07:07.4149398Z created by simplex.NewMonitor in goroutine 55
2025-02-27T17:07:07.4149562Z /home/runner/work/Simplex/Simplex/monitor.go:38 +0x25d
2025-02-27T17:07:07.4149570Z
2025-02-27T17:07:07.4149678Z goroutine 59 [sync.Cond.Wait, 9 minutes]:
2025-02-27T17:07:07.4149810Z sync.runtime_notifyListWait(0xc00028c520, 0x0)
2025-02-27T17:07:07.4150011Z /opt/hostedtoolcache/go/1.23.6/x64/src/runtime/sema.go:587 +0x159
2025-02-27T17:07:07.4150111Z sync.(*Cond).Wait(0xc00028c510)
2025-02-27T17:07:07.4150292Z /opt/hostedtoolcache/go/1.23.6/x64/src/sync/cond.go:71 +0x75
2025-02-27T17:07:07.4150474Z simplex_test.(*InMemStorage).waitForBlockCommit(0xc00028c500, 0x3)
2025-02-27T17:07:07.4150662Z /home/runner/work/Simplex/Simplex/epoch_test.go:950 +0x10f
2025-02-27T17:07:07.4151085Z simplex_test.notarizeAndFinalizeRound(0xc00007e4e0, {0xc00027e180, 0x4, 0x4b7289?}, 0x4, 0x3, 0xc0001048c0, 0xc0005df080, 0x3, 0xc00028c500, ...)
2025-02-27T17:07:07.4151271Z /home/runner/work/Simplex/Simplex/epoch_test.go:350 +0x83a
2025-02-27T17:07:07.4151692Z simplex_test.TestEpochLeaderFailoverWithEmptyNotarization.func1(0xc00007e4e0, 0xc0001048c0, 0xc0005df080, 0xc00028c500, 0x0?)
2025-02-27T17:07:07.4151916Z /home/runner/work/Simplex/Simplex/epoch_failover_test.go:123 +0xd4
2025-02-27T17:07:07.4152094Z simplex_test.runCrashAndRestartExecution.func2(0xc00007e4e0)
2025-02-27T17:07:07.4152307Z /home/runner/work/Simplex/Simplex/epoch_failover_test.go:806 +0x4ba
2025-02-27T17:07:07.4152484Z testing.tRunner(0xc00007e4e0, 0xc000562700)
2025-02-27T17:07:07.4152705Z /opt/hostedtoolcache/go/1.23.6/x64/src/testing/testing.go:1690 +0x227
2025-02-27T17:07:07.4152820Z created by testing.(*T).Run in goroutine 55
2025-02-27T17:07:07.4153312Z /opt/hostedtoolcache/go/1.23.6/x64/src/testing/testing.go:1743 +0x826
2025-02-27T17:07:07.4153320Z
2025-02-27T17:07:07.4153452Z goroutine 60 [sync.Cond.Wait, 9 minutes]:
2025-02-27T17:07:07.4153584Z sync.runtime_notifyListWait(0xc0001e6a28, 0x1)
2025-02-27T17:07:07.4153802Z /opt/hostedtoolcache/go/1.23.6/x64/src/runtime/sema.go:587 +0x159
2025-02-27T17:07:07.4153900Z sync.(*Cond).Wait(0xc0001e6a18)
2025-02-27T17:07:07.4154098Z /opt/hostedtoolcache/go/1.23.6/x64/src/sync/cond.go:71 +0x75
2025-02-27T17:07:07.4154327Z simplex.(*scheduler).run(0xc0001e6a00)
2025-02-27T17:07:07.4154498Z /home/runner/work/Simplex/Simplex/sched.go:96 +0x159
2025-02-27T17:07:07.4154641Z created by simplex.NewScheduler in goroutine 59
2025-02-27T17:07:07.4154803Z /home/runner/work/Simplex/Simplex/sched.go:28 +0x225
2025-02-27T17:07:07.4154808Z
2025-02-27T17:07:07.4154914Z goroutine 61 [select, 9 minutes]:
2025-02-27T17:07:07.4155014Z simplex.(*Monitor).run(0xc00028c690)
2025-02-27T17:07:07.4155182Z /home/runner/work/Simplex/Simplex/monitor.go:75 +0x169
2025-02-27T17:07:07.4155315Z created by simplex.NewMonitor in goroutine 59
2025-02-27T17:07:07.4155477Z /home/runner/work/Simplex/Simplex/monitor.go:38 +0x25d
2025-02-27T17:07:07.4155569Z FAIL simplex 600.053s
2025-02-27T17:07:07.4155656Z ok simplex/wal 1.032s
2025-02-27T17:07:07.4155724Z FAIL
2025-02-27T17:07:07.4169263Z ##[error]Process completed with exit code 1.
2025-02-27T17:07:07.4275796Z Post job cleanup.
2025-02-27T17:07:07.5207953Z [command]/usr/bin/git version