Ontology Testnet not syncing past block 16613129 #1410
Description
What version of ontology are you using (ontology --version)?
v2.3.7
Does this issue reproduce with the latest release?
Yes
What operating system are you using (such as macOS, Linux and Windows)?
Linux
What did you do?
Started up a testnet validator that was syncing but stopped at block 16613129.
What did you expect to see?
Expected to the validator continue syncing to the top of the chain (current at around 16.8 million)
What did you see instead?
Error (level: WARN) message:
saveBlock Height:16613130 AddBlock error:saveBlock error state merkle root mismatch. expected: d20081eb949e3d9e5c57347c0a5b3e083173821032519d215dfa57ed1a61a283, got: bb847547558081ae2929f9f376cd9d4bd20ba652f903950a768ee44138dc8ba7
Latest logs:
2022/06/13 19:06:09.286355 [INFO ] GID 2600, [block-sync] delete node: 9eddec6d3b4a0214cf453d10075fe9beffc98d00
2022/06/13 19:06:09.286369 [WARN ] GID 2600, [block-sync] saveBlock Height:16613130 AddBlock error:saveBlock error state merkle root mismatch. expected: d20081eb949e3d9e5c57347c0a5b3e083173821032519d215dfa57ed1a61a283, got: bb847547558081ae2929f9f376cd9d4bd20ba652f903950a768ee44138dc8ba7
2022/06/13 19:06:10.475977 [INFO ] GID 204, server 4294967295 statemgr update, current state: 1, from peer: 9, peercnt: 7, v1: 3524, v2: 3499
2022/06/13 19:06:11.430156 [INFO ] GID 411, [p2p]error read from 54.234.1.252:20338 :EOF
2022/06/13 19:06:11.430213 [INFO ] GID 411, remove peer 000000000000000000000000eb3bcf45d8f11514 from net server
2022/06/13 19:06:11.430251 [INFO ] GID 411, closing connection: peer 000000000000000000000000eb3bcf45d8f11514, address: 54.234.1.252:20338
2022/06/13 19:06:11.430332 [INFO ] GID 2633, [block-sync] delete node: 000000000000000000000000eb3bcf45d8f11514
2022/06/13 19:06:11.430436 [INFO ] GID 2632, inbound peer 54.234.1.252:46922 connected, id=000000000000000000000000eb3bcf45d8f11514, version=v1.9.0-0-g1d4e4f57
2022/06/13 19:06:11.894141 [INFO ] GID 204, server 4294967295 statemgr update, current state: 1, from peer: 7, peercnt: 7, v1: 3524, v2: 3499
2022/06/13 19:06:12.349278 [INFO ] GID 298, CurrentBlockHeight = 16613129
2022/06/13 19:06:17.835968 [INFO ] GID 204, server 4294967295 statemgr update, current state: 1, from peer: 3, peercnt: 7, v1: 3524, v2: 3499
2022/06/13 19:06:18.343362 [INFO ] GID 204, server 4294967295 statemgr update, current state: 1, from peer: 5, peercnt: 7, v1: 3524, v2: 3499
2022/06/13 19:06:18.349275 [INFO ] GID 298, CurrentBlockHeight = 16613129
2022/06/13 19:06:19.835990 [INFO ] GID 204, server 4294967295 statemgr update, current state: 1, from peer: 1, peercnt: 7, v1: 3524, v2: 3499
2022/06/13 19:06:21.555548 [INFO ] GID 204, server 4294967295 statemgr update, current state: 1, from peer: 6, peercnt: 7, v1: 3524, v2: 3499
2022/06/13 19:06:21.585138 [INFO ] GID 204, server 4294967295 statemgr update, current state: 1, from peer: 4, peercnt: 7, v1: 3524, v2: 3499
2022/06/13 19:06:24.349246 [INFO ] GID 298, CurrentBlockHeight = 16613129
2022/06/13 19:06:30.349290 [INFO ] GID 298, CurrentBlockHeight = 16613129
2022/06/13 19:06:30.476282 [INFO ] GID 204, server 4294967295 statemgr update, current state: 1, from peer: 9, peercnt: 7, v1: 3524, v2: 3499
2022/06/13 19:06:31.894060 [INFO ] GID 204, server 4294967295 statemgr update, current state: 1, from peer: 7, peercnt: 7, v1: 3524, v2: 3499
2022/06/13 19:06:36.349284 [INFO ] GID 298, CurrentBlockHeight = 16613129
2022/06/13 19:06:37.836099 [INFO ] GID 204, server 4294967295 statemgr update, current state: 1, from peer: 3, peercnt: 7, v1: 3524, v2: 3499
2022/06/13 19:06:38.343475 [INFO ] GID 204, server 4294967295 statemgr update, current state: 1, from peer: 5, peercnt: 7, v1: 3524, v2: 3499
2022/06/13 19:06:39.836049 [INFO ] GID 204, server 4294967295 statemgr update, current state: 1, from peer: 1, peercnt: 7, v1: 3524, v2: 3499
2022/06/13 19:06:41.555446 [INFO ] GID 204, server 4294967295 statemgr update, current state: 1, from peer: 6, peercnt: 7, v1: 3524, v2: 3499
2022/06/13 19:06:41.585306 [INFO ] GID 204, server 4294967295 statemgr update, current state: 1, from peer: 4, peercnt: 7, v1: 3524, v2: 3499
2022/06/13 19:06:42.349285 [INFO ] GID 298, CurrentBlockHeight = 16613129
2022/06/13 19:06:45.708095 [INFO ] GID 612, server 4294967295 received consensus msg, blk 16861191, type: 0 from 5
2022/06/13 19:06:45.711400 [INFO ] GID 509, server 4294967295 received consensus msg, blk 16861191, type: 1 from 3
2022/06/13 19:06:45.790831 [INFO ] GID 1269, server 4294967295 received consensus msg, blk 16861191, type: 1 from 9
2022/06/13 19:06:45.792457 [INFO ] GID 1269, server 4294967295 received consensus msg, blk 16861191, type: 2 from 9
...
Additional context
The validator was started at a much lower block height and after syncing with the network for several days it stopped gaining additional block height at block 16613129 with the error (see logs) above.
My observability dashboard shows that the validator stopped gaining block height late 6/10, early 6/11.