node stopped getting new blocks at 7083588

[Ajes1337]

Node stopped getting new blocks at 7083588 (Wed Feb 26 2025 14:59:59 GMT+0000 (UTC)

I did upgrade the docker images with the newest in v.1.2.0 a day before needed (I triple checked everything).

We are atleast 3-4 dudes on Ink's dev-general channel on discord, that experienced the same problem.

I have tried restarting docker compose, also completely removing the instance with "docker compose down -v" and let it resync from scratch with and without the rock torrent download thingy. even with other settings than archive, like full and snap (snap is still syncing though) - but archive, after it has finished downloading and begun to catch up, it always keeps around ~43329 blocks behind the official top.

I have tried different L1-rpc's for execution and beacon, usually I use my own local hosted (nethermind, reth, lighthouse), but I have also tried online services like chainstack for a full unpruned beacon node.

software: ubuntu 22 lxc-container on proxmox, and followed the guide on this github.

errors:

protocols not supported: [/opstack/req/payload_by_number/57073/0]

healthcheck-1   | {"level":30,"time":1740640127229,"targetHeight":7098275,"referenceHeight":7141714,"heightDifference":43439,"minBlockNumber":7098275,"msg":"latest block heights"}

healthcheck-1   | {"level":50,"time":1740640127448,"extra":{"blockNumber":7098275,"referenceHash":"0xeafdc9982d3aef8529fdfe6a01fdd58527cdcbc20f0c612d6e5a276ab0264577","targetHash":"0xb7567a7e6907d1767778d043f2c80f0b95eaf7b5ebfbf816e4fea06fe8798224"},"msg":"reference client has different hash for block"}

healthcheck-1   | {"level":30,"time":1740640127448,"msg":"beginning binary search to find first mismatched block"}

op-node-1       | t=2025-02-27T08:02:38+0000 lvl=warn msg="L1 head signal indicates a possible L1 re-org" old_l1_head=0x2a8eefb8c0d5c16116fe3778e0ab44a85b27594c9f252104d8cc5a1656420ed3:21936344 new_l1_head_parent=0x3a56745e071a91f12c88ce52c919dc4d43a6cd49da6ef3fee81fe2b48d086708 new_l1_head=0x91fec02644b1bc1bd05a15543254ce283d360a40de455fd9e4e7eb5c7cec669c:21936346

op-node-1       | t=2025-02-27T08:02:40+0000 lvl=warn msg="dropping future span batch" origin=0x52c65466f01c6181e1bbf61285390818c8b1b19d55fa0a3ec86f8c7d87692358:21934660 epoch=0x74bae72d1bc891c1cf4ee2a7f4bc42b07767d3ffef2f43ba1b22c882bae4f60a:21932561 batch_type=SpanBatch batch_timestamp=1740617012 parent_check=0xe5afc7ab0f1e97e4ba9a53684b76fe915d541938 origin_check=0x8f42a6f7d0b0c3c66d194bd0d623221a932c04ea start_epoch_number=21934159 end_epoch_number=21934160 block_count=10 txs=0 next_timestamp=1740597671

op-node-1       | t=2025-02-27T08:02:40+0000 lvl=warn msg="Dropping invalid span batch, flushing channel" origin=0x52c65466f01c6181e1bbf61285390818c8b1b19d55fa0a3ec86f8c7d87692358:21934660 epoch=0x74bae72d1bc891c1cf4ee2a7f4bc42b07767d3ffef2f43ba1b22c882bae4f60a:21932561 batch_type=SpanBatch batch_timestamp=1740617012 parent_check=0xe5afc7ab0f1e97e4ba9a53684b76fe915d541938 origin_check=0x8f42a6f7d0b0c3c66d194bd0d623221a932c04ea start_epoch_number=21934159 end_epoch_number=21934160 block_count=10 txs=0

op-node-1       | t=2025-02-26T15:12:39+0000 lvl=warn msg="Dropping invalid unsafe payload" hash=0xe614f2437ad350142a3a461e2b299579efd00a650b74db3cf86ed87a5f3482aa number=7083589 timestamp=1740582000

op-node-1       | t=2025-02-26T15:12:39+0000 lvl=warn msg="Payload was invalid" block=0xe614f2437ad350142a3a461e2b299579efd00a650b74db3cf86ed87a5f3482aa:7083589 err="execution payload 0xe614f2437ad350142a3a461e2b299579efd00a650b74db3cf86ed87a5f3482aa:7083589 was INVALID! Latest valid hash is <nil>, ignoring bad block: 0xc011d16080" timestamp=1740582000

op-node-1       | t=2025-02-26T15:12:39+0000 lvl=warn msg="Engine temporary error" err="temp: cannot process unsafe payload: new - 0xe614f2437ad350142a3a461e2b299579efd00a650b74db3cf86ed87a5f3482aa:7083589; parent: 0x8362847a2fbd7e0291892836c1761af953ad3979abd2e3d9ca1972e387bae502:7083588; err: execution payload 0xe614f2437ad350142a3a461e2b299579efd00a650b74db3cf86ed87a5f3482aa:7083589 was INVALID! Latest valid hash is <nil>, ignoring bad block: 0xc011d16080"

[algotradingstrategies]

I see the same.
For full node (using snap sync). I am using quicknode as L1.

It started exactly at 2025-02-26T15:00:00+0000 (which interestingly is mentioned in the release note of v1.2 here ).

Specificially, I keep seeing lines like this in my log

437ad350142a3a461e2b299579efd00a650b74db3cf86ed87a5f3482aa:7083589 was INVALID! Latest valid hash is , ignoring bad block: 0xc010fa2c20" ```

510a14c1f91d79496c617b22353db30cd6b28c9012a6012ba83a2de34cd716:7056600 l2_pending_safe=0xce510a14c1f91d79496c617b22353db30cd6b28c9012a6012ba83a2de34cd716:7056600 l2_unsafe=0x8362847a2fbd7e0291892836c1761af
953ad3979abd2e3d9ca1972e387bae502:7083588 l2_backup_unsafe=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_time=1740581999
op-node-1  | t=2025-02-26T14:59:58+0000 lvl=info msg="successfully processed payload" ref=0x8362847a2fbd7e0291892836c1761af953ad3979abd2e3d9ca1972e387bae502:7083588 txs=2
op-node-1  | t=2025-02-26T15:00:00+0000 lvl=info msg="Received signed execution payload from p2p" id=0xe614f2437ad350142a3a461e2b299579efd00a650b74db3cf86ed87a5f3482aa:708358 peer=16Uiu2HAmRoLyUVcFJ 3BPwLuxB7qEYRGUjSYwJsnuMJEq6oGx1vnp txs=2
op-node-1  | t=2025-02-26T15:00:00+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xe614f2437ad350142a3a461e2b299579efd00a650b74db3cf86ed87a5f3482aa:7083589
op-node-1  | t=2025-02-26T15:00:00+0000 lvl=info msg="failed to insert payload" ref=0xe614f2437ad350142a3a461e2b299579efd00a650b74db3cf86ed87a5f3482aa:7083589 txs=2 err="temp: cannot process unsafe p
ayload: new - 0xe614f2437ad350142a3a461e2b299579efd00a650b74db3cf86ed87a5f3482aa:7083589; parent: 0x8362847a2fbd7e0291892836c1761af953ad3979abd2e3d9ca1972e387bae502:7083588; err: execution payload 0xe614f2
437ad350142a3a461e2b299579efd00a650b74db3cf86ed87a5f3482aa:7083589 was INVALID! Latest valid hash is <nil>, ignoring bad block: 0xc010fa2c20"                            ```

Restart or resync (snapsync) all not helping/working.