Invalid Block on Hoodi

[ahshum]

Describe the bug
Invalid block happened on Hoodi at slot 1316437 (https://light-hoodi.beaconcha.in/slot/0x50e203e7f83ffd3148549f63c23d8795281a7488c4550c23917e2a81cd3cd64d)

To Reproduce
Not able to reproduce but the behavior was consistent across different Hoodi nodes at the slot 1316437.

Logs from one of the node service.20250916_0900.log.gz

There were reorgs around the slot.

{"lvl":"INF","ts":"2025-09-16 08:17:33.700+00:00","msg":"State replayed","topics":"chaindag","blocks":5,"slots":53,"current":"5ce3ac5c:1316436@1316437","ancestor":"26415d97:1316383@1316384","target":"2f4cd453:1316388@1316437","ancestorStateRoot":"d844b4f7","targetStateRoot":"481cbdf1","found":false,"assignDur":"128ms69us124ns","replayDur":"2s519ms293us436ns"}
{"lvl":"INF","ts":"2025-09-16 08:17:35.648+00:00","msg":"State replayed","topics":"chaindag","blocks":21,"slots":22,"current":"50e203e7:1316437","ancestor":"73f226fb:1316415@1316416","target":"5ce3ac5c:1316436@1316438","ancestorStateRoot":"46099666","targetStateRoot":"c3832423","found":false,"assignDur":"127ms532us506ns","replayDur":"513ms109us371ns"}

Screenshots

Additional context
From the logs there are two reorgs happened around slot 1316437. The transition is working fine by manually trigger the reorg between slot 1316436 and 1316438 and insert the block 1316437. With the patch

diff --git a/ncli/ncli_db.nim b/ncli/ncli_db.nim
index 41ac58164..59e78f31d 100644
--- a/ncli/ncli_db.nim
+++ b/ncli/ncli_db.nim
@@ -312,6 +312,14 @@ proc cmdBench(conf: DbConf, cfg: RuntimeConfig) =
       false, cache, dag.updateFlags)
 
   template processBlocks(blocks: auto) =
+    let
+      t1blck =
+        try:
+          readSszForkedSignedBeaconBlock(
+            cfg, readAllBytes("./block-1316437-50e203e7f83ffd3148549f63c23d8795281a7488c4550c23917e2a81cd3cd64d.ssz").tryGet())
+        except CatchableError:
+          raiseAssert "error reading signed beacon block"
+
     for b in blocks.mitems():
       if shouldShutDown: quit QuitSuccess
       while getStateField(stateData[], slot) < b.message.slot:
@@ -337,6 +345,37 @@ proc cmdBench(conf: DbConf, cfg: RuntimeConfig) =
         withTimer(timers[tDbStore]):
           dbBenchmark.putBlock(b)
 
+      # after slot
+      withState(stateData[]):
+
+        if b.message.slot == 1316436.Slot:
+          let obsi = getBlockIdAtSlot(forkyState, 1316436.Slot)
+
+          var cache2: StateCache
+          process_slots(
+            dag.cfg, stateData[], 1316437.Slot, cache2,
+            info, {}).expect("Slot processing can't fail with correct inputs")
+
+          dag.withUpdatedState(stateData[],
+            BlockSlotId(
+              bid: BlockId(slot: 1316388.Slot, root: t1blck.electraData.message.parent_root),
+              slot: 1316437.Slot)) do:
+            debugEcho "reorg done 1316388@1316437"
+          do:
+            debugEcho "reorg failed 1316388@1316437"
+
+          let res = state_transition_block(dag.cfg, stateData[], t1blck.electraData, cache2, {}, noRollback)
+          if res.isErr:
+            quit 1
+
+          dag.withUpdatedState(stateData[],
+            BlockSlotId(
+              bid: obsi.get.bid,
+              slot: 1316438.Slot)) do:
+            debugEcho "reorg done 1316436@1316438"
+          do:
+            debugEcho "reorg failed 1316436@1316438"
+
       withState(stateData[]):
         if forkyState.data.slot.is_epoch and conf.storeStates:
           if forkyState.data.slot.epoch < 2:

Also, in order to reproduce the same reorg as in the logs, one will have to modify nimbus_beacon_node to disable state prune or otherwise only every 32 epoch of state data are saved.