Can't fetch some logs in new fork after a reorg (go-ethereum 1.15.7)

[davidtaikocha]

System information

Geth version: ethereum/client-go:v1.15.7
CL client & version: sigp/lighthouse:v7.0.0-beta.5

Actual behaviour

I'm running a Holesky full node, and found that, after a chain reorg, I try to call eth_getLogs to fetch the logs in the new fork, but seems geth can't return the logs at the exact reorged height, it will only return those new logs before / after the reorged height.

[zsfelfoldi]

Hello! Thanks for the report. Just to clarify:

• was your node running during the reorg?
• how many blocks did this reorg affect? exactly in which blocks are the logs missing? is it exactly the reorged section? does the last common ancestor have the logs?

[davidtaikocha]

Hello! Thanks for the report. Just to clarify:

• was your node running during the reorg?
• how many blocks did this reorg affect? exactly in which blocks are the logs missing? is it exactly the reorged section? does the last common ancestor have the logs?

1. Yes it (my Holesky full-node) was running
2. 1 block (Holesky 3,638,778 0x598a9c => 0x846b9)
3. In 3,638,778 the logs are missing
   - I fetched my log from 3,638,778 0x598a9c, but after reorg, I used the same params (start + end) to re-fetch the log, then the log was missing in 0x846b9 ( but it does exist in 0x846b9: https://holesky.etherscan.io/tx/0x8edb0fc8e1023e15ca0dd245e2b59409134ff2eaf6bbb622f3c0086717c94ffd#eventlog )
4. Yes exactly the reorged section
5. Not sure if the last common ancestor has the logs, when it happens again, i can double-check

[davidtaikocha]

And it happened again on my Holesky (geth 1.15.7) node height 3,645,230 (reorg 0fdcea => 6c972f), I kept querying eth_getLogs(address: 0xd3f681bd6b49887a48cc9c9953720903967e9dc0, from: 3,645,226, to: 3,645,234) but the log in 3,645,230 6c972f was always missing in the response ( https://holesky.etherscan.io/tx/0x0854ec7fa93c82ad9f3537fafcdca6c13521c3f74b3ac504258d0c677647188a#eventlog ), however, the logs in 3,645,226 ( https://holesky.etherscan.io/tx/0xfee854b8ee3f3b145919921b811a5c068a044837b2db0451e7696760574bdef3#eventlog ) and 3,645,234 ( https://holesky.etherscan.io/tx/0xa0a2d764f9721636431346e42452cf5eb600411ce38b68731799998cec841241#eventlog ) were in the response.

And at the query side, I was using contrctName.FilterEventName(&bind.FilterOpts{Start: start.Number.Uint64(), End: &endHeight, Context: ctx}, nil) from abigenV1 (geth 1.15.3).

[davidtaikocha]

After I roll-backed to geth 1.15.5, this hasn’t happened for > 24 hours.

[zsfelfoldi]

@davidtaikocha Thanks for the detailed report. If a reorg can break the log index then it is indeed a serious bug. I am currently investigating the issue with a new test tool. The bug did not occur yet to me but it's only been running for 8 hours so I'll keep trying to reproduce it.

[davidtaikocha]

@davidtaikocha Thanks for the detailed report. If a reorg can break the log index then it is indeed a serious bug. I am currently investigating the issue with a new test tool. The bug did not occur yet to me but it's only been running for 8 hours so I'll keep trying to reproduce it.

Thanks! Since our app (L2) relies heavily on the (continues) logs, so atm geth 1.15.7 is unusable to us 🙏 And geth <1.15.6 works fine for us. And based on my previous experience, after running my Holesky 1.15.7 node ~12 - 24 hrs, it will happen.

[zsfelfoldi]

@davidtaikocha Thanks for the detailed report. If a reorg can break the log index then it is indeed a serious bug. I am currently investigating the issue with a new test tool. The bug did not occur yet to me but it's only been running for 8 hours so I'll keep trying to reproduce it.

Thanks! Since our app (L2) relies heavily on the (continues) logs, so atm geth 1.15.7 is unusable to us 🙏 And geth <1.15.6 works fine for us. And based on my previous experience, after running my Holesky 1.15.7 node ~12 - 24 hrs, it will happen.

Just one more question: are you running Geth with an external beacon node or are you using the internal beacon light client (--beacon.api flag)? (does not directly affect the indexer but might affect the occurence of reorgs)

[davidtaikocha]

@davidtaikocha Thanks for the detailed report. If a reorg can break the log index then it is indeed a serious bug. I am currently investigating the issue with a new test tool. The bug did not occur yet to me but it's only been running for 8 hours so I'll keep trying to reproduce it.

Thanks! Since our app (L2) relies heavily on the (continues) logs, so atm geth 1.15.7 is unusable to us 🙏 And geth <1.15.6 works fine for us. And based on my previous experience, after running my Holesky 1.15.7 node ~12 - 24 hrs, it will happen.

Just one more question: are you running Geth with an external beacon node or are you using the internal beacon light client (--beacon.api flag)? (does not directly affect the indexer but might affect the occurence of reorgs)

I was running my Holesky geth with a beacon node: sigp/lighthouse:v7.0.0-beta.5, --beacon.api flag was not used.