Prsym repeatedly reporting error processing deposit logs

[MattTW]

Describe the bug

Every couple of minutes, Prysm beacon-chain issues this error message:

level=error msg="Unable to process past deposit contract logs, perhaps your execution client is not fully synced" error="processPastLogs: Could not process deposit log: received incorrect merkle index: wanted 2023989 but got 2023990" prefix=execution

Geth is fully synched and has been for quite some time.

A couple of other people in the Prysm discord are reporting the same error currently appearing for them as well, referencing the same index values. This seems to suggest a broader issue and not local to a specific node/setup.

Has this worked before in a previous version?

🔬 Minimal Reproduction

No response

Error

Platform(s)

Linux (x86)

What version of Prysm are you running? (Which release)

5.3.2

Anything else relevant (validator index / public key)?

No response

[Jor-Tech]

Confirming same issue here:

Could not process deposit log: received incorrect merkle index: wanted 2023989 but got 2023990"

And additionally the following info message where the number in difference keeps increasing:

Falling back to historical headers and logs sync. Current difference is 15387"

[KevinVitale]

Confirming I also experienced this issue; Prysm+Geth.

[terencechain]

Hi, I believe this is a Geth regression first reported here: ethereum/go-ethereum#31593 and addressed in this pull request: ethereum/go-ethereum#31642
Prysm relies on transaction logs and requires them to be correctly indexed. If Geth returns logs in the wrong order, Prysm won't be able to process the deposit log. I updated my Geth node to the commit that includes the fix and haven't encountered the issue since. Please reopen the issue if it persists after Geth update

[KevinVitale]

Hi, I believe this is a Geth regression first reported here: ethereum/go-ethereum#31593 and addressed in this pull request: ethereum/go-ethereum#31642 Prysm relies on transaction logs and requires them to be correctly indexed. If Geth returns logs in the wrong order, Prysm won't be able to process the deposit log. I updated my Geth node to the commit that includes the fix and haven't encountered the issue since. Please reopen the issue if it persists after Geth update

I've since updated to the latest Geth while also restarting Prysm. No errors reported thus far.

[Jor-Tech]

Did update to latest geth but didn't restarted Prysm.
Now after doing so it seems to be solved indeed.

[wafeishushu]

When I upgraded from 5.3.1 to 6.0.0, I had a similar problem.
My geth is 1.15.9.

[Jor-Tech]

If the message keeps showing up, just try to restart your clients, both Geth and Prysm.

[wafeishushu]

If the message keeps showing up, just try to restart your clients, both Geth and Prysm.

i tried. but it still doesn't work.

[Jor-Tech]

Strange, I'm running same versions with the issue any more. I'd advise to seek help on the Prysm Discord.