Client occasionally stops importing new blocks

[mdben1247]

• OpenEthereum version: v3.0.0-stable-b079d1746
• Operating system: Linux
• Installation: built from source
• Fully synchronized: yes
• Network: ethereum
• Restarted: yes

Openethereum occasianally (1-3x per month) stops importing new blocks, simply goes silent without producing any error. Then we issue SIGTERM and we get "Finishing work, please wait..." message. But it does not finish in many minutes... so we kill the process with SIGKILL. Upon restarting everything works normally, until next such 'freeze'.

We first noticed this behaviour in Parity 2.7.2 several months ago and it is now present with Openethereum 3.0.0. We noticed this independently on two different hardware configurations, one fast synced, the other fatdb synced from scratch. There is a similar issue reported in #11737 and also #11539, but the latter is linked to slow block import. That is not the case here. Except for the 'freezes', everything is working responsively, fast and overall very well, block imports are very fast.

Any suggestions on how to debug this issue much appreciated. Thanks.

[dvdplm]

This sounds concerning and debugging it seems non-trivial. The best thing would be a backtrace of what all the threads are doing when the node freezes up, which requires attaching gdb to the process when it's is in the bad state and executing thread apply all bt.
One way of maybe collecting a hint on what is going on could be to run the node with -l shutdown and see what is logged after SIGTERM: I'd expect you to see messages from the various subsystems shutting down (until one of them don't; try comparing what you see on a non-stuck node to a stuck one).
You could also turn on logging for the sync subsystem with -l sync=debug but that will produce a LOT of logs and might fill up your disk if you need it on for a month.

Can you post your configuration please, it might help us reproduce.

[mtbitcoin]

we are also experiencing similar issues after the upgrade to ver 3.0

[mdben1247]

dvdplm, thanks for suggestions, will try all that and report back.

Attaching a config.toml from current hardware with 64gb ram that is fatdb synced. Previous hardware (that also exhibited the same problem) was fast synced and had 32gb, so configuration had smaller caches sizes, but otherwise it was similar.

config.toml.txt

[trailtacos]

@dvdplm There is a thread backtrace on https://github.com/openethereum/openethereum/issues/11737 which seems like the same issue as this one.

[palkeo]

I see the same issue, it matches the 1-3x per month originally reported. I saw it happen on parity 2.7.2 and saw it again on openethereum 3.0.0.

I noticed this only on nodes with tracing enabled, and #11737 also has tracing. But @Marko429 you don't have it do you?

[palkeo]

@dvdplm Could you maybe develop on what you would want exactly from gdb? Only the output of "threads apply all bt" ? I tried it once but it only spits out addresses as there is no debug symbols. That's still helpful to you ?
I need to know in advance to I can run that quickly before restarting the node if I see it happen again.

[mdben1247]

@palkeo, I'll make the gdb trace the next time it happens. Can you please share your cache-sizes and ram size?

[palkeo]

@palkeo, I'll make the gdb trace the next time it happens. Can you please share your cache-sizes and ram size?

My cache size are here: https://github.com/openethereum/openethereum/issues/11735
Ram size is 16G.

[nysxah]

Experiencing both slow block import and periodic freeze on both full/fast and archive nodes since 2.7.2, tracing enabled.

2.5.13 runs fine.

#11737 / #11494 — same behavior.

Could this be because of “database format changes” introduced in 2.7.2?

[dvdplm]

I tried it once but it only spits out addresses as there is no debug symbols. That's still helpful to you ?

You are right, without the debug symbols in there it's not useful. My bad.

[peterbitfly]

We are also experiencing those issues since version 2.7.2 on multiple independent nodes.

[denisgranha]

I can confirm this happens, without more data points / metrics we are blind to diagnose, @adria0 is working on having prometheus metrics and that might add more light here

[palkeo]

Is there anything you can think of that could be useful to you to troubleshoot it?
Maybe providing a debug build (with symbols and anything else that could help) could help in getting you interesting core dumps and backtraces? Any logging option you think could help?

[trailtacos]

Is there anything that is helpful for debugging while it is stuck ? I have a stuck node right now ...