Node stopped syncing and doing weird stuff - then being off tip

[tmeinlschmidt]

• Erigon 1.3.9 (we didn't have this issues back in 1.3.6)

• plenty of ram, seeing this behaviour with 128g, 256 or 768g ram

• zfs, nvme disks

• running in docker

• started cca 01:08 (see images), node started being late, then stopped sending metrics (see gaps) and it's basically off-line for common usage

config options:

                "--authrpc.jwtsecret",
                "/data/erigon/jwt_secret",
                "--bodies.cache",
                "4GB",
                "--chain",
                "bsc",
                "--datadir",
                "/data/erigon",
                "--db.pagesize",
                "16K",
                "--db.size.limit",
                "10TB",
                "--etl.bufferSize",
                "6GB",
                "--log.dir.path",
                "/data/erigon/logs",
                "--maxpeers",
                "100",
                "--port",
                "30001",
                "--private.api.addr",
                "127.0.0.1:7106",
                "--rpc.batch.concurrency",
                "128",
                "--rpc.batch.limit",
                "400",
                "--rpc.returndata.limit",
                "200000",
                "--torrent.download.rate",
                "1000mb",
                "--torrent.port",
                "30002",
                "--torrent.upload.rate",
                "10mb",
                "--prune.mode",
                "archive",
                "--nat",
                "extip:<public ip?",
                "--metrics",
                "--metrics.addr=<public ip>",
                "--metrics.port=6061",
                "--diagnostics.disabled",
                "--http.api",
                "eth,debug,net,trace,web3,erigon",
                "--http.compression",
                "--http.vhosts",
                "any",
                "--http.port",
                "7104",
                "--ws",
                "--ws.port",
                "7103",
                "--sync.loop.block.limit=10_000",
                "--batchSize",
                "2g",
                "--verbosity",
                "info"

Node is running, then starts pruning blocks (see image), even it's set as an archive node.

(log scale)

nothing important in logs, just

[INFO] [06-15|23:06:34.821] BuildFilesInBackground                   step=5000 lastInDB=5000q
...
[WARN] [06-15|01:31:30.397] Caller gas above allowance, capping      requested=124148016 cap=50000000
[WARN] [06-15|01:31:43.482] Caller gas above allowance, capping      requested=123181431 cap=50000000
[WARN] [06-15|01:32:01.648] Caller gas above allowance, capping      requested=124511013 cap=50000000
[WARN] [06-15|01:32:02.611] Caller gas above allowance, capping      requested=124511013 cap=50000000
[WARN] [06-15|01:32:02.611] Gas estimation capped by limited funds   original=50000000 balance=344700000000000 sent=0 maxFeePerGas=10000000000 fundable=34470

[blxdyx]

What about the db size.

[tmeinlschmidt]

real db size 4.81T (whole erigon folder), db.size.limit set in erigon - 10TB. So plenty of space.

[blxdyx]

real db size 4.81T (whole erigon folder), db.size.limit set in erigon - 10TB. So plenty of space.

Sorry, i mean the chaindata fold size. Those warning are about rpc request (not about sync error). You could set this flag for bigger gaslimit --rpc.gascap.

[tmeinlschmidt]

already set

                "--rpc.gascap",
                "140000000"

and it does the same. Chaindata size (mb):

146567	chaindata

perhaps I should enable debug logging to get more logs what's going on

[blxdyx]

You could try latest main branch. Remove chaindata and restart.
Suggest .
--db.pagesize=4K --sync.loop.block.limit=1000

[tmeinlschmidt]

we already have --sync.loop.block.limit=10000 in config. Blocksize is 16kB, does it make that differnce? In some other readme, there's even 64kB recommendation.

[blxdyx]

we already have --sync.loop.block.limit=10000 in config. Blocksize is 16kB, does it make that differnce? In some other readme, there's even 64kB recommendation.

sorry, updated. Suggest 4kb, the chaindata size should not bigger than 50GB

[tmeinlschmidt]

tried 4kb, got

[INFO] [06-16|13:36:07.671] logging to file system                   log dir=/data/erigon/logs file prefix=erigon log level=info json=false
[INFO] [06-16|13:36:07.672] Enabling metrics export to prometheus    path=http://<ip>:6061/debug/metrics/prometheus
[INFO] [06-16|13:36:07.672] Build info                               git_branch=main git_tag=v1.3.9-3-g9ea7769-dirty git_commit=9ea77697056a855dcc927887028cdd0581a8de5d
[INFO] [06-16|13:36:07.672] Starting Erigon on BSC mainnet...
[INFO] [06-16|13:36:07.674] Maximum peer count                       total=100
[INFO] [06-16|13:36:07.674] starting HTTP APIs                       port=7104 APIs=eth,debug,net,trace,web3,erigon
[INFO] [06-16|13:36:07.676] Set global gas cap                       cap=50000000
[INFO] [06-16|13:36:07.680] torrent verbosity                        level=WRN
[INFO] [06-16|13:36:09.716] [Downloader] Running with                ipv6-enabled=true ipv4-enabled=true download.rate=1000mb upload.rate=10mb
[INFO] [06-16|13:36:09.716] Opening Database                         label=chaindata path=/data/erigon/chaindata
[EROR] [06-16|13:36:09.716] Erigon startup                           err="mdbx_env_set_geometry: MDBX_TOO_LARGE: Database is too large for current system, e.g. could NOT be mapped into RAM"
mdbx_env_set_geometry: MDBX_TOO_LARGE: Database is too large for current system, e.g. could NOT be mapped into RAM

[blxdyx]

Also remove this flag --db.size.limit, and remove chaindata

[Ajes1337]

I have also experienced this problem for a good amount of weeks.. falls behind, even on "good" hardware (both fullnode and archival) - while its doing "[snapshots] pruning state"

[Arkenoth]

I have the same issue. Usually it happens several times a day where my node is behind takes a lot of time to come back sync because the block are mined too fast

[avinashbo]

The issue is still prevalent on v1.3.10 where a fully synced node goes off sync and the only corresponding interesting events as per the metrics is the pruning activity kicking in

The only solution that works for now is to remove the chaindata folder and restart the node so that it catches up to the tip, stays on the tip consistently for 2-3 days before the issue kicks in again. Currently this forces BSC nodes to be cleaned up (chaindata) on a recurring basis to keep them performing well

[blxdyx]

Use the latest main branch, i fixed that issue. But it need resync from genesis.

[Ajes1337]

gib docker img of main branch 🤞

[tmeinlschmidt]

@blxdyx tried latest main and it doesn't work.. removed all the data, syncing from scratch ended up like this

[INFO] [07-16|07:55:57.634] [1/8 OtterSync] Downloading              progress="(288/425 files) 100.00% - 12.1GB/12.1GB" time-left=999hrs:99m total-time=14h20m40s download-rate=0B/s completion-rate=0B/s alloc=2.2GB sys=2.9GB
[INFO] [07-16|07:55:57.636] [snapshots] no metadata yet              files=137 list=v1-008000-008500-headers.seg,v1-019500-020000-headers.idx,v1-028000-028500-headers.seg,v1-030000-030500-headers.idx,v1-035000-035500-headers.idx,...

see that nice progress math calculation - 288/425 files is 100% ...