Bootstrapping an archive node

[abbaahmad]

Hi. I've tried bootstrapping an archive node by restoring from S3. At the end of the process, I got this message:

2025-04-17T17:30:30.341076Z [tokio-runtime-worker] INFO storage/backup/backup-cli/src/backup_types/transaction/restore.rs:671 Replay finished. {"accumulative_tps":3898,"total_replayed":71976}
2025-04-17T17:30:31.756159Z [state_batch_committer] INFO storage/aptosdb/src/state_merkle_db.rs:235 Committing StateMerkleDb. {"version":2568999993}
2025-04-17T17:30:31.757305Z [state_batch_committer] INFO storage/aptosdb/src/state_store/state_merkle_batch_committer.rs:78 State snapshot committed. {"base_version":2568928024,"root_hash":"37856c93308565e889aa2de8948cc2725dbbdbe2090e8863f24635ead3c88232","version":2568999993}
2025-04-17T17:30:31.761508Z [state-committer] INFO storage/schemadb/src/lib.rs:324 Dropped RocksDB. {"rocksdb_name":"state_merkle_db"}
2025-04-17T17:30:32.512911Z INFO storage/aptosdb/src/state_store/hot_state.rs:134 HotState committer quitting.
2025-04-17T17:30:32.512978Z [tokio-runtime-worker] INFO storage/schemadb/src/lib.rs:324 Dropped RocksDB. {"rocksdb_name":"ledger_db"}
2025-04-17T17:30:32.553704Z [tokio-runtime-worker] INFO storage/backup/backup-cli/src/backup_types/transaction/restore.rs:266 transaction restore succeeded.
2025-04-17T17:30:32.554426Z [tokio-runtime-worker] INFO storage/backup/backup-cli/src/coordinators/restore.rs:85 Restore coordinator exiting with success.
{ 
  "Result": null
}

Which I interpreted as successful. Here was the command used: ./aptos node bootstrap-db --ledger-history-start-version 0 --target-version 2613213792 --command-adapter-config ./s3-public.yaml --target-db-dir ./. I also turned off pruning before starting the node. Now the node is running, but here is the output of status:

{
  "chain_id": 1,
  "epoch": "10936",
  "ledger_version": "2569658445",
  "oldest_ledger_version": "2568000004",
  "ledger_timestamp": "1743942316620829",
  "node_role": "full_node",
  "oldest_block_height": "317437843",
  "block_height": "317703912",
  "git_hash": "cf7e87b6b0a61066841e2d89fc4b01c7e2393386"
}

I would have expected both oldest_ledger_version and oldest_block_height to be 0. Is that a wrong impression of what restore does or did I miss a step somewhere?

[jonathanudd]

I have tried with Google cloud as well with similar results.

[jonathanudd]

The ends of the logs looks like it succeeded but I have made some more tests and it might have been that the run crashed and that it only showed earlier in the logs?

With my current tests the process uses a lot of ram. OOMing with 128GB available.
But it looks like it's able to pick up where it left of so maybe it will be done eventually after some restarts?

[jonathanudd]

I think it's progressing like this but it doesn't look optimal.
When it runs out of memory and restarts it takes a long time for it to get back to processing new blocks etc even if I'm using --metadata-cache-dir

Is this normal or could I do anything to improve it except giving it more RAM?

[jonathanudd]

This is the memory usage for the last 24h and when the memory usage drops it runs out of memory and restarts the script.
I tried to tweak the script with different different values for concurrency which I think affected the time it was able to run before running out of memory. Used different values for --concurrent-downloads and --replay-concurrency-level.

[jonathanudd]

I was able to bootstrap eventually after a lot of restarts and it took a long time.