Aptos node going OOM

[gflow33]

Discord user ID

No response

Describe your question in detail.

I am wondering what could be making our aptos mainnet nodes go out of memory, the containers running this node have been allocated 70gb RAM and are OOMing.

What error, if any, are you getting?

So prior we had been getting this error

Jul 30 04:11:58 juju-d28cfa-0 aptos-node[1650201]: 2025-07-30T04:11:58.433494Z [network-Pub-3321] WARN network/framework/src/connectivity_manager/mod.rs:1165 Failed to resolve network address /dns/vfn.aptos.dsrvlabs.net/tcp/6182/noise-ik/0xdf0eb85221ed6f116dd5caa1e7461d2ec2e3c6be4b252ac90246b3384e46d96d/handshake/0: Too many open files (os error 24) {"network_context":{"role":"full_node","network_id":"Public","peer_id":"b0e8786be73b4702a3bef419b1734b099e9864c0e3cb625f6a78afdb46514c7d"}}
Jul 30 04:11:58 juju-d28cfa-0 aptos-node[1650201]: 2025-07-30T04:11:58.433499Z [network-Pub-3306] WARN network/framework/src/connectivity_manager/mod.rs:1165 Failed to resolve network address /dns/aptos-mainnet-node-0.nodereal.io/tcp/6182/noise-ik/0xb6efef1ce4bb0245a169b5fe15ae186eb84b176023d789f51d0deb5661c68b42/handshake/0: Too many open files (os error 24) {"network_context":{"role":"full_node","network_id":"Public","peer_id":"b0e8786be73b4702a3bef419b1734b099e9864c0e3cb625f6a78afdb46514c7d"}}
Jul 30 04:11:58 juju-d28cfa-0 aptos-node[1650201]: 2025-07-30T04:11:58.433513Z [network-Pub-3318] WARN network/framework/src/connectivity_manager/mod.rs:1165 Failed to resolve network address /dns/fullnode.aptoslabs.ltd/tcp/6182/noise-ik/0x9801bf1bb28f45cebe83cdd33550eff87c4727d3e85edbd001f047fdb1038f7c/handshake/0: Too many open files (os error 24) {"network_context":{"role":"full_node","network_id":"Public","peer_id":"b0e8786be73b4702a3bef419b1734b099e9864c0e3cb625f6a78afdb46514c7d"}}
Jul 30 04:11:58 juju-d28cfa-0 aptos-node[1650201]: 2025-07-30T04:11:58.433542Z [network-Pub-3307] WARN network/framework/src/connectivity_manager/mod.rs:1165 Failed to resolve network address /dns/aptos-mainnet-figment-dp-vfn-3.staking.production.figment.io/tcp/6182/noise-ik/0xd8bfdfd70df4cb43d872814c293c66fb6e6df5abdfe25cbc588a3b264dd33b22/handshake/0: Too many open files (os error 24) {"network_context":{"role":"full_node","network_id":"Public","peer_id":"b0e8786be73b4702a3bef419b1734b099e9864c0e3cb625f6a78afdb46514c7d"}}
Jul 30 04:11:58 juju-d28cfa-0 aptos-node[1650201]: 2025-07-30T04:11:58.433543Z [network-Pub-3331] WARN network/framework/src/connectivity_manager/mod.rs:1165 Failed to resolve network address /dns/fullnode.mainnet.aptos.forgoo.com/tcp/6182/noise-ik/0x8f49c6d7d4c49ac7297b31cc34ed185d30934b22e6d30ef7e44a1be8435c7656/handshake/0: Too many open files (os error 24) {"network_context":{"role":"full_node","network_id":"Public","peer_id":"b0e8786be73b4702a3bef419b1734b099e9864c0e3cb625f6a78afdb46514c7d"}}
Jul 30 04:11:58 juju-d28cfa-0 aptos-node[1650201]: 2025-07-30T04:11:58.433557Z [network-Pub-3329] WARN network/framework/src/connectivity_manager/mod.rs:1165 Failed to resolve network address /dns/aptos-mainnet-vfn.verse2.io/tcp/6182/noise-ik/0x5ec1236035e269b1c1d13406d139bd8fe47eeeb9bcb25e93b1e54ed177801d0d/handshake/0: Too many open files (os error 24) {"network_context":{"role":"full_node","network_id":"Public","peer_id":"b0e8786be73b4702a3bef419b1734b099e9864c0e3cb625f6a78afdb46514c7d"}}
Jul 30 04:11:58 juju-d28cfa-0 aptos-node[1650201]: 2025-07-30T04:11:58.433568Z [network-Pub-3319] WARN network/framework/src/connectivity_manager/mod.rs:1165 Failed to resolve network address /dns/fullnode.b5e7e23d-0672-49b5-9139-565ea0d1a16e.aptos.herd.run/tcp/6182/noise-ik/0x430c1f838c93e4bc80a4a65fad1fd5448328a0d6d0cfb342e680fb196a68da39/handshake/0: Too many open files (os error 24) {"network_context":{"role":"full_node","network_id":"Public","peer_id":"b0e8786be73b4702a3bef419b1734b099e9864c0e3cb625f6a78afdb46514c7d"}}

And as time progressed we started to get this error when starting up

Jul 30 23:59:48 juju-d28cfa-0 aptos-node[1841476]: 2025-07-30T23:59:48.701764Z [main] WARN aptos-node/src/utils.rs:113 System RLIMIT_NOFILE hard limit too small. {"hard_limit":10000,"required":999999}

and eventually OOM after this log message

Jul 31 10:15:25 juju-d28cfa-0 aptos-node[1900865]: 2025-07-31T10:15:09.459142Z [main] INFO config/src/config/node_config.rs:106 Using validator_network config: null
Jul 31 10:15:25 juju-d28cfa-0 aptos-node[1900865]: 2025-07-31T10:15:09.459143Z [main] INFO config/src/config/node_config.rs:106 Using indexer_db_config config: {"enable_transaction":false,"enable_event":false,"enable_event_v2_translation":false,"event_v2_translation_ignores_below_version":0,"enable_statekeys":false,"batch_size":10000}
Jul 31 10:15:25 juju-d28cfa-0 aptos-node[1900865]: 2025-07-31T10:15:09.465905Z [admin-24] INFO crates/aptos-admin-service/src/server/mod.rs:135 Started AdminService at 0.0.0.0:9102, enabled: false.
Jul 31 10:15:25 juju-d28cfa-0 aptos-node[1900865]: 2025-07-31T10:15:25.302545Z [main] INFO storage/schemadb/src/lib.rs:185 Opened RocksDB. {"open_mode":"ReadWrite","rocksdb_name":"ledger_db"}
Jul 31 10:15:25 juju-d28cfa-0 aptos-node[1900865]: 2025-07-31T10:15:25.302827Z [main] INFO storage/aptosdb/src/ledger_db/mod.rs:414 Opened ledger_db at "/home/aptos/.aptos/db/ledger_db"!
Jul 31 10:15:25 juju-d28cfa-0 aptos-node[1900865]: Truncating State KV DB: current: 3144875270, target: 3144875270

What have you tried or looked at? Or how can we reproduce the error?

I looked at the service file to determine what we set as the limit for open files and increased it to what the error message recommended, which is 1_000_000. This did not make and difference

Which operating system are you using?

Linux (Ubuntu, Fedora, Windows WSL, etc.)

Which SDK or tool are you using? (if any)

N/A

Describe your environment or tooling in detail

OS: Linux (Ubuntu 22.04)
Arch: x86_64
Binary: Aptos Node 1.32.2
RAM: 70gb
ROM: 25tb

[wqfish23]

How did you run the node? Do you have the exact command we can use to reproduce the error?

[gflow33]

Sorry for the delay but this is the command we use to run the node
./aptos-node -f fullnode.yaml

The fullnode.yaml

api:
  address: 0.0.0.0:8080
  enabled: true
base:
  data_dir: /home/aptos/.aptos
  role: full_node
  waypoint:
    from_file: /home/aptos/waypoint.txt
execution:
  genesis_file_location: /home/aptos/genesis.blob
full_node_networks:
- discovery_method: onchain
  listen_address: /ip4/127.0.0.1/tcp/6180
  network_id: public
state_sync:
  state_sync_driver:
    bootstrapping_mode: ApplyTransactionOutputsFromGenesis
    continuous_syncing_mode: ApplyTransactionOutputs
storage:
  storage_pruner_config:
    ledger_pruner_config:
      enable: false

[wqfish23]

How long does it typically take for the node to OOM after starting? Is it after syncing a lot of transactions?

[gflow33]

We have given the containers more RAM, but it OOMs after like 20 mins after restarting

[grao1991]

In your log, do you see the node is making progress or it stuck at some point?

[gflow33]

It is not making progress, it get stuck at this point and then goes out of memory

Jul 31 10:15:25 juju-d28cfa-0 aptos-node[1900865]: Truncating State KV DB: current: 3144875270, target: 3144875270

[grao1991]

Try the latest release, we have a fix for this.