When I update aptos node v1.32 to v1.33, why there has breaking change like persisted_auxiliary_info_db, Force Aptos Node wont use old ledgedb database?

[aaababaaz]

Discord user ID

No response

Describe your question in detail.

➜  storage grep -rin persisted_auxiliary_info_db            
aptosdb/src/db/include/aptosdb_writer.rs:264:                    .persisted_auxiliary_info_db()
aptosdb/src/db/test_helper.rs:941:        .persisted_auxiliary_info_db()
aptosdb/src/db_debugger/examine/print_db_versions.rs:87:                .persisted_auxiliary_info_db_raw()
aptosdb/src/ledger_db/mod.rs:13:        persisted_auxiliary_info_db_column_families, transaction_accumulator_db_column_families,
aptosdb/src/ledger_db/mod.rs:20:        persisted_auxiliary_info_db::PersistedAuxiliaryInfoDb,
aptosdb/src/ledger_db/mod.rs:45:pub(crate) mod persisted_auxiliary_info_db;
aptosdb/src/ledger_db/mod.rs:47:mod persisted_auxiliary_info_db_test;
aptosdb/src/ledger_db/mod.rs:66:pub const PERSISTED_AUXILIARY_INFO_DB_NAME: &str = "persisted_auxiliary_info_db";
aptosdb/src/ledger_db/mod.rs:77:    pub persisted_auxiliary_info_db_batches: SchemaBatch,
aptosdb/src/ledger_db/mod.rs:90:            persisted_auxiliary_info_db_batches: SchemaBatch::new(),
aptosdb/src/ledger_db/mod.rs:110:    persisted_auxiliary_info_db: PersistedAuxiliaryInfoDb,
aptosdb/src/ledger_db/mod.rs:152:                persisted_auxiliary_info_db: PersistedAuxiliaryInfoDb::new(Arc::clone(
aptosdb/src/ledger_db/mod.rs:171:        let mut persisted_auxiliary_info_db = None;
aptosdb/src/ledger_db/mod.rs:194:                persisted_auxiliary_info_db = Some(PersistedAuxiliaryInfoDb::new(Arc::new(
aptosdb/src/ledger_db/mod.rs:196:                        ledger_db_folder.join(PERSISTED_AUXILIARY_INFO_DB_NAME),
aptosdb/src/ledger_db/mod.rs:197:                        PERSISTED_AUXILIARY_INFO_DB_NAME,
aptosdb/src/ledger_db/mod.rs:266:            persisted_auxiliary_info_db: persisted_auxiliary_info_db.unwrap(),
aptosdb/src/ledger_db/mod.rs:322:                .persisted_auxiliary_info_db()
aptosdb/src/ledger_db/mod.rs:323:                .create_checkpoint(cp_ledger_db_folder.join(PERSISTED_AUXILIARY_INFO_DB_NAME))?;
aptosdb/src/ledger_db/mod.rs:348:        self.persisted_auxiliary_info_db
aptosdb/src/ledger_db/mod.rs:380:    pub(crate) fn persisted_auxiliary_info_db(&self) -> &PersistedAuxiliaryInfoDb {
aptosdb/src/ledger_db/mod.rs:381:        &self.persisted_auxiliary_info_db
aptosdb/src/ledger_db/mod.rs:384:    pub(crate) fn persisted_auxiliary_info_db_raw(&self) -> &DB {
aptosdb/src/ledger_db/mod.rs:385:        self.persisted_auxiliary_info_db.db()
aptosdb/src/ledger_db/mod.rs:461:            PERSISTED_AUXILIARY_INFO_DB_NAME => persisted_auxiliary_info_db_column_families(),
aptosdb/src/ledger_db/mod.rs:476:            PERSISTED_AUXILIARY_INFO_DB_NAME => gen_persisted_auxiliary_info_cfds(db_config),
aptosdb/src/ledger_db/mod.rs:502:        self.persisted_auxiliary_info_db
aptosdb/src/ledger_db/mod.rs:503:            .write_schemas(schemas.persisted_auxiliary_info_db_batches)?;
aptosdb/src/ledger_db/persisted_auxiliary_info_db_test.rs:16:        .persisted_auxiliary_info_db()
aptosdb/src/ledger_db/persisted_auxiliary_info_db_test.rs:90:        let iter = db.ledger_db.persisted_auxiliary_info_db()
aptosdb/src/pruner/ledger_pruner/persisted_auxiliary_info_pruner.rs:5:    ledger_db::{persisted_auxiliary_info_db::PersistedAuxiliaryInfoDb, LedgerDb},
aptosdb/src/pruner/ledger_pruner/persisted_auxiliary_info_pruner.rs:33:            .persisted_auxiliary_info_db()
aptosdb/src/pruner/ledger_pruner/persisted_auxiliary_info_pruner.rs:44:            ledger_db.persisted_auxiliary_info_db_raw(),
aptosdb/src/db_options.rs:53:pub(super) fn persisted_auxiliary_info_db_column_families() -> Vec<ColumnFamilyName> {
aptosdb/src/db_options.rs:206:    let cfs = persisted_auxiliary_info_db_column_families();

So, I have like 10TB AptosDB database, the configuration is like this:

base:
  role: "full_node"
  data_dir: "./data"
  waypoint:
    from_file: "./waypoint.txt"

execution:
  genesis_file_location: "./genesis.blob"
  concurrency_level: 32

full_node_networks:
- network_id: "public"
  discovery_method: "onchain"
  listen_address: "/ip4/0.0.0.0/tcp/6182"
  #max_concurrent_network_reqs: 65535
  max_outbound_connections: 16
  max_inbound_connections: 16
  identity:
    type: "from_config"
    key: ""
    peer_id: ""

mempool:
  shared_mempool_max_concurrent_inbound_syncs: 200 # default 4
  max_broadcasts_per_peer: 200 # default 1
  default_failovers: 0 # default 3
  
api:
  enabled: true
  address: "0.0.0.0:8080"
  content_length_limit: 536870912
  max_submit_transaction_batch_size: 65535
  max_transactions_page_size: 10000
  max_events_page_size: 10000
  max_account_resources_page_size: 10000
  max_account_modules_page_size: 10000
  tls_cert_path: null
  tls_key_path: null


state_sync:
    state_sync_driver:
        bootstrapping_mode: ExecuteTransactionsFromGenesis
        continuous_syncing_mode: ExecuteTransactions
    data_streaming_service:
      max_concurrent_requests: 64
      max_concurrent_state_requests: 64
      max_notification_id_mappings: 1200
      progress_check_interval_ms: 100
    aptos_data_client:
      use_compression: true
  

storage:
  enable_indexer: true
  storage_pruner_config:
      ledger_pruner_config:
          enable: false

indexer:
  enabled: false
  check_chain_id: true
  emit_every: 1000
  processor: "default_processor"
  fetch_tasks: 10
  processor_tasks: 10

My question is, I got a Aptos Node breaking change, that leads my fullnode unable to read my old aptos blockchain data, instead, the aptos node is try to bootstrap data, without bother the old blockchain data.

../../@aptos/data_overlay/db/ledger_db/metadata
../../@aptos/data_overlay/db/ledger_db/transaction_db
../../@aptos/data_overlay/db/ledger_db/transaction_info_db
../../@aptos/data_overlay/db/ledger_db/persisted_auxiliary_info_db
../../@aptos/data_overlay/db/ledger_db/transaction_auxiliary_data_db
../../@aptos/data_overlay/db/ledger_db/write_set_db
../../@aptos/data_overlay/db/ledger_db/transaction_accumulator_db
../../@aptos/data_overlay/db/ledger_db/event_db

My database wasnt on above folders, it flatten on ../../@aptos/data_overlay/db/ledger_db/, that is breaking change, my aptos node didnt read the old blockchain data.

(venv) ➜  HRM-Text git:(main) ✗ du -ha /home/@aptos/data | sort -h | tail -n 50
220M	/home/@aptos/data/db/ledger_db/356156.sst
221M	/home/@aptos/data/db/ledger_db/354814.sst
221M	/home/@aptos/data/db/ledger_db/355117.sst
221M	/home/@aptos/data/db/ledger_db/356307.sst
221M	/home/@aptos/data/db/ledger_db/358110.sst
222M	/home/@aptos/data/db/ledger_db/355567.sst
222M	/home/@aptos/data/db/ledger_db/356387.sst
222M	/home/@aptos/data/db/ledger_db/359877.sst
222M	/home/@aptos/data/db/ledger_db/360599.sst
225M	/home/@aptos/data/db/ledger_db/355808.sst
225M	/home/@aptos/data/db/ledger_db/356686.sst
226M	/home/@aptos/data/db/ledger_db/355967.sst
227M	/home/@aptos/data/db/ledger_db/356356.sst
229M	/home/@aptos/data/db/ledger_db/LOG.old.1746526939697746
234M	/home/@aptos/data/db/ledger_db/LOG.old.1749703615519892
238M	/home/@aptos/data/db/ledger_db/084322.sst
239M	/home/@aptos/data/db/ledger_db/LOG.old.1741922808895283
243M	/home/@aptos/data/db/ledger_db/LOG.old.1748879408794068
250M	/home/@aptos/data/db/state_merkle_db/LOG.old.1753025905124724
256M	/home/@aptos/data/db/ledger_db/084367.sst
269M	/home/@aptos/data/db/ledger_db/LOG.old.1742964655154924
271M	/home/@aptos/data/db/ledger_db/LOG.old.1744780592637799
275M	/home/@aptos/data/db/ledger_db/084513.sst
281M	/home/@aptos/data/db/ledger_db/LOG.old.1732854220992801
287M	/home/@aptos/data/db/ledger_db/084426.sst
288M	/home/@aptos/data/db/ledger_db/004700.sst
289M	/home/@aptos/data/db/ledger_db/084465.sst
290M	/home/@aptos/data/db/ledger_db/084392.sst
292M	/home/@aptos/data/db/ledger_db/LOG.old.1740564650380271
303M	/home/@aptos/data/db/ledger_db/084473.sst
305M	/home/@aptos/data/db/ledger_db/LOG.old.1717813895127629
317M	/home/@aptos/data/db/ledger_db/LOG.old.1721978142875721
389M	/home/@aptos/data/db/ledger_db/LOG.old.1752149215613255
408M	/home/@aptos/data/db/ledger_db/LOG.old.1727547341244431
416M	/home/@aptos/data/db/ledger_db/LOG.old.1735427503912228
516M	/home/@aptos/data/db/ledger_db/LOG.old.1730373663725373
560M	/home/@aptos/data/db/ledger_db/LOG.old.1724993816113539
607M	/home/@aptos/data/db/state_merkle_db/LOG.old.1749899297256595
821M	/home/@aptos/data/db/state_merkle_db/LOG.old.1753802006584281
973M	/home/@aptos/data/db/index_indexer_db
1.7G	/home/@aptos/data/db/state_merkle_db/LOG.old.1750327307864228
3.2G	/home/@aptos/data/db/state_merkle_db/LOG.old.1753719967553510
3.4G	/home/@aptos/data/db/state_merkle_db/LOG.old.1754543343866830
3.7G	/home/@aptos/data/db/state_merkle_db/LOG.old.1749703655075374
3.7G	/home/@aptos/data/db/state_merkle_db/LOG.old.1753009588728585
12G	/home/@aptos/data/db/state_merkle_db/LOG.old.1752149397688734
342G	/home/@aptos/data/db/state_merkle_db
9.7T	/home/@aptos/data/db/ledger_db
10T	/home/@aptos/data
10T	/home/@aptos/data/db

What error, if any, are you getting?

No response

What have you tried or looked at? Or how can we reproduce the error?

Just look aptos node v1.32 to v1.33 how the behavior about database file reading!

Which operating system are you using?

Linux (Ubuntu, Fedora, Windows WSL, etc.)

Which SDK or tool are you using? (if any)

N/A

Describe your environment or tooling in detail

Arch Linux

[JoshLind]

Hi, thanks for reaching out. 😄 If you're seeing your node re-syncing all data, this is likely due to node sharding being enabled. Please see the attached message about migrating your node to use a sharded database:

• Sharding will be enabled by default. If you have not migrated already, please migrate now.
• If you're using indexer APIs, review the migration options (steps 2 and 3) carefully — if your node is not migrated before upgrading, your node will break.
• Migration Instructions: https://aptoslabs.notion.site/DB-Sharding-Migration-Public-Full-Nodes-1978b846eb7280b29f17ceee7d480730

[aaababaaz]

Hi, thanks for reaching out. 😄 If you're seeing your node re-syncing all data, this is likely due to node sharding being enabled. Please see the attached message about migrating your node to use a sharded database:

• Sharding will be enabled by default. If you have not migrated already, please migrate now.
• If you're using indexer APIs, review the migration options (steps 2 and 3) carefully — if your node is not migrated before upgrading, your node will break.
• Migration Instructions: https://aptoslabs.notion.site/DB-Sharding-Migration-Public-Full-Nodes-1978b846eb7280b29f17ceee7d480730

a full DB wipe is required ? Seriously..? That is not respect the option:

  storage_pruner_config:
      ledger_pruner_config:
          enable: false

My AptosDB don't prune ledger since the Aptos Genesis, that is going to be troublesome, plus the 10TB database need to be proceed. The benefit could be seen in this way, Free to check anything in specific "version" is it exists or what is it on that time.

You know what, I used SSD raid 0 and overlayfs to manage out the 10TB data storages. My future plan is goes like that, overlayfs on overlayfs on overlayfs on overlayfs... Such recursion planing to storing AptosDB.

so, how to disable node sharding? I see:

storage:
  rocksdb_configs:
    enable_storage_sharding: False

However that looks like, one day you guys will force enable storage sharding, is it? And yes, I am running an "Archival" Fullnode.

I wonder, have you guys fixed error about execution sync process when we try sync from brand new version Fullnode about always execute the transactions wont get any error in execution sync all times ?

Well, Look like I need the time to do the annoying resync job.

[grao1991]

You can choose to do output sync to avoid error.

sharding will be enforced, in few months. 10T is small, do the migration ASAP.

Note with the increased adoption of Aptos, the storage growth will be super fast. It would be hard for us to continue support once the storage beyond certain size. I'd suggest you to think about having a different solution for your use case, or you will be on your own.

[aaababaaz]

You can choose to do output sync to avoid error.

sharding will be enforced, in few months. 10T is small, do the migration ASAP.

Note with the increased adoption of Aptos, the storage growth will be super fast. It would be hard for us to continue support once the storage beyond certain size. I'd suggest you to think about having a different solution for your use case, or you will be on your own.

I see.