Error while restoring `NooBaa brain` (PostgreSQL DB) to new cluster after disaster recovery

[anisurrahman75]

Hi NooBaa team,

I'm encountering an error while attempting to restore the NooBaa brain (PostgreSQL DB) into a new cluster during a disaster recovery scenario.

Setup:
I have a primary cluster (management) and two data center clusters: dc1 and dc2.
In the primary cluster, I configured mirrored tier bucket classes pointing to dc1 and dc2.

I periodically take backups of:

• The NooBaa PostgreSQL database (brain)
• The required NooBaa YAML manifests (e.g., backingstore, bucketclass, etc.)

Disaster Recovery Scenario
A disaster occurred in the original primary cluster. I provisioned a new Kubernetes cluster to act as the new primary. I installed a new NooBaa operator and trying to restore the backed-up NooBaa brain PostgreSQL DB into the new cluster. During this restore process, I encountered an error from NooBaa.

I'm following this approach after reinstalling noobba:

Problem

• NooBaa crashes after starting up with the restored DB.
• There is no clear documentation or official procedure on restoring the NooBaa brain into a fresh cluster.

Questions

• Is restoring the NooBaa brain PostgreSQL DB into a new cluster officially supported?
• Is there a documented disaster recovery process?
• What are the expected steps for restoring NooBaa and its state into a new cluster, assuming the original is lost?

Additional Info
Please let me know what logs or information you need. I can provide:

The error trace:

Jun-24 8:41:08.520 [Upgrade/25]    [L0] core.server.system_services.system_store:: system_store is running in standalone mode. skip _register_for_changes
Jun-24 8:41:08.667 [Upgrade/25] [ERROR] core.server.system_services.system_store:: SystemStore: load failed Error: NO_SUCH_KEY
    at MasterKeysManager.get_master_key_by_id (/root/node_modules/noobaa-core/src/server/system_services/master_key_manager.js:165:35)
    at MasterKeysManager._resolve_master_key (/root/node_modules/noobaa-core/src/server/system_services/master_key_manager.js:185:32)
    at MasterKeysManager.get_master_key_by_id (/root/node_modules/noobaa-core/src/server/system_services/master_key_manager.js:167:27)
    at MasterKeysManager._resolve_master_key (/root/node_modules/noobaa-core/src/server/system_services/master_key_manager.js:185:32)
    at MasterKeysManager.get_master_key_by_id (/root/node_modules/noobaa-core/src/server/system_services/master_key_manager.js:167:27)
    at MasterKeysManager.decrypt_all_accounts_secret_keys (/root/node_modules/noobaa-core/src/server/system_services/master_key_manager.js:312:36)
    at /root/node_modules/noobaa-core/src/server/system_services/system_store.js:441:51
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Semaphore.surround (/root/node_modules/noobaa-core/src/util/semaphore.js:71:84)
    at async init_db_upgrade (/root/node_modules/noobaa-core/src/upgrade/upgrade_manager.js:42:9)
Jun-24 8:41:08.667 [Upgrade/25]    [L0] core.server.system_services.system_store:: system_store is running in standalone mode. skip _register_for_changes
Jun-24 8:41:08.668 [Upgrade/25] [ERROR] UPGRADE:: failed to load system store!! Error: NO_SUCH_KEY
    at MasterKeysManager.get_master_key_by_id (/root/node_modules/noobaa-core/src/server/system_services/master_key_manager.js:165:35)
    at MasterKeysManager._resolve_master_key (/root/node_modules/noobaa-core/src/server/system_services/master_key_manager.js:185:32)
    at MasterKeysManager.get_master_key_by_id (/root/node_modules/noobaa-core/src/server/system_services/master_key_manager.js:167:27)
    at MasterKeysManager._resolve_master_key (/root/node_modules/noobaa-core/src/server/system_services/master_key_manager.js:185:32)
    at MasterKeysManager.get_master_key_by_id (/root/node_modules/noobaa-core/src/server/system_services/master_key_manager.js:167:27)
    at MasterKeysManager.decrypt_all_accounts_secret_keys (/root/node_modules/noobaa-core/src/server/system_services/master_key_manager.js:312:36)
    at /root/node_modules/noobaa-core/src/server/system_services/system_store.js:441:51
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Semaphore.surround (/root/node_modules/noobaa-core/src/util/semaphore.js:71:84)
    at async init_db_upgrade (/root/node_modules/noobaa-core/src/upgrade/upgrade_manager.js:42:9)
Jun-24 8:41:08.668 [Upgrade/25] [ERROR] UPGRADE:: failed to init upgrade process!!
upgrade_manager failed with exit code 1
noobaa_init.sh finished
noobaa_init failed with exit code 1. aborting

NooBaa version:

INFO[0000] CLI version: 5.18.4                          
INFO[0000] noobaa-image: noobaa/noobaa-core:master-20240520 
INFO[0000] operator-image: noobaa/noobaa-operator:5.18.4 

Thanks in advance for your help!