Queue disks for upsert when a transient error occurs (#3800)

Check every 5 seconds if there are any queued disks and we have a ready
key manager, meaning that this replica has a key share or it is using
the `HardcodedKeyRetriever`. If there are disks that need upserting then
try to do that. This allows not just for cold boot but for hot insert of
new disks when the trust quorum may not be available.

Current the only transient error is a `KeyManager::Error` which
indicates that the trust quorum is unavailable becaue not enough sleds
are online.

Fixes #3789

Author: andrewjstone

sled-agent/src/bootstrap/agent.rs

@@ -500,8 +500,8 @@ impl Agent {
             addr: SocketAddrV6::new(ip, BOOTSTORE_PORT, 0, 0),
             time_per_tick: std::time::Duration::from_millis(250),
             learn_timeout: std::time::Duration::from_secs(5),
-            rack_init_timeout: std::time::Duration::from_secs(60),
-            rack_secret_request_timeout: std::time::Duration::from_secs(30),
+            rack_init_timeout: std::time::Duration::from_secs(300),
+            rack_secret_request_timeout: std::time::Duration::from_secs(5),
             fsm_state_ledger_paths: bootstore_fsm_state_paths(
                 &storage_resources,
             )