test failed in CI: `test_region_snapshot_replacement_step_after_rop_remove_target_gone`

[david-crespo]

This test failed on a CI run on #8564:

https://github.com/oxidecomputer/omicron/pull/8564/checks?check_run_id=45737263049

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01JZTHHRW0Z4XZNP2QDAN725NX/vRd388kKuDNEoacSDWR6JkiVQpK7Y1RfB54em22tE0W2iZc0/01JZTHJ39YNFMEJJZ02VVW1R7Y#S7295

Excerpt from the log showing the failure:

      FAIL [  84.633s] omicron-nexus::test_all integration_tests::crucible_replacements::test_region_snapshot_replacement_step_after_rop_remove_target_gone
stdout ───
  running 1 test
  test integration_tests::crucible_replacements::test_region_snapshot_replacement_step_after_rop_remove_target_gone has been running for over 60 seconds
  test integration_tests::crucible_replacements::test_region_snapshot_replacement_step_after_rop_remove_target_gone ... FAILED
  failures:
  failures:
      integration_tests::crucible_replacements::test_region_snapshot_replacement_step_after_rop_remove_target_gone
  test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 446 filtered out; finished in 84.20s
  
stderr ───
  log file: /var/tmp/omicron_tmp/test_all-d7048d765dc5ce31-test_region_snapshot_replacement_step_after_rop_remove_target_gone.22155.0.log
  note: configured to log to "/var/tmp/omicron_tmp/test_all-d7048d765dc5ce31-test_region_snapshot_replacement_step_after_rop_remove_target_gone.22155.0.log"
  DB URL: postgresql://root@[::1]:41476/omicron?sslmode=disable
  DB address: [::1]:41476
  log file: /var/tmp/omicron_tmp/test_all-d7048d765dc5ce31-test_region_snapshot_replacement_step_after_rop_remove_target_gone.22155.2.log
  note: configured to log to "/var/tmp/omicron_tmp/test_all-d7048d765dc5ce31-test_region_snapshot_replacement_step_after_rop_remove_target_gone.22155.2.log"
  log file: /var/tmp/omicron_tmp/test_all-d7048d765dc5ce31-test_region_snapshot_replacement_step_after_rop_remove_target_gone.22155.3.log
  note: configured to log to "/var/tmp/omicron_tmp/test_all-d7048d765dc5ce31-test_region_snapshot_replacement_step_after_rop_remove_target_gone.22155.3.log"
  thread 'integration_tests::crucible_replacements::test_region_snapshot_replacement_step_after_rop_remove_target_gone' panicked at nexus/tests/integration_tests/crucible_replacements.rs:1671:13:
  assertion failed: self.datastore.read_only_target_addr(&region_snapshot_replace_request).await.unwrap().is_none()
  stack backtrace:
     0: __rustc::rust_begin_unwind
               at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/panicking.rs:697:5
     1: core::panicking::panic_fmt
               at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/panicking.rs:75:14
     2: core::panicking::panic
               at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/panicking.rs:145:5
     3: {async_fn#0}
               at ./tests/integration_tests/crucible_replacements.rs:1671:13
     4: {async_fn#0}
               at ./tests/integration_tests/crucible_replacements.rs:1959:49
     5: {async_block#0}
               at ./tests/integration_tests/crucible_replacements.rs:1933:1

[jgallagher]

Saw again on #8570: https://buildomat.eng.oxide.computer/wg/0/details/01JZV4212MSKP2QMSXZ3TNQE2E/9AYNqnNfYaYtcIDC3rr3g9fM7tMdB8SCMNadvcS72lBVRH12/01JZV4308Y9N0G96MQ2STJW1XN

[jgallagher]

Failed on main also: https://buildomat.eng.oxide.computer/wg/0/details/01JZX35GGB142R8YC43S5ZWPMC/CkGt4UHher0Sq4LbTdptoJxk8faoNAfI187DxDbDAuuaq116/01JZX35YSBFYZE48EX8Q4D1CRT