[nexus] SP ereport ingestion (#8296)

This branch adds a Nexus background task for ingesting ereports from
service processors via MGS, using the MGS API endpoint added in #7903.
These APIs in turn expose the MGS/SP ereport ingestion protocol added in
oxidecomputer/management-gateway-service#370.

For more information on the protocol itself, refer to the following
RFDs:

- [RFD 520  Control Plane Fault Ingestion and Data Model][RFD 520]
- [RFD 544 Embedded E-Report Formats][RFD 544]
- [RFD 545 Firmware E-Report Aggregation and Evacuation][RFD 545]

In addition to the ereport ingester background task, this branch also
adds database tables for storing ereports from SPs, which are necessary
to implement the ingestion task. I've also added a table for storing
ereports from the sled host OS, which will eventually be ingested via
sled-agent. While there isn't currently anything that populates that
table, I wanted to begin sketching out how we would represent the two
categories of ereports we expect to deal with, and how we would query
both tables for ereports.

Finally, this branch also adds OMDB commands for querying the ereports
stored in the database. These OMDB commands may be useful both for
debugging the ereport ingestion subsystem itself *and* for diagnosing
issues once the SP firmware actually emits ereports. At present, the
higher-level components of the fault-management subsystem, which will
process ereports, diagnose faults, and generate alerts, have yet to be
implemented. Therefore, the OMDB ereport commands serve as an interim
solution for accessing the lower-level data, which may be useful for
debugging such faults until the higher-level FMA components exist.

[RFD 520]: https://rfd.shared.oxide.computer/rfd/0520
[RFD 544]: https://rfd.shared.oxide.computer/rfd/0544
[RFD 545]: https://rfd.shared.oxide.computer/rfd/0545

---------

Co-authored-by: Sean Klein <sean@oxide.computer>

Author: hawkw

Cargo.lock

@@ -11,6 +11,7 @@ workspace = true
 base64.workspace = true
 chrono.workspace = true
 daft.workspace = true
+ereport-types.workspace = true
 gateway-messages.workspace = true
 gateway-types.workspace = true
 omicron-uuid-kinds.workspace = true

clients/gateway-client/Cargo.toml

@@ -70,12 +70,15 @@ progenitor::generate_api!(
         SpIgnition = { derives = [PartialEq, Eq, PartialOrd, Ord] },
         SpIgnitionSystemType = { derives = [Copy, PartialEq, Eq, PartialOrd, Ord] },
         SpState = { derives = [PartialEq, Eq, PartialOrd, Ord] },
-        SpType = { derives = [daft::Diffable] },
+        SpType = { derives = [daft::Diffable, PartialEq, Eq, PartialOrd, Ord] },
         SpUpdateStatus = { derives = [PartialEq, Hash, Eq] },
         UpdatePreparationProgress = { derives = [PartialEq, Hash, Eq] },
     },
     replace = {
         RotSlot = gateway_types::rot::RotSlot,
+        Ena = ereport_types::Ena,
+        Ereport = ereport_types::Ereport,
+        TypedUuidForEreporterRestartKind = omicron_uuid_kinds::EreporterRestartUuid,
         TypedUuidForMupdateKind = omicron_uuid_kinds::MupdateUuid,
     },
 );

clients/gateway-client/src/lib.rs

@@ -24,6 +24,7 @@ csv.workspace = true
 diesel.workspace = true
 dropshot.workspace = true
 dyn-clone.workspace = true
+ereport-types.workspace = true
 futures.workspace = true
 gateway-client.workspace = true
 gateway-messages.workspace = true

dev-tools/omdb/Cargo.toml

@@ -19,6 +19,7 @@
 
 use crate::Omdb;
 use crate::check_allow_destructive::DestructiveOperationToken;
+use crate::db::ereport::cmd_db_ereport;
 use crate::helpers::CONNECTION_OPTIONS_HEADING;
 use crate::helpers::DATABASE_OPTIONS_HEADING;
 use crate::helpers::const_max_len;
@@ -180,6 +181,7 @@ use tabled::Tabled;
 use uuid::Uuid;
 
 mod alert;
+mod ereport;
 mod saga;
 
 const NO_ACTIVE_PROPOLIS_MSG: &str = "<no active Propolis>";
@@ -356,6 +358,8 @@ enum DbCommands {
     Disks(DiskArgs),
     /// Print information about internal and external DNS
     Dns(DnsArgs),
+    /// Query and display error reports
+    Ereport(ereport::EreportArgs),
     /// Print information about collected hardware/software inventory
     Inventory(InventoryArgs),
     /// Print information about physical disks
@@ -1494,6 +1498,9 @@ impl DbArgs {
                             &args,
                             token,
                         ).await
+                    },
+                    DbCommands::Ereport(args) => {
+                        cmd_db_ereport(&datastore, &fetch_opts, &args).await
                     }
                 }
             }