snarf ereports into service bundles

[hawkw]

PR #8269 added CRDB tables for storing ereports received from both service processors and the sled host OS. These ereports are generated to indicate a fault or other important event, so they contain information that's probably worth including in service bundles. So we should do that.

Ereports in CRDB are formatted as JSON. It's worth some additional thought to work out how we might best organize them in the service bundle for easy analysis.

Depending on the scope of the service bundle, we may only want to include ereports that were collected by Nexus within some time window. Eventually, we may also consider only including ereports from particular serial numbers, if we implement more "targeted" service bundles that only contain data from a particular system.

[hawkw]

cc @smklein — would love your input on this!

[smklein]

Especially if you're pulling data from the database, or "trying to add something to the bundle from Nexus' point-of-view", the place to start is: https://github.com/oxidecomputer/omicron/blob/main/nexus/src/app/background/tasks/support_bundle_collector.rs

Specifically, the body of collect_bundle_as_file is really where we collect the entire support bundle in a temporary directory. You shouldn't need to care about any lifetime stuff or storage management for the bundle itself.

See: https://github.com/oxidecomputer/omicron/blob/main/nexus/src/app/background/tasks/support_bundle_collector.rs#L635-L714

This is all a method on the BundleCollection struct:

https://github.com/oxidecomputer/omicron/blob/main/nexus/src/app/background/tasks/support_bundle_collector.rs#L448-L457

which contains the datastore, if you want to query the db for info (e.g., ereports) and also the BundleRequest, which is where we've started adding some controls for "what data should we include in the bundle". (This isn't exposed through the API yet, but we're using it for tests).

Seems reasonable to me to add a query to "get the ereports within some time window", and shove them into the bundle.

The output of the bundle in dir is transformed into a zipfile, but it's pretty arbitrary - we have a directory for each "sled-specific" piece of info (keyed by sled UUID) but we could possibly make an ereports/ directory for emitting errors too.