if the sled agent reboots, it takes out the internal dns zone, leading to a panic

[jmpesp]

In debugging another issue, the sled agent panicked. It then tore down the internal dns zone on startup:

[ May 12 20:52:56 Stopping because all processes in service exited. ]
[ May 12 20:52:56 Executing stop method (:kill). ]
[ May 12 20:52:56 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
[ May 12 20:52:56 Method "start" exited with status 0. ]
note: configured to log to "/dev/stdout"
20:52:56.888Z INFO SledAgent: Starting mg-ddm service
20:52:56.889Z INFO SledAgent: Importing mg-ddm service
    path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
20:53:03.204Z INFO SledAgent: Setting mg-ddm interfaces
    interfaces = ("ixgbe0/ll" "net1/ll")
20:53:03.259Z INFO SledAgent: Enabling mg-ddm service
20:53:03.270Z INFO SledAgent: detecting (real or simulated) SP
20:53:03.270Z INFO SledAgent: setting up bootstrap agent server
20:53:03.317Z INFO SledAgent: Sending prefix to ddmd for advertisement
    DdmAdminClient = [::1]:8000
    prefix = Ipv6Prefix { addr: fdb0:1b:21c1:fd24::, len: 64 }
20:53:03.337Z WARN SledAgent: Deleting existing zone
    zone_name = oxz_internal_dns

Later on, it panicked because it couldn't resolve an internal address:

20:53:08.955Z WARN SledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view
20:53:08.955Z INFO SledAgent (BootstrapAgent): new DNS resolver
    addresses = [[fd00:1122:3344:1::1]:53]
20:53:08.956Z INFO SledAgent (BootstrapAgent): Ensuring service zone is initialized: InternalDns
20:53:08.964Z INFO SledAgent (BootstrapAgent): Service zone oxz_internal_dns does not yet exist
20:53:08.976Z INFO SledAgent (BootstrapAgent): Configuring new Omicron zone: oxz_internal_dns
20:53:08.997Z INFO SledAgent (BootstrapAgent): Installing Omicron zone: oxz_internal_dns
20:53:12.209Z INFO SledAgent (BootstrapAgent): Zone booting
    zone = oxz_internal_dns
20:53:17.704Z INFO SledAgent (BootstrapAgent): Ensuring address fd00:1122:3344:1::1 exists
20:53:17.704Z INFO SledAgent (BootstrapAgent): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:1::1, prefix: 64 }))
    zone = oxz_internal_dns
20:53:18.274Z INFO SledAgent (BootstrapAgent): Ensuring address fd00:1122:3344:1::1 exists - OK
20:53:18.274Z INFO SledAgent (BootstrapAgent): GZ addresses: [
        fd00:1122:3344:1::2,
    ]
20:53:18.274Z INFO SledAgent (BootstrapAgent): Ensuring GZ address fd00:1122:3344:1::2 exists
20:53:18.286Z INFO SledAgent (BootstrapAgent): Sending prefix to ddmd for advertisement
    DdmAdminClient = [::1]:8000
    prefix = Ipv6Prefix { addr: fd00:1122:3344:1::, len: 64 }
20:53:18.569Z INFO SledAgent (BootstrapAgent): Setting up internal-dns service
20:53:18.673Z INFO SledAgent (BootstrapAgent): Ensuring service zone is initialized: Ntp
20:53:18.680Z INFO SledAgent (BootstrapAgent): Service zone oxz_ntp does not yet exist
20:53:23.682Z INFO SledAgent (BootstrapAgent): Monitoring for hardware updates
20:53:23.682Z INFO SledAgent (BootstrapAgent): Performing full hardware scan
20:53:23.682Z INFO SledAgent (BootstrapAgent): Disabling switch zone (already complete)
sled-agent: Error managing sled agent: Could not start sled agent server: Error resolving DNS name: no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
[ May 12 20:53:23 Stopping because all processes in service exited. ]
[ May 12 20:53:23 Executing stop method (:kill). ]
[ May 12 20:53:23 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
[ May 12 20:53:23 Method "start" exited with status 0. ]

[smklein]

Cold boot doesn't work yet, see #3022 and https://docs.google.com/document/d/1bmOnm-LYEm4eh5RJCBHYHMwe044_bzt4oU2loBSv2no/edit?usp=sharing for more background