Skip to content

[sled-agent] Unify requests for services and services backed by datasets #3026

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 18 commits into from

Conversation

smklein
Copy link
Collaborator

@smklein smklein commented May 5, 2023

Previously, the sled agent had to request for "services using datasets" and "services not using datasets" with different APIs - one acting on /filesystem, and one acting on /services. These APIs evolved independently, even though they have significant overlap in utility.

By merging the two of them together, several aspects of service management are simpler:

  • Storing "config for what services should run" can be stored in a single config file on the M.2s
  • "The set of every service that should run on a sled" can be tagged with a generation number by Nexus
  • A smaller API surface is exposed, with less overlap

TODO: I think this will be blocked on:

@smklein smklein changed the title [sled-agent] Unify requests for services and datasets [sled-agent] Unify requests for services and services backed by datasets May 5, 2023
smklein and others added 16 commits May 5, 2023 11:46
Some minor docs improvements, as I dug through this codebase recently:

- I wanted to clarify, "Does this CTE only provision Crucible datasets
to U.2s, and not M.2s?". The answer to this question is: "yes, but
implicitly decided when we provision the datasets themselves". Added a
comment to clarify this during region selection.
- Additionally, when referencing a virtual table from an earlier `WITH
candidate_zpools AS ...` arm of the CTE, make that reference explicit.
This is a no-op change, but it clarifies the dependencies within the
CTE.
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.96 to
1.0.97.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/serde-rs/json/releases">serde_json's
releases</a>.</em></p>
<blockquote>
<h2>v1.0.97</h2>
<ul>
<li>Add <code>io_error_kind()</code> method to serde_json::Error:
<code>fn io_error_kind(&amp;self) -&gt;
Option&lt;std::io::ErrorKind&gt;</code> (<a
href="https://redirect.github.com/serde-rs/json/issues/1026">#1026</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/serde-rs/json/commit/a0ddb25ff6b86f43912f8fc637797bcbb920c61e"><code>a0ddb25</code></a>
Release 1.0.97</li>
<li><a
href="https://github.com/serde-rs/json/commit/8b681ff8d1ee25c7d7c1040e16b1e23b6038d4b5"><code>8b681ff</code></a>
Merge pull request <a
href="https://redirect.github.com/serde-rs/json/issues/1026">#1026</a>
from dtolnay/errorkind</li>
<li><a
href="https://github.com/serde-rs/json/commit/9308d97b3d9e729438ccaae7506b054c157b708b"><code>9308d97</code></a>
Add Error::io_error_kind</li>
<li><a
href="https://github.com/serde-rs/json/commit/136b773f5355cbdfa326c2627b812055b51ba3aa"><code>136b773</code></a>
Merge pull request <a
href="https://redirect.github.com/serde-rs/json/issues/1025">#1025</a>
from dtolnay/io</li>
<li><a
href="https://github.com/serde-rs/json/commit/207a57b68880769c81d525f9f5b38d3be1340806"><code>207a57b</code></a>
Standardize on &quot;I/O&quot; instead of &quot;IO&quot;</li>
<li><a
href="https://github.com/serde-rs/json/commit/6fda3853e4b5cd2d6e2e7cc773f453999498a8cb"><code>6fda385</code></a>
Merge pull request 1021 from ndmitchell/patch-2</li>
<li><a
href="https://github.com/serde-rs/json/commit/009a53b472017dbec18442611fb849b9801db41d"><code>009a53b</code></a>
Switch to using null</li>
<li><a
href="https://github.com/serde-rs/json/commit/931ee23b1abf6b302df7c20a3a5866bf1a8a33ca"><code>931ee23</code></a>
Show error details during miri setup in CI</li>
<li><a
href="https://github.com/serde-rs/json/commit/0d7b0d3c18166bb65643afda9651e962b6f03d55"><code>0d7b0d3</code></a>
Add an example of producing a Null in a json! literal</li>
<li>See full diff in <a
href="https://github.com/serde-rs/json/compare/v1.0.96...v1.0.97">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=serde_json&package-manager=cargo&previous-version=1.0.96&new-version=1.0.97)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 ci: use --netrc for downloading hubris artifacts as well
An actions scope is needed to download artifacts from the Github API.
Pass `--netrc` to use the buildomat provided API key as we do for the
other curl invocations.

 ci: tuf-repo: harden curl invocations
Some useful flags for curl in a scripted context:
  -f    fail on server errors
  -s    silent mode
  -S    show error on failure
  -L    follow redirects
Updated dendrite to commit 3857dac.

---------

Co-authored-by: reflector[bot] <130185838+reflector[bot]@users.noreply.github.com>
RSS wants to know the bootstrap addresses of the sleds to include; this
PR teaches wicketd how to find them. It also updates the wicket TOML
config introduced in #3326 to include the sled's bootstrap address (if
known) in a comment in the sled list; I'm unsure if this is something
we'll want to keep once customers are running RSS, but it seems handy
for debugging for now.

Builds on #3326.
smklein added a commit that referenced this pull request Jun 29, 2023
…oo (#3390)

## Prelude 

This combines:

- #3022
- #3025
- #3026

Into one PR. Trying to split them out resulted in intermediate states
that were not worthwhile debugging - I'd rather invest the energy into
fixing this "more-desirable end-state".

## What does this PR do

- (Internal and External) DNS services are explicitly backed by
datasets, making it possible to "delete and re-initialize" arbitrary
zones without data loss. This is necessary for correct behavior in cold
boot.
- Merges the concept of "datasets" into "services".
- Previously, "datasets" were considered services managing data, and
"services" were considered services *without datasets* - the two were
totally disjoint. This was a bit arbitrary, and made it more complicated
to conform sleds during service management.
- As one example of an inconsistency, "requesting services" was
vectorized ("ask for all services that should be on the sled"),
"requesting datasets" was not (it was an ask-for-one-at-a-time API).
- Now, "services" are universal, and they may optionally manage
datasets. This unifies pathways a bit, and simplifies the API.

Fixes #3018
Part of #2978
@smklein
Copy link
Collaborator Author

smklein commented Jul 4, 2023

Merged in #3390

@smklein smklein closed this Jul 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants