Bring back constituents to Hydra #340
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Ericson2314
force-pushed
the
aggregate-jobs
branch
from
91cd604 to
5b9412d
Compare
The part about finding `_hydraAggregate`/`constituents` is basically
derived from `hydra-eval-jobs`, however the part about
`namedConstituents` has been changed: we still stream out jobs when they
appear, however we suppress this for aggregate jobs.
These jobs are post-processed at the end, i.e. if `namedConstituents`
exist, these will be mapped to the drvPath of the other jobs. Then, the
drv will be rewritten to contain the drvPath of said jobs[1] and the
JSON containing the rewritten `drvPath` will be printed out.
[1] This was an optimization to reduce the memory footprint of
evaluating e.g. the `tested` job in nixpkgs.
(cherry picked from commit 76f32ed29e25b22d69467d28b72efcc9ac22e1bf)
Ericson2314
force-pushed
the
aggregate-jobs
branch
from
5b9412d to
bd5e93d
Compare
Ericson2314
pushed a commit
to NixOS/hydra
that referenced
this pull request
Nov 25, 2024
lf-
pushed a commit
to lix-project/lix
that referenced
this pull request
Mar 2, 2025
Hydra used to support aggregate jobs that only succeeded when their constituents succeed. This is still used by e.g. nixpkgs[1]. Prior art: * https://git.lix.systems/lix-project/nix-eval-jobs/pulls/17: got ported into the CppNix implementation[2] * nix-community/nix-eval-jobs#349: implements glob expressions for constituents - something we needed at work. This also restructures the code a bit which is what I re-used here. The globbing is not part of this patch. Essentially, the following things happen here (assuming `--constituents` is set): * Derivations with `_hydraAggregate = true;` are considered aggregates. These are not written to stdout when received by a worker, but stored until the end. * Constituents can be drv paths or strings (that must be the `attr` of another job). In that case, the derivation of the aggregate job is rewritten so that it depends on the drv of the constituent job. * At the very end the aggregate jobs are also written to stdout. Additionally, this fixes one bug, the old `hydra-eval-jobs` implementation had (and we actually hit at work): Given the leaf jobs `packages.foo` & `packages.bar`, an aggregate job `aggregate0` with _hydraAggregate = true; constituents = [ "packages.bar" "packages.foo" ]; and an aggregate job `aggregate1` with constituents = [ "aggregate0" ]; then it may happen depending on the order of evaluation that `aggregate1` depends on the old derivation of `aggregate0` (i.e. the one without rewritten constituents) and doesn't depend on `packages.foo` and `packages.bar` because it was rewritten before `aggregate0` was rewritten. This is done in here correctly, but topologically sorting the aggregate jobs before rewriting those. [1] https://github.com/NixOS/nixpkgs/blob/bba6b37c9d0898867a7d9c38a1b5b77efcfb07b9/nixos/release-combined.nix#L69 [2] nix-community/nix-eval-jobs#340 Change-Id: I5baad5e57336b4985ef8595e903814de83eb01c1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a commit from @Ma27 https://git.lix.systems/ma27/nix-eval-jobs/commit/5ce92e2c4d318b30edaafa4d8664b9e15e941f32 I rebased. Thanks @Ma27!
This (or something like it) is needed for
nix-eval-jobsto replacehydra-eval-jobs.The part about finding
_hydraAggregate/constituentsis basically derived fromhydra-eval-jobs, however the part aboutnamedConstituentshas been changed: we still stream out jobs when they appear, however we suppress this for aggregate jobs.These jobs are post-processed at the end, i.e. if
namedConstituentsexist, these will be mapped to the drvPath of the other jobs. Then, the drv will be rewritten to contain the drvPath of said jobs[1] and the JSON containing the rewrittendrvPathwill be printed out.[1] This was an optimization to reduce the memory footprint of
evaluating e.g. the
testedjob in nixpkgs.(cherry picked from commit 76f32ed29e25b22d69467d28b72efcc9ac22e1bf)