ci: split tests into chunks

[khaneliman]

Similar goal of #6539 but applying it to the actual tests themselves for parallelization. Again, blocked by being able to change our checks ( I can't do that )

We have lots of tests and would like to add more. However, adding more testing coverage comes at the cost of a slower CI when we run them sequentially. This adds test outputs that are chunked however we'd like to tune for batch sizes. Allowing us to create a parallelized CI workflow.

Been testing in my fork and seems to cut time in half https://github.com/khaneliman/home-manager/actions/runs/16084828656

Description

Checklist

• Change is backwards compatible.

• Code formatted with nix fmt or
  nix-shell -p treefmt nixfmt-rfc-style deadnix keep-sorted --run treefmt.

• Code tested through nix-shell --pure tests -A run.all
  or nix build --reference-lock-file flake.lock ./tests#test-all using Flakes.

• Test cases updated/added. See example.

• Commit messages are formatted like

  {component}: {description}
  
  {long description}
  

  See CONTRIBUTING for more information and recent commit messages for examples.

• If this PR adds a new module

  • Added myself as module maintainer. See example.

Maintainer CC

[zowoq]

Have you looks at using buildbot for the tests? After the initial run I'd expect it to be quick as most tests would be cached. Darwin would be a bit bottlenecked so I'd suggest only testing aarch64-darwin and not x86_64-darwin.

[khaneliman]

Have you looks at using buildbot for the tests? After the initial run I'd expect it to be quick as most tests would be cached. Darwin would be a bit bottlenecked so I'd suggest only testing aarch64-darwin and not x86_64-darwin.

Yes, that's been something I have thought about but wasn't sure how to get set up in the checks here and everything there.

[zowoq]

As there is a separate flake.nix for the tests I'll need to implement flake_dir in buildbot but I think this diff is all that would need to be done in home-manager. I can configure buildbot to just run on a branch without enabling it for the entire repo so can see it working without causing any disruption in other PRs. I should be able to have this done in the next day or two.

diff --git a/buildbot-nix.toml b/buildbot-nix.toml
new file mode 100644
index 00000000..55433a03
--- /dev/null
+++ b/buildbot-nix.toml
@@ -0,0 +1,3 @@
+attribute = "packages"
+flake_dir = "./tests"
+lock_file = "./flake.lock"
diff --git a/tests/flake.nix b/tests/flake.nix
index 34f39d35..f5d183d0 100644
--- a/tests/flake.nix
+++ b/tests/flake.nix
@@ -20,7 +20,10 @@
   outputs =
     { nixpkgs, ... }:
     let
-      forAllSystems = nixpkgs.lib.genAttrs nixpkgs.lib.systems.flakeExposed;
+      forAllSystems = nixpkgs.lib.genAttrs [
+        "aarch64-darwin"
+        "x86_64-linux"
+      ];
     in
     {
       devShells = forAllSystems (

[khaneliman]

As there is a separate flake.nix for the tests I'll need to implement flake_dir in buildbot but I think this diff is all that would need to be done in home-manager. I can configure buildbot to just run on a branch without enabling it for the entire repo so can see it working without causing any disruption in other PRs. I should be able to have this done in the next day or two.

diff --git a/buildbot-nix.toml b/buildbot-nix.toml
new file mode 100644
index 00000000..55433a03
--- /dev/null
+++ b/buildbot-nix.toml
@@ -0,0 +1,3 @@
+attribute = "packages"
+flake_dir = "./tests"
+lock_file = "./flake.lock"
diff --git a/tests/flake.nix b/tests/flake.nix
index 34f39d35..f5d183d0 100644
--- a/tests/flake.nix
+++ b/tests/flake.nix
@@ -20,7 +20,10 @@
   outputs =
     { nixpkgs, ... }:
     let
-      forAllSystems = nixpkgs.lib.genAttrs nixpkgs.lib.systems.flakeExposed;
+      forAllSystems = nixpkgs.lib.genAttrs [
+        "aarch64-darwin"
+        "x86_64-linux"
+      ];
     in
     {
       devShells = forAllSystems (

I think in nixvim we ended up creating a separate flake output for buildbot itself so that we could keep the flake building on all platforms and only forward certain platforms to the ci.

https://github.com/nix-community/nixvim/blob/main/buildbot-nix.toml#L1-L1 looks like it lets us tell buildbot which outputs to use?

[zowoq]

Yes, can use different flake output. The ~1400 tests in packages is too much for buildbot but chunking them seems to be fine.

qowoz@60215a9

https://buildbot.nix-community.org/#/builders/4466/builds/5

I deployed the flake_dir change just to test this, still need to upstream it to buildbot-nix repo so it can be deployed again. nix-community/buildbot-nix#450

[Mic92]

Longterm buildbot-nix probably will not schedule "skipped builds" for repositories with a lot of flake outputs. I need to do some refactoring for updating gcroots though.

[zowoq]

The buildbot changes are deployed now if you'd like to move forward with that, note that the nix-community-buildbot label no longer works, instead the repo needs to be added to the list in the infra repo.

[khaneliman]

The buildbot changes are deployed now if you'd like to move forward with that, note that the nix-community-buildbot label no longer works, instead the repo needs to be added to the list in the infra repo.

Thanks, I'll try and do that within the next couple days.

[zowoq]

I've discovered that the buildbot method I used previously for testing only seems to work for the repo default branch so I can't test this PR directly, instead I've pushed the commits to master on my fork.

https://buildbot.nix-community.org/#/builders/4466/builds/17
https://buildbot.nix-community.org/#/builders/4466/builds/18

[khaneliman]

Looks like we have some checks that we dont normally run in CI that are failing. Will need to look into those.