`build-dir` has "no space left on device" when `root-safe-haven` is too small

[MattSturgeon]

I'm unsure if this is something nothing-but-nix can solve or not, but this feels like the best place to discuss this either way.

I spend a few hours yesterday trying to get GitHub Actions to build my nix config, and nothing-but-nix has certainly helped with that. However while nothing-but-nix seems to be creating a lot of space in the nix-store, at least one package is running out of space in /build during its build.

The package in question is nexusmods-app-unfree, which only has a closure size of ~500MB, however due to the nixpkgs dotnet infrestructure probably uses more space than that during its build.

$ export NIXPKGS_ALLOW_UNFREE=1
$ nix path-info nixpkgs#nexusmods-app-unfree --impure --closure-size --human-readable
/nix/store/86chn7cga31mi4vqkq09rdm7s6px9zp6-nexusmods-app-unfree-0.10.2  553.4 MiB

Because it is an unfree package, it doesn't benefit from hydra caching. Therefore I was hoping my GitHub Actions workflow would cache it to my Cachix cache.

You can see it failing on several workflow runs, e.g. this one.

Switching to the cached "free" variant causes my workflow to complete successfully.

1. Is /build mounted separately, so it doesn't benefit from nothing-but-nix?
2. If so, could this change?
3. Is there a good way to inspect how much space is used by a package build?
4. And/or how much free space is available in /build?
5. Is this a known issue? If so should it be mentioned in the README?

Sidenote: I'm one of the package maintainers for nexusmods-app, so if this issue is indicating a packaging issue I'd also be interested in feedback for how to improve the package

[MattSturgeon]

I believe the underlying issue is NixOS/nixpkgs#54707 (also https://discourse.nixos.org/t/no-space-left-on-device-error-when-rebuilding-but-plenty-of-storage-available/43862/12).

I think nix's build-dir defaults to using TMPDIR, which is set to /home/runner/work/_temp by default. exporting TMPDIR=/nix/tmp or configuring build-dir = /nix/build solves the issue.

Writing directories under /nix feels wrong, but it does take advantage of the space reclaimed by nothing-but-nix.

The alternative is increasing root-safe-haven, since by default TMPDIR and build-dir will be on the root device. root-safe-haven: 25600 (25GB) seems to work, but I have no gauge of whether or not that's way too big or small than it needs to be.

Would nothing-but-nix want to do anything to ensure the build-dir has access to more disk space out of the box and/or in a more integrated manner?

[flexiondotorg]

@MattSturgeon Thanks for the detailed report. I have experienced this issue and consider it a bug in Nothing but Nix.

Here's a workaround while I figure out a proper fix: enable the holster "Hatchet Protocol". This will prevent claiming any space from / but will create a ~65GB /nix by reclaiming space from the /mnt partition.

- uses: wimpysworld/nothing-but-nix@main
  with:
    hatchet-protocol: 'holster'

[flexiondotorg]

Writing directories under /nix feels wrong, but it does take advantage of the space reclaimed by nothing-but-nix.

For a build in CI, using /nix for the build-dir doesn't make me 🤮 too much.

The alternative is increasing root-safe-haven, since by default TMPDIR and build-dir will be on the root device. root-safe-haven: 25600 (25GB) seems to work, but I have no gauge of whether or not that's way too big or small than it needs to be.

When nothing-but-nix is enabled in your workflow, a post-action dumps a df report so you can see how much space was used. I think I should make this a bit more specific and useful for the directories we'd care about for Nix.

Would nothing-but-nix want to do anything to ensure the build-dir has access to more disk space out of the box and/or in a more integrated manner?

Absolutely 👍

[MattSturgeon]

@flexiondotorg thanks for the fix! 🎉

[ryand56]

I'm still having this issue when using the action with building my flake for some reason:

error: Cannot build '/nix/store/03qwfg3alxsr7ghxgsi8rqgm5an24nxn-wrangler-pnpm-deps.drv'.
       Reason: builder failed with exit code 1.
       Output paths:
         /nix/store/srgnfq5sx0smz61wdfxklgayfh83ccyq-wrangler-pnpm-deps
       Last 25 log lines:
       > Progress: resolved 2199, reused 3, downloaded 1777, added 1274
       > Progress: resolved 2199, reused 3, downloaded 1794, added 1274
       > Progress: resolved 2199, reused 3, downloaded 1809, added 1274
       > Progress: resolved 2199, reused 3, downloaded 1819, added 1274
       > Progress: resolved 2199, reused 3, downloaded 1844, added 1274
       > Progress: resolved 2199, reused 3, downloaded 1869, added 1274
       > Downloading typescript@5.3.3: 0.00 B/5.76 MB
       > Downloading typescript@5.3.3: 15.78 kB/5.76 MB
       > Progress: resolved 2199, reused 3, downloaded 1884, added 1274
       > Progress: resolved 2199, reused 3, downloaded 1885, added 1274
       > Progress: resolved 2199, reused 3, downloaded 1895, added 1274
       > Downloading typescript@5.3.3: 5.76 MB/5.76 MB, done
       > Progress: resolved 2199, reused 3, downloaded 1895, added 1274
       > Progress: resolved 2199, reused 3, downloaded 1897, added 1274
       > Progress: resolved 2199, reused 3, downloaded 1905, added 1274
       > Progress: resolved 2199, reused 3, downloaded 1932, added 1274
       > Downloading typescript@3.9.10: 0.00 B/9.58 MB
       > Downloading typescript@3.9.10: 16.38 kB/9.58 MB
       > Progress: resolved 2199, reused 3, downloaded 1956, added 1274
       > Progress: resolved 2199, reused 3, downloaded 1997, added 1274
       > Downloading typescript@3.9.10: 9.58 MB/9.58 MB, done
       > Progress: resolved 2199, reused 3, downloaded 1997, added 1274
       > Progress: resolved 2199, reused 3, downloaded 2016, added 1274
       >  ERR_PNPM_ENOSPC  ENOSPC: no space left on device, copyfile '/nix/store/srgnfq5sx0smz61wdfxklgayfh83ccyq-wrangler-pnpm-deps/v3/files/dc/5ee837bbb416b4618f7c814a2dbb16148634506e5d75f6ac4d415d6e8b055185d258b079e87183fc9fbbc577a435895564b1232fc24733bd0ace0cfc3e6159' -> '/build/source/node_modules/.pnpm/ajv@8.13.0/node_modules/ajv_tmp_79/dist/refs/json-schema-2020-12/index.d.ts'
       > Progress: resolved 2199, reused 3, downloaded 2016, added 1274
       For full logs, run:
         nix log /nix/store/03qwfg3alxsr7ghxgsi8rqgm5an24nxn-wrangler-pnpm-deps.drv
       note: build failure may have been caused by lack of free disk space
error: Cannot build '/nix/store/h947nyjpy94jw1xjl0f1w6p3rxa4nzvm-wrangler-4.19.0.drv'.
       Reason: 1 dependency failed.
       Output paths:
         /nix/store/f4k6l7gcr9vqbpcfj9x0lhygvmp1hqkg-wrangler-4.19.0

[MattSturgeon]

I'm still having this issue when using the action with building my flake for some reason:

This workflow run? https://github.com/ryand56/wrangler/actions/runs/15426194467/job/43413616320

---

Testing with cachix disabled, I can also reproduce still reproduce the issue: https://github.com/MattSturgeon/nix-config/actions/runs/15426768259/job/43415590103?pr=57 😢

[flexiondotorg]

I see the issue. From your logs:

>  ERR_PNPM_ENOSPC  ENOSPC: no space left on device, copyfile '/nix/store/srgnfq5sx0smz61wdfxklgayfh83ccyq-wrangler-pnpm-deps/v3/files/dc/5ee837bbb416b4618f7c814a2dbb16148634506e5d75f6ac4d415d6e8b055185d258b079e87183fc9fbbc577a435895564b1232fc24733bd0ace0cfc3e6159' -> '/build/source/node_modules/.pnpm/ajv@8.13.0/node_modules/ajv_tmp_79/dist/refs/json-schema-2020-12/index.d.ts'

This is a guess for now, but it looks like /build is configured as the build-dir, and space on / is being exhausted. It is also odd that the notthing-but-nix post-action reports this:

Space used by TMPDIR:
4.0K	/mnt/tmp.6CNxGD6esJ
4.0K	total

That implies something is overwriting $TMPDIR or otherwise configuring build-dir, since nothing is written to where nothing-but-nix is directing $TMPDIR to. I use the DeterminateSystems/nix-installer-action for my NixOS builds and TMPDIR, as set by nothing-but-nix, is being honored.

Space used by TMPDIR:
26M	/mnt/tmp.fdxK8VJPF4
26M	total

I'll look into this further, starting with an update that captures some Nix configuration information and additional disk metrics to help determine precisely what's happening.

However, here's a workaround for you, and it should look familiar 😃. Using the holster Hatchet Protocol should ensure everything works.

- uses: wimpysworld/nothing-but-nix@main
  with:
    hatchet-protocol: 'holster'

This will leave / untouched with ~20GB of free space, while /nix will get created via sparse disk images in /mnt, providing ~65 GB. Hopefully, that should be a sufficient split for your builds to complete.

[ryand56]

Thanks, I'll test this when I get a chance 🙂