Unnecessary permanent GC roots?

[B4dM4n]

Running nix-eval-jobs without specifying --gc-roots-dir creates a temporary folder, where a permanent GC root for each evaluated derivation is created. But since this folder is removed before nix-eval-jobs terminates, those GC roots are not really permanent and become dangling immediately.

Doing this against a large package set, like nixpkgs, will create around 100k GC roots per run, which can quickly grow into the millions when executed multiple times and exhaust the available filesystem inodes.

The temporary folder creation was introduced by #342, sadly without an explanation why it's necessary.

nix-eval-jobs/src/nix-eval-jobs.cc

Lines 468 to 471 in 4b392b2

	if (myArgs.gcRootsDir.empty()) {
	const nix::Path tmpDir = nix::createTempDir();
	gcRootsDir.emplace(tmpDir, true);
	myArgs.gcRootsDir = tmpDir;

There are currently two places where the --gc-roots-dir value is used:

nix-eval-jobs/src/worker.cc

Lines 237 to 249 in 4b392b2

	if (!args.gcRootsDir.empty()) {
	const nix::Path root =
	args.gcRootsDir + "/" +
	std::string(nix::baseNameOf(drv.drvPath));
	if (!nix::pathExists(root)) {
	auto localStore =
	state->store
	.dynamic_pointer_cast<nix::LocalFSStore>();
	auto storePath =
	localStore->parseStorePath(drv.drvPath);
	localStore->addPermRoot(storePath, root);
	}
	}

nix-eval-jobs/src/nix-eval-jobs.cc

Lines 573 to 576 in 4b392b2

	assert(!myArgs.gcRootsDir.empty());
	const nix::Path root =
	myArgs.gcRootsDir + "/" +
	std::string(nix::baseNameOf(newDrvPathS));

The first one checks if a value is present, which is currently always true. The second place asserts that a value is present.
But both just create dangling GC roots in the end.

Could the temporary folder creation be disabled again?

If it's necessary for --constituents processing, could it be changed, that it only happens when --constituents is given?

[Mic92]

I believe it was crashing when no gcroots directory was specified, but this might be false memory. If not, than I at least wouldn't warn if no gcroots directory is specifed.

[Mic92]

My question is, how are you using nix-eval-jobs, if you don't depend on the derivations to be present afterwards?

[B4dM4n]

I'm trying to switch nixpkgs-review to nix-eval-jobs for local evaluation (see Mic92/nixpkgs-review#473).

In this case, GC roots should not be necessary when evaluating the whole package-set, but only for the actual packages, which will be built (which the mentioned PR currently does not do).

If not, than I at least wouldn't warn if no gcroots directory is specifed.

I agree. If not specifying --gc-roots-dir is allowed, it shouldn't produce a warning.

[Mic92]

@B4dM4n I think this might be actually too slow, because nix-eval-jobs writes derivations to disk, which is a lot slower than what nix-env does in memory. Before putting too much time into this, please run time nix-eval-jobs and compare with nix-env for the same nixpkgs.

[B4dM4n]

I compared the two and considered nix-eval-jobs "fast enough" compared to nix-env in my tests.

I don't have the numbers how long nix-eval-jobs runs without the derivations present on disc, but if most are already present, nix-eval-jobs with one worker is "only" 3x slower than nix-env and it can be speed up by increasing the number of workers to the point, where it is equally as fast.

But the big difference is in memory consumption. Last time I checked, nix-env consumed close to 19GB RAM when evaluating nixpkgs on my machine, whereas with nix-eval-jobs this is configurable.