Approach described in WIKI page fails

[573]

Hi community,

I tried and tested an approach to use a remote builder inside a (qemu-spawned) vm via WSL (same physical machine).

Right now I got failures building any package though and search for hints fixing these.

I. e. I run the following command:

NIX_BUILD_TOP=/tmp/build nix-build '<nixpkgs>' -A hello --option system aarch64-linux --builders 'ssh://nixos-shell' -j0

and get:

these derivations will be built:
/nix/store/g25sqngd7wc02lqgrcwrlahcqpcsi8wv-binutils-wrapper-2.31.1.drv
/nix/store/ap88q7lwvjnyrh6a0r72kna2ra1ar9ng-gcc-wrapper-9.3.0.drv
/nix/store/k30s9nzcsg1q1l1sfr40f27gjr28y47a-findutils-4.7.0.drv
/nix/store/pjzbkzvgfhqs34ap0q5b2sb74b6nal0l-diffutils-3.7.drv
/nix/store/7j9nfaykry0jxpcz2abn6fy8dcgjic7j-stdenv-linux.drv
/nix/store/alscp83wvivaf8hn8yfak92izjbfxxbg-hello-2.10.drv
building '/nix/store/g25sqngd7wc02lqgrcwrlahcqpcsi8wv-binutils-wrapper-2.31.1.drv'...
building '/nix/store/pjzbkzvgfhqs34ap0q5b2sb74b6nal0l-diffutils-3.7.drv' on 'ssh://nixos-shell'...
waiting for locks or build slots...
�while setting up the build environment: executing '/nix/store/fj18xk7dwk706mfki61klg69dg84gsim-bootstrap-tools/bin/bash': Exec format error
resize: can't open terminal /dev/tty
builder for '/nix/store/g25sqngd7wc02lqgrcwrlahcqpcsi8wv-binutils-wrapper-2.31.1.drv' failed with exit code 1
cannot build derivation '/nix/store/7j9nfaykry0jxpcz2abn6fy8dcgjic7j-stdenv-linux.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/alscp83wvivaf8hn8yfak92izjbfxxbg-hello-2.10.drv': 1 dependencies couldn't be built
error: build of '/nix/store/alscp83wvivaf8hn8yfak92izjbfxxbg-hello-2.10.drv' failed

As you see the build is actually spawned but there seems to be a problem with the binfmt_wrapper (I suspect bash is not aarch64 but tries to run aarch64 stuff - EDITED bash is aarch64, see #102 (reply in thread)) and IDK exactly how to trace if it is spawned anyway or how to debug the "Exec format error" line.

Might somebody help with this ?

[573]

All I can say for now is I can run:

/run/binfmt/aarch64 /nix/store/fj18xk7dwk706mfki61klg69dg84gsim-bootstrap-tools/bin/bash

when logged in nixos-shell (the remote machine used as a builder).
But I still have no clue how to put that together.

[t184256]

Just generic troubleshooting: what architecture is that bash? (file /nix/store/fj18xk7dwk706mfki61klg69dg84gsim-bootstrap-tools/bin/bash). Does invoking it manually lead to the same error? Does inspecting /proc/sys/fs/binfmt_misc reveal anything helpful?

[573]

I got

cat /proc/sys/fs/binfmt_misc/aarch64-linux
enabled
interpreter /run/binfmt/aarch64-linux
flags:
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00ffffffffffff00fffeffffff
cat /proc/sys/fs/binfmt_misc/status
enabled
nix run nixpkgs.file -c file /nix/store/fj18xk7dwk706mfki61klg69dg84gsim-bootstrap-tools/bin/bash
/nix/store/fj18xk7dwk706mfki61klg69dg84gsim-bootstrap-tools/bin/bash: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/fj18xk7dwk706mfki61klg69dg84gsim-bootstrap-tools/lib/ld-linux-aarch64.so.1, for GNU/Linux 2.6.32, stripped

let's see what I can do with that.

[573]

nix show-derivation /nix/store/alscp83wvivaf8hn8yfak92izjbfxxbg-hello-2.10.drv

{
  "/nix/store/alscp83wvivaf8hn8yfak92izjbfxxbg-hello-2.10.drv": {
    "outputs": {
      "out": {
        "path": "/nix/store/hk6gnq348j98pdakq7lsrqvyrdkslq18-hello-2.10"
      }
    },
    "inputSrcs": [
      "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
    ],
    "inputDrvs": {
      "/nix/store/3vffbnail1g7l8n012ran0ndn5d1a1jc-hello-2.10.tar.gz.drv": [
        "out"
      ],
      "/nix/store/7j9nfaykry0jxpcz2abn6fy8dcgjic7j-stdenv-linux.drv": [
        "out"
      ],
      "/nix/store/l2ivff0i9xzvl2hn0n1zf2mdql0f8bl9-bash-4.4-p23.drv": [
        "out"
      ]
    },
    "platform": "aarch64-linux",
    "builder": "/nix/store/mbfv4dfdyn06fvc41ri03byi7q82b5a3-bash-4.4-p23/bin/bash",
    "args": [
      "-e",
      "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
    ],
    "env": {
      "buildInputs": "",
      "builder": "/nix/store/mbfv4dfdyn06fvc41ri03byi7q82b5a3-bash-4.4-p23/bin/bash",
      "configureFlags": "",
      "depsBuildBuild": "",
      "depsBuildBuildPropagated": "",
      "depsBuildTarget": "",
      "depsBuildTargetPropagated": "",
      "depsHostHost": "",
      "depsHostHostPropagated": "",
      "depsTargetTarget": "",
      "depsTargetTargetPropagated": "",
      "doCheck": "1",
      "doInstallCheck": "",
      "name": "hello-2.10",
      "nativeBuildInputs": "",
      "out": "/nix/store/hk6gnq348j98pdakq7lsrqvyrdkslq18-hello-2.10",
      "outputs": "out",
      "patches": "",
      "pname": "hello",
      "propagatedBuildInputs": "",
      "propagatedNativeBuildInputs": "",
      "src": "/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz",
      "stdenv": "/nix/store/s0lxh9yimwkjhi7kqpqp1hyx7p30p429-stdenv-linux",
      "strictDeps": "",
      "system": "aarch64-linux",
      "version": "2.10"
    }
  }
}

[573]

@Mic92, do you have an idea ?

[Mic92]

sorry, no. I have not used binfmt before. I only know the mask must match the elf header in binfmt.

[573]

Having

NIX_BUILD_TOP=/tmp/build nix-build '<nixpkgs>' -A hello --argstr system aarch64-linux --option sandbox false --builders 'ssh://nixos-shell aarch64-linux' -j0

invokes everything as it should in the remote vm, i. e.

22010 nixbld1 20 0 163M 15052 10364 S 0.0 3.1 0:05.01 /run/binfmt/aarch64-linux /nix/store/fj18xk7dwk706mfki61klg69dg84gsim-bootstrap-tools/bin/bash /nix/store/dlz5svx0fwi8mwq5bwdzc7bld7md7ahj-bootstr...

I'm trying to recollect my setup so far here in the discussion and will for now as well delete the WIKI page until things cleared.