Unable to rebuild on certain architectures

[ocaisa]

For A64FX and Grace we have accounts on real systems which we use for building. However, those systems do not support fakeroot so we currently have no way to rebuild packages (affecting #1101 and #1102)

[bedroge]

One possible solution could be to bind mount /cvmfs from the host, and bind mount an empty host directory on top of it to replace an existing installation. That seems to work:

$ apptainer shell -B /cvmfs/software.eessi.io -B $PWD/emptydir:/cvmfs/software.eessi.io/versions/2023.06/software/linux/aarch64/a64fx/software/buildenv/default-foss-2023a docker://ghcr.io/eessi/build-node:debian11
INFO:    Using cached SIF image
Apptainer> ls /cvmfs/software.eessi.io/versions/2023.06/software/linux/aarch64/a64fx/software/buildenv/default-foss-2023a
Apptainer> touch /cvmfs/software.eessi.io/versions/2023.06/software/linux/aarch64/a64fx/software/buildenv/default-foss-2023a/newfile
Apptainer> 

You can't bind mount it as a new directory, then you get read-only file system errors, but for rebuilds that's not an issue, as we always want to replace an existing directory.

The only issue is that we do need to have /cvmfs on the host in this case, bind mounting on top of a /cvmfs that was fuse-mounted by the container does not seem to work. I guess all/most of our build cluster do have /cvmfs on the host, so it may not be an issue right now, but we always assumed that we may be building on nodes that don't have this.

[bedroge]

But I guess we could definitely add it as a second (and even the preferred) rebuild method, i.e. check if /cvmfs is available on the host and then use this approach. If it's not available, we could fall back to fakeroot (if support for that is available).

[boegel]

Do we know why fakeroot is not supported on Deucalion & JURECA? Is that deliberate?

Anyway, the approach where we leverage /cvmfs from the host if it's available seems reasonable to me.

[ocaisa]

It sounds like it is feasible, we build with an overlay and rebuild with a bind mount. Is there any bot where CVMFS is not available natively (Grace maybe)?

It should also make tarballs smaller as I believe right now we are also tarring up the CVMFS cache.

[boegel]

Just noting that the reason we're relying on fakeroot is to dance around the problems in #556

I'm slowly getting annoyed enough to try and come up with a solution for this once and for all, and I'm still hoping there's a way without having to rely on an EESSI mount provided by the host (since that comes with downsides as well).

[boegel]

Latest version of bubblewrap (bwrap) has introduced support for creating writable overlays, that looks promising...

Step 0:

• build and install bwrap from source in Debian 11 build container filesystem-layer#220

[boegel]

unionfs as replacement for fuse-overlayfs to create writable overlay looks more promising, see:

• add support to eessi_container.sh for using unionfs overlay tool software-layer-scripts#9
• use unionfs overlay tool (+ debian12 build container that includes it) by default software-layer-scripts#26