Should we deprecate building musl binaries on Glibc based systems?

[henderkes]

@crazywhalecc It would decrease our maintenance burden to maintain two different build systems (alpine gcc + musl gcc) for the same result. Ubuntu/RHEL can use docker or podman with bin/spc-alpine-docker to build.

It would allow us to move away from the whole SPC_LIBC logic, we wouldn't need our gcc package anymore, we don't need the whole environment setting logic in Linuxbuilder and it would make usage with glibc easier, too.

[crazywhalecc]

It seems easy to just remove musl-toolchain, but we lose the ability to build pure static binaries on the host. This seems to have little to do with SPC_LIBC. Even if we no longer support musl-toolchain, we will need SPC_LIBC to distinguish musl-gcc from glibc-gcc, and it is conceivable that our dependence on gcc is more specific.

It's like going back to the original bash-version branch, simple but can't do much. Since we now support these multiple features, and the Linux ecosystem does not seem to be unified for the time being, there is no big benefit for us to abandon SPC_LIBC-something that we hate but can only compromise.

I won't be completely for or against this decision, but it's not worth it at this point.

[henderkes]

It seems easy to just remove musl-toolchain, but we lose the ability to build pure static binaries on the host.
Technically yes, but with docker or podman nothing changes. We'd still maintain alpine support. With bin/spc-alpine-docker an ubuntu system can still build a fully static version.

This seems to have little to do with SPC_LIBC.

We wouldn't need the environment variable anymore and all the checks it introduces. We'd just always build against the system libc - musl on alpine, glibc on rebian/rhel, macos framework on mac.

I won't be completely for or against this decision, but it's not worth it at this point.

I'm fine with keeping it for now, but there's very little point to it, I feel like. All the projects using spc to build musl targeting binaries that I'm aware of use alpine docker on their own. We wouldn't lose any features there.

I'll spin up a PR that we don't have to merge yet. It's just something to consider.

[henderkes]

At least judging by the downloads, our hosted musl-toolchain has more downloads than the pkg-config source code

That doesn't surprise me one bit. Everyone who uses spc and doesn't immediately set an environment variable SPC_LIBC to glibc will inevitably download the musl toolchain. I have it downloaded on most of my builder systems and only ever use it when the CI fails on 24.04 but succeeds on ubuntu-latest.

When I created the musl toolchain, we didn't offer glibc builds at all. Now we do - and it's pretty much the recommended path if complete portability is not necessary. Performance and stability of glibc, even built on ancient Centos 7, is much better than on musl.

What I would much rather do, instead of maintaining cross-compiling musl on a glibc dist, is to offer shared musl builds (in the same way that glibc builds) - everything linked statically, except libc, so we can use dlopen with shared extensions.

[crazywhalecc]

I don't mind glibc becoming a first-class citizen, but at least this feature should not be removed, and it should be possible to build glibc-compatible binaries on glibc-based distributions without docker.

[henderkes]

I don't mind glibc becoming a first-class citizen, but at least this feature should not be removed, and it should be possible to build glibc-compatible binaries on glibc-based distributions without docker.

It is! Rhel 9 and 10 required almost zero patches, Most patches you see for glibc only exist because of the centos 7 build. The old compiler and system libraries have various issues that php stopped caring about a long time ago. Their official stance is that people who want to build for older systems than 2.34 will have to patch php to keep compatibility.

That's also why I finished rhel 9 & rhel 10 packages before the rhel 7 ones.

[crazywhalecc]

The core problem is that the benefits of removing it are not so obvious. Most builds need to work out of the box and don't care what libc we used, which is why we provide spc binaries in 2.0. Not only is it a demonstration of complete portability, but it is also a tool that make php code into one executable. Once first-class citizens rely on docker, almost all projects that previously relied on spc will be unusable (or lack of portability). So even if we all agree to do this, it should be a gradual process, at least for 3.0.

But what we can do now is to focus on more stable gnu builds for developers or downstream projects who want better performance or dynamically loaded extensions. Rather than saying that maintaining musl-toolchain is difficult, it is better to say that Linux itself is a very fragmented system.

[henderkes]

Once first-class citizens rely on docker, almost all projects that previously relied on spc will be unusable (or lack of portability).

I agree, but all projects that rely on SPC that I'm aware of are already using alpine linux environments to compile. I'm not aware of any project that actually builds fully static musl binaries on ubuntu.

So even if we all agree to do this, it should be a gradual process, at least for 3.0.

100% agree, which is why I'm not proposing we remove it right now. I would say we deprecate building musl binaries on a glibc platform now with a warning, so people have ample time to make the switch to either build on alpine or produce glibc builds instead.

If we don't do it now, we'll have to maintain two different options of building fully static binaries, one of which is associated with our self hosted musl toolchain. At least until 4.0, which is far, far, far in the future.

We currently maintain MacOs, Linux glibc, Linux musl and Linux glibc with musl wrapper, which produces the same binaries as Alpine Linux. I'd only deprecate relying on our musl wrapper toolchain to build static musl binaries on ubuntu. Until 3.0 it will still be supported by us, after that people would have to either a) use alpine or b) use a musl toolchain (such as ours) but configure their PATH, CC, CXX, AR and LD manually instead of relying on spc to do it for them.

[crazywhalecc]

I will try crosstool-ng when I have time. Maybe this can fundamentally solve the problem of maintaining the cross toolchain. If we can easily build different proper toolchains, then the problem of maintaining toolchains is solved - we only need to continue to build toolchains that match our ideal environment. And it's also benefits the docker environment.

but all projects that rely on SPC that I'm aware of are already using alpine linux environments to compile. I'm not aware of any project that actually builds fully static musl binaries on ubuntu.

Individual who use SPC personally use SPC binary to build, and most of them do not use Docker. If all projects really use Docker, then the SPC binary and musl-toolchain should not have so many downloads.

But in short, it's not easily advance the start of 3.0 because we have not been finding a solution or finishing full-investigation that can be used out of the box and does not rely on docker to build static binaries on gnu. Also we have a lot of issue that not closed. Rather than maintaining a Docker distribution, it is more far-sighted to maintain a long-term, cross-platform tool chain (if possible).

[henderkes]

If you're looking at cross toolchains, you may want to have a look a Zig. It can cross compile to any architecture and libc. For example CC="zig cc -target x86_64-linux-gnu.2.17".

I tried it but it had problems with some libraries so I stopped, because I want to advance the package repositories instead.

[henderkes]

Oh, I think I found why some libraries ran into issues when I tried it. When compiling for a different libc version, we need to set -L/path/to/zig/glibcversion before it sees the system paths.

[henderkes]

No, that wasn't the issue, it was already made default since the post.

One issue was apparently different error sensitivity of Clang and gcc. The other issue was that cc and cxx having spaces creates weird errors.

I solved it by creating:

zig-cc:

#!/usr/bin/env bash
zig cc $@

and likewise for zig-c++.

[henderkes]

this has real potential. I was able to build php with curl (all suggested libs) for glibc 2.39, 2.17 and musl all from my rhel 10 machine. 2.17 needed some patches for proper header versioning (I will commit those upstream). This could simplify spc a lot.

Except that we would use clang (zig) instead of gcc, I guess, but we already do that on mac.