Rewrite release CI

[hasufell]

PR at #11072

Motivation

1. have a reliable release CI that runs (mostly) on ephemeral, stateless runners
2. have access to GitHubs CI infrastructure for faster and more reliable execution
3. unify the CI so that we only need to maintain one place/flavor of CI
4. improve bus factor and make contribution/adjustments to release CI easier
5. reduce the gap between GHCup's and upstream's CI, possibly allowing to ship vanilla bindists in GHCup once again

Design goals

• run the integration test as part of the release (since cabal-install is the main binary and the build config diverges from the validate pipeline)
• use a re-usable workflow, so that it can be triggered from other workflows (e.g. just to run build+integration test)
• support FreeBSD via a HF sponsored private runner (managed by me)
• avoid depending on custom docker images (we only do this for linux-i386 and linux-arm at the moment, because GitHub doesn't support node for those platforms, complicating the job description and setup a bit)

Supported platforms

The current platform build matrix is excessive and hard to maintain. It requires constant tracking of EOLed and new distro versions.

Instead, we propose to reduce the linux builds to only 2:

• one fully statically linked against musl
• one dynamically linked against the oldest glibc that's not EOL, but statically against other libraries (gmp, ncurses etc)... this works via pointing --extra-lib-dirs to a dir which includes only static library archives (cabal only seems to link against gmp and I have a working example binary)

Then all known glibc based distros can use the glibc one and alpine/unknown distros use the static musl one.

We end up with the following build environments:

Build env	Remark
x86_64-linux-rocky8	glibc
x86_64-linux-alpine3_22	musl
x86_64-freebsd14_2
x86_64-darwin
x86_64-windows
aarch64-linux-rocky8	glibc
aarch64-linux-alpine3_22	musl
aarch64-darwin
i386-linux-deb11	glibc
i386-linux-alpine3_22	musl

[ffaf1]

one fully statically linked against musl

What are the drawbacks of using musl? Perfomance? Size? What else?

one dynamically linked against the oldest glibc that's not EOL, but statically against other libraries (gmp, ncurses etc)

How would that work if the user has not got that specific glibc?

---

In general reducing everything-linux to two builds seems too good to be true. There have to be some disadvantages to this approach, right?

[hasufell]

There's a plethora of discussions wrt musl vs glibc. This page also includes some further links: https://edu.chainguard.dev/chainguard/chainguard-images/about/images-compiled-programs/glibc-vs-musl/

The main one is probably DNS issues:

• https://wiki.musl-libc.org/functional-differences-from-glibc.html#Name-Resolver/DNS
• DNS over TCP was only recently implemented: https://www.openwall.com/lists/musl/2023/05/02/1

On some platforms, musl linked Haskell binaries seem to have mysterious segfaults: https://gitlab.haskell.org/ghc/ghc/-/issues/25945

Generally, other toolchains follow a similar pattern: https://forge.rust-lang.org/infra/other-installation-methods.html

You could argue that if a distro is glibc based, it's also safe to assume the user would prefer glibc linked binaries. Yes, they are a tad less portable.

How would that work if the user has not got that specific glibc?

If the user has a newer glibc, then everything is fine. Only if they have an older glibc, they could not execute the binary. Since we use rockylinux8 with the oldest glibc of a non-EOL distro, it's very hard to maneuver yourself into that state. But it is possible.

---

I mean, this isn't anything new. We do this already. We already provide glibc based bindists in the current release CI. That doesn't change. The only thing that changes is that we provide only one glibc... because we start linking libgmp statically.

So the only new drawback would be: libgmp is now statically linked. Static vs dynamic linking has the usual differences: to update a statically linked library, you have to rebuild your binary. But it's more portable. Etc. etc.

[hasufell]

Wrt DNS: this might be irrelevant to cabal, since it generally uses curl/wget for fetching.

[ffaf1]

Thanks!

So the only new drawback would be: libgmp is now statically linked. Static vs dynamic linking has the usual differences: to update a statically linked library, you have to rebuild your binary. But it's more portable. Etc. etc.

What are the implications seecurity wise? Would we need to monitor security issues by ourselves?

Performance wise, we had users complaining about slowness in the past. This might be a problem for musl, according to your link. I wonder how big of a problem, it would be interesting to see benchmarks.

[angerman]

Performance of cabal-install being noticable due to musl would be a GHC bug, and should be fixed. If the libc choice has end user noticeable performance implications, something else is broken. fwiw we have cardano-node builds that are much more performance sensitive where people use musl builds with no performance issues at all.

Security wise, I think the whole dymaic vs static discussion is a red herring. Yes distros could take care of security implications and offer newer dependencies. It's still up to the user to actually install those dependencies. At the same time dynamic linking means someone can trivially inject a malicious dependency (LD_PRELOAD, ...).

Should we monitor dependencies for issues? Yes. But we should be/have been doing this to begin with if we are really concerned about this. hen again the next question is: how do you ensure your end user has an appropriately new version?

Size wise statically linking offers significant benefits as the linker can perform dead code elimination at link time. Runtime is slightly different as applications might not be sharing the same shared library in memory. Whether or not that is a real world issue is again very hard to properly analyze as it depends on the applications the end user runs concurrently with cabal-install and if there is a useful subtree of dependencies that are shared.

From a simplicity perspective static linking wins hands down.

[ffaf1]

Thanks for you summary, Moritz.

I am glad to know that performance issues are not there.

Yes distros could take care of security implications and offer newer dependencies. It's still up to the user to actually install those dependencies.

Distros do it, and I suspect we would have to do that too, if we are to go down that road. Indeed “how do you ensure your end user has an appropriately new version?” is a good question.

[angerman]

We will have some most-recent version check in the stable-haskell/cabal releases whether or not that is something that's upstreamable only time will tell.

Ultimately I don't trust users to keep their distributions up to date either.

[hasufell]

What are the implications seecurity wise? Would we need to monitor security issues by ourselves?

In my personal opinion, statically linked binaries are always less secure. But this isn't due to some technical issue. It's more about processes and workflows.

Linux distributions have elaborate workflows about warning users about required security updates etc. Once you go the static route, you don't just have to track vulnerable libraries, but also all binaries that statically link it and have the means to recompile and ship and update quickly.

Practically for cabal, I don't think there's any real security issue. There's only 3 libraries involved:

• libc
• gmp
• zlib

Tracking these is not hard. We could notify the haskell security response team about static linking in the Haskell Core Toolchain.

Performance wise, we had users complaining about slowness in the past. This might be a problem for musl, according to your link. I wonder how big of a problem, it would be interesting to see benchmarks.

I'm afraid I will not invest time in benchmarking against different libcs.

[mpickering]

It seems to me there are two separate tasks here.

• Move the release CI to use github
• A discussion about how to build release binaries, with the possibility of a new distribution method.

I think these should be tackled individually, in two separate MRs and issues. The first one seems uncontroversial but the latter seems to be a much larger change in the packaging story. I would like to make progress on the first issue without getting bogged down on the latter issue.