Add midstream bindist proposal

Author: hasufell

proposals/0000-midstream-bindists.md

@@ -0,0 +1,292 @@
+# Midstream bindists
+
+## Abstract
+
+Bindists for the Haskell toolchain have been produced by upstream (the developers of each respective tool) for
+a long time and many tools rely on these "official" bindists (e.g. GHCup and stack).
+
+We propose here that bindists are additionally built and maintained in joint effort by the projects that
+provide installation experiences, removing the hard dependency on upstream bindists entirely.
+
+## Background
+
+Historically, installers like GHCup and stack have used upstream bindists for mainly one reason: it's easy to do
+so and doesn't require further efforts.
+
+However, using upstream bindists directly is extremely rare in the Linux world of distribution. Most distributions
+build, package, test and curate binary packages themselves, not only because they have custom formats, but for
+reasons of control, trust and quality.
+
+The relationship of GHCup and bindists has also been described in the blog post
+[GHCup is not an installer](https://hasufell.github.io/posts/2023-11-14-ghcup-is-not-an-installer.html) recently.
+
+## Problem Statement
+
+From the perspective of a GHCup developer, there are several issues with relying on upstream bindists.
+
+### Platform support
+
+GHC and other tools have in the past dropped support for certain platforms either entirely or requested
+the community to step up and do the work (e.g. on GHC CI).
+
+E.g. [GHC ARMv7 support was dropped silently without any call for help](https://gitlab.haskell.org/ghc/ghc/-/issues/21177#note_470440). Similarly, FreeBSD support just ceased to exist when the GHC FreeBSD CI stopped working. Later the community asked for a [revival](https://gitlab.haskell.org/groups/ghc/-/epics/5), but nothing signifcant has happened so far.
+GHCup still produces bindists from time to time for FreeBSD, but e.g. the [HLS release manager for 2.5.0.0 recently refused to add FreeBSD bindists](https://github.com/haskell/ghcup-metadata/pull/159).
+
+Similarly, stack used to have issues with Aarch64 bindists:
+
+* https://github.com/commercialhaskell/stack/issues/5709
+* https://github.com/commercialhaskell/stack/issues/5854
+* https://github.com/commercialhaskell/stack/issues/5540
+* https://github.com/commercialhaskell/stack/issues/5610
+* https://github.com/commercialhaskell/stack/issues/5619
+* https://github.com/commercialhaskell/stack/issues/6141
+* https://github.com/commercialhaskell/stack/issues/6142
+
+Recently, cabal-install had issues with i386 binaries and alpine, delaying a GHCup metadata PR:
+
+* https://github.com/haskell/ghcup-metadata/pull/127#issuecomment-1766020410
+
+These issues are frequent and so far the GHCup developers used to single handedly fix all those missing bindists manually
+and provide them here: https://downloads.haskell.org/~ghcup/unofficial-bindists/
+
+This is unfunded and significant work.
+
+### Bindist maintenance
+
+Sometimes, bindists are broken, e.g. for GHC there are a couple of instances:
+
+* 9.0.2 shipping without profiling info: https://gitlab.haskell.org/ghc/ghc/-/issues/21841
+* DESTDIR variable ignored by `make install`: https://gitlab.haskell.org/ghc/ghc/-/issues/19646
+
+Sometimes, bindists have been built for very old version of linux distros and won't run well on newer linux versions.
+This is currently a problem since Debian has removed ncurses5:
+
+* https://github.com/haskell/ghcup-hs/issues/902
+* https://answers.launchpad.net/ubuntu/+source/ncurses/+question/707838
+* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1025964
+
+For cabal, there has been the infamous hLock issue:
+
+* https://github.com/haskell/cabal/issues/7313
+* https://github.com/haskell/cabal/issues/7950
+
+This shows that bindists, for current and historical versions, need continuous maintenance. However, upstream developers
+so far have very rarely engaged in this type of maintenance work, pushing it down to GHCup. As an example, here are all the
+manually patched and re-packaged bindists that fix the DESTDIR bug outlined above: https://downloads.haskell.org/ghcup/unofficial-bindists/ghc/curated/
+
+This type of work also requires significant time.
+
+### Quality gateway
+
+There's something pleasing about upstream providing bindists: there's a perceived trust about it in the community,
+e.g. the [haskell.org committee expressed concerns about GHCup changing bindists in the past](https://github.com/haskell-infra/www.haskell.org/issues/212#issuecomment-1272312911).
+
+However, rarely do upstream developers have signifcant experience in redistribution, nor do they have the time to focus
+on all the issues that come with it. Bindists are mostly provided "as-is" and support beyond what the release CI outputs is
+left to midstream (GHCup, stack, ...).
+
+Conceptually, it is a good idea to separate concerns: upstream provides the sources and has tested it. Distributions build the binaries,
+validate that the program can be successfully built from source and make sure that the final artifacts pass the test suite.
+This is good, because we want to know whether end users can build e.g. a functioning GHC from source. If only the release CI outputs
+a GHC that passes the test suite, then something is fundamentally broken.
+
+Distributors are often closer to the end-users and can provide additional support and efforts for the installation experience.
+
+### Security backports
+
+Cabal has recently found to be vulnerable to [HSEC-2023-0015](https://github.com/haskell/security-advisories/blob/main/advisories/hackage/cabal-install/HSEC-2023-0015.md).
+
+This vulnerability [had not been communicated to the GHCup team](https://github.com/haskell/security-advisories/issues/129) prior to disclosure,
+causing high distress for a backport, since at the time of disclosure, cabal-3.6.2.0
+was still 'recommended' by GHCup, since [cabal-3.10.2.0 is still broken on windows](https://github.com/haskell/cabal/issues/9334).
+
+As such, GHCup developers needed a quick and efficient way to:
+
+* patch cabal
+* build release binaries for cabal
+* ship the binaries
+
+This had been done as a downstream release `3.6.2.0-p1` roughly a week after the disclosure.
+
+Meanwhile, cabal upstream still has not finished their backport due to issues
+with hackage dependencies: https://github.com/haskell/cabal/pull/9457
+
+This shows that a certain amount of independences from upstream CI and upstream workflow
+is essential to fulfill swift security backports to potentially under-maintained
+branches/versions.
+
+### GHC nightlies
+
+As a special case, I want to point out that GHC nightlies have been frequently broken beyond repair:
+
+* https://gitlab.haskell.org/ghc/ghcup-metadata/-/issues/2
+* https://gitlab.haskell.org/ghc/ghc/-/issues/24000
+
+The breakage was left unattended and some bindists completely vanished from gitlab CI artifacts, because of misconfiguration.
+Here's a graph of nightlies availability: https://grafana.gitlab.haskell.org/d/ab109e66-a8a1-4ae9-b976-40e2dfe281ab/availabilitie-of-ghc-nightlies-via-ghcup?orgId=2
+
+## Prior Art, Related Efforts and alternative solutions
+
+So far, GHCup developers have tried to close the gap, doing signifcant work on upstream CIs and building bindists manually where
+necessary.
+
+Bindists for cabal-install are now produced by GHCup's own CI: https://github.com/haskell/ghcup-metadata/blob/develop/.github/workflows/cabal-release.yaml
+
+HLS will likely follow shortly.
+
+As mentioned before, there have been attempts to improve the coordination and collaboration across the entire Haskell
+toolchain, view the tooling end-user experience in a holistic way and make decisions based on that end-user experience: https://github.com/haskellfoundation/tech-proposals/issues/48
+
+The proposers believe that the community structure at the moment does not allow such an approach and there needs to be
+significant work to align goals, perception and priorities. Otherwise there will be too much friction.
+
+The most important currency in the open source volunteer world is **energy**. It is not code or technical effort. As such we
+believe that the amount of saved energy by being more independent of upstream release processes and decisions far outweighs
+potential costs of technical redundancy/duplication.
+
+A future proposal may very well attempt to create a unified user experience across the entire Haskell toolchain
+through joint management and collaboration. But that is not in the scope of this proposal and we have no concrete
+idea how to achieve that.
+
+## Technical Content
+
+We propose here to create a joint project of "installation experience" developers to get funding and maintain
+all tooling bindists centrally and autonomically.
+
+This will allow:
+
+* greater quality assurance
+* good platform support, we want to support at least these consistently:
+* more bindists for uncommon linux distros
+* continuously updated and maintained bindists (including historical GHC versions)
+* actually working nightlies
+
+### Phase 1
+
+In Phase 1 we will build the entire Haskell toolchain autonomically.
+The way this will be implemented is to start a central GitHub repository that builds bindists for releases of:
+
+- GHC
+- HLS
+- cabal
+- stack
+
+For the following platforms:
+
+- FreeBSD x86_64
+- Linux i386
+- Linux x86_64
+- Linux armv7
+- Linux aarch64
+- Darwin x86_64
+- Darwin aarch64
+- Windows x86_64
+
+For the following Linux x86_64 distros:
+
+- Debian
+- Ubuntu
+- Mint
+- Fedora
+- CentOS
+- RedHat
+- Rocky Linux
+- Void Linux
+- Amazon Linux
+- Alpine
+
+Linux i386, armv7 and aarch64 will be confined to Debian or Ubuntu.
+
+### Phase 2
+
+In Phase 2 we want to:
+
+* enhance the quality of the bindists by
+  - running the entire test suite for all of the tools
+  - having a mechanism to report test failures back upstream
+  - publishing test failures for end users to see
+  - communicte test status of bindists clearly through e.g. GHCup
+  - resolve GHC issues related to test bindists:
+    * https://gitlab.haskell.org/ghc/ghc/-/issues/22726
+    * https://gitlab.haskell.org/ghc/ghc/-/issues/22723
+    * https://gitlab.haskell.org/ghc/ghc/-/issues/22727
+* make fixing bindists easier
+  - implement revisions in GHCup: https://github.com/haskell/ghcup-hs/issues/361
+  - make it easy to update an older GHC branch and re-run the release pipeline
+- Make building upstream release binaries easier
+  * https://github.com/haskell/cabal/issues/9461
+  * https://github.com/haskell/haskell-language-server/issues/3878
+
+One main idea is that bindists should be primarily tested **on the users system**, because that is where they're going to run.
+It is great to know that e.g. the test suite passes on GHC CI, but that may have little value in different environments.
+Additionally, issues with tests can flow back to upstream developers and we may develop workflows and processes to streamline
+this type of feedback. Early release candidates can assist with this workflow.
+
+Another perception shift necessary is that upstream projects should consider that their build system
+are end-user interfaces, making it easier for both distributors and end-user to build release binaries correctly themselves.
+
+### Phase 3
+
+In phase 3, we will mainly focus on making nightlies available for GHC and cabal. This will require
+coming up with a permanent storage solution and very robust nightly pipelines.
+
+HLS nightlies may be added as well.
+
+## Timeline
+
+* phase 1 (6 months): proof of concept of a central GitHub CI building the entire toolchain
+* phase 2 (9 months): bindist testing workflow, automated integration with GHCup, etc.
+* phase 3 (12+ months): nightlies working for GHC and cabal and maybe HLS
+
+## Budget
+
+We would like to have a full-time developer work on this, at least partly. But at the moment,
+we don't have anyone to propose and hiring an external contractor for this role may or may not
+be helpful, since this requires significant DevX experience and commitment to the specified goals.
+
+As such, we propose the "phase 1" of this proposal that focuses on advancing the existing GitHub CI
+support for the GitHub Haskell organization, spearheaded by @angerman (providing aarch64 linux and
+darwin M1 self-hosted machines). This infrastructure is already used by GHCup, HLS and other projects
+and needs more runners.
+
+Github actions at the moment does not provide all platforms. So the budget proposal to close the platform gap on GitHub actions is as follows (all prices in USD):
+
+- FreeBSD x86_64 private runner on [vultr](https://www.vultr.com/pricing/), utilizing [github-act-runner](https://github.com/ChristopherHX/github-act-runner), instance with 8vCPU, 16 GB Memory, 150GB storage
+  * monthly: $160
+  * yearly: $1,920
+- Linux aarch64 private runner on [AWS](https://aws.amazon.com/marketplace/pp/prodview-uzg6o44ep3ugw) (m6g.xlarge instance)
+  * hourly: $0.154
+  * monthly: $112.5
+  * yearly: $1,350
+- Darwin M1 private runner on [macstadium](https://www.macstadium.com/pricing), instance with M1 8 Core, 16 GB Memory, 1 TB SSD
+  * monthly: $179
+  * yearly: $2,148
+
+To host one additional private runner per all these platform, the yearly cost would be: **$5,418**
+
+We may request more runners depending on the demand, so it may very well be 3 runners per platform, resulting in yearly cost of: $16,254
+
+For FreeBSD it is also possible to buy [Cirrus CI compute credits](https://cirrus-ci.org/pricing/#compute-credits),
+which amount to a similar value of: $1,576.8
+
+As such, we estimate the cost for phase 1 to be between 5000 to 16000 USD. The benefit of this investment is not confined to
+the midstream bindist proposal, but to the entirety of the GitHub Haskell organization CI infrastructure
+(including GHCup, HLS, Cabal, bytestring, text, unix, ...).
+
+Phase 2 and 3 may need follow-up proposals with more concrete implementation strategy, cost analysis
+and potentially hiring a full-time employee.
+
+## Stakeholders
+
+* GHC developers
+* cabal developers
+* stack developers
+* HLS developers
+* VSCode Haskell developers
+* GHCup developers
+* Haskell toolchain end users
+
+## Success
+
+* reliable, continuously maintained bindists, readily available
+