Let pacman-helper deploy also to the GitHub repo git-for-windows/pacman-repo

[dscho]

In my endeavor to disentangle Git for Windows' Pacman repository from my personal Azure account, I have started mirroring said Pacman repositories' contents to https://github.com/git-for-windows/pacman-repo/.

Most of the contents, namely the archives reflecting outdated package versions, were uploaded to https://github.com/git-for-windows/pacman-repo/releases.

The archives reflecting the current package versions, as well as the package databases, were uploaded to the x86_64, i686, and aarch64 branches of the git-for-windows/pacman-repo repository. The idea is to eventually switch Git for Windows to that Pacman repository by modifying /etc/pacman.conf as follows:

diff --git a/etc/pacman.conf b/etc/pacman.conf
index 99179ac6024..9555ae89af4 100644
--- a/etc/pacman.conf
+++ b/etc/pacman.conf
@@ -70,17 +70,17 @@ LocalFileSigLevel = Optional
 # Server = https://repo.msys2.org/staging/
 # SigLevel = Never
-[git-for-windows-aarch64]
-Server = https://wingit.blob.core.windows.net/aarch64
+[git-for-windows-clangarm64]
+Server = https://raw.githubusercontent.com/git-for-windows/pacman-repo/refs/heads/aarch64
 [clangarm64]
 Include = /etc/pacman.d/mirrorlist.mingw
-[git-for-windows]
-Server = https://wingit.blob.core.windows.net/x86-64
+[git-for-windows-x86_64]
+Server = https://raw.githubusercontent.com/git-for-windows/pacman-repo/refs/heads/x86_64
 [git-for-windows-mingw32]
-Server = https://wingit.blob.core.windows.net/i686
+Server = https://raw.githubusercontent.com/git-for-windows/pacman-repo/refs/heads/i686
 [mingw32]
 Include = /etc/pacman.d/mirrorlist.mingw

There is a slight downside to using this approach: GitHub does not report a Last-Modified: for raw blob URLs, and hence pacman -Sy will always download Git for Windows' package databases. Since those weigh only a couple dozen kilobytes, that should be okay, especially given the fact that most users (apart from those who are on their own) will clone git-sdk-64 and not update via Pacman at all.

An alternative I had considered was to use GitHub Pages, but I am loathe to deploy ~130MB for x86_64, ~174MB for aarch64 and ~161MB for i686 every single time any small package wants to be updated.

Another alternative I considered was using the GitHub Releases, but since Pacman repositories have to have a flat directory layout, meaning that the package archives' URLs are expected to be siblings of the package databases' URLs, that would have needed ugly work-arounds:

• Either stay with a new GitHub Release for every deployment (an idea I like very much) but then needing a real hack such as (ab-)using the XferCommand setting to interpret a fake URL together with a (to be generated) mapping file that lives in every new GitHub Release so as to route the request to the correct GitHub Release for each package archive to download, or
• Move to a single, honking big GitHub Release that contains all of the archive files (even for superseded package versions) and whose package databases are replaced all the time (which cannot be done atomically on GitHub, it requires a DELETE followed by a new upload, and during that time window the Pacman repository is broken, and also it would be tricky to deal with concurrent deployments).

One alternative I dismissed quite early on was to follow Homebrew's footsteps who figured that GitHub Packages should really be used to host, you know, packages, but there seems not to have been any investment in a flexible-enough architecture to accommodate but the most common package management systems, Pacman not being one of them (or for that matter, Homebrew, who have to pretend that their packages are attachments to Docker containers even though they do not have any Docker containers, and then they still need to jump through major hoops just to get their hosted packages to download).

In the end, I think I found an acceptable solution that will work for us and won't need much in the way of maintenance, nor do I expect it to cause any burden on GitHub.

With all of that background, here is the explanation what this PR tries to accomplish: To start the transition away from Azure Blobs toward git-for-windows/pacman-repo, this PR adjusts Git for Windows' deployment work horse, pacman-helper.sh quick_add, to not only deploy to Azure Blobs, but also to git-for-windows/pacman-repo at the same time.

This will allow us to verify that this thing is working without changing /etc/pacman.conf for the time being. Once it proves to be working all right, we can then change /etc/pacman.conf and eventually drop the parts of pacman-helper.sh that still want to upload to Azure Blobs (and then we can even retire wingit-snapshot-helper.sh!).

Note that this PR will still need a git-for-windows-automation companion to generate and use an installation access token in the build-and-deploy workflow so that it can push the updates to git-for-windows/pacman-repo. I spent the best part of today getting to this point, therefore that PR will need to be patient and await tomorrow.

[jeremyd2019]

I don't know the syntax here, but it seems odd there's a leading slash on just this one item.

[mjcheetham]

The resulting exclude file would be:

/git-for-windows.db*
git-for-windows.files*

..which means, exclude only files matching git-for-windows.db* from the root of the repository /, and exclude files matching git-for-windows.files* at any depth path.

[dscho]

Right, that was an oversight. I should probably also add the leading slashes to the sparse-checkout lines. Not that it matters a lot, as I've been complaining about, Pacman repositories must have a flat directory structure (i.e. no subdirectories, let alone sibling directories allowed). Therefore those patterns can only match files in the root tree because there is only a root tree.

[jeremyd2019]

Ouch. This sed pattern appears several times, so this comment applies to each of them: it is not necessarily safe: a version can start with a letter (such a version would compare less than a version that starts with a number). What msys2 does to get the package name from a filename is

pkgname="$(echo "$pkg" | rev | cut -d- -f4- | rev)"

You could probably just drop the [0-9]s if you wanted to use sed.

[dscho]

Huh. So what you're saying is that sed 's/\(-[^-]*\)\{3\}$//' would do the job, too? I guess that would work, but it would remove some of the sanity checking... I think I want to go with sed 's/\(-[^-]*\)\{3\}\.pkg\.tar\.\(xz\|zst\)$/-[0-9]*/' instead.

Note that the replacement -[0-9]* is a shell wildcard pattern designed to catch earlier package versions, and this still relies on version numbers starting with, ahem, numbers.

That's fine, though, as Git for Windows' Pacman repository does not contain any of those packages with funny "version numbers". In MSYS2, I only see colormake-git, cygnative, gyp, pactoys, mingw-w64-breakpad-git, mingw-w64-cppcoro, mingw-w64-daala-git, mingw-w64-devcon-git, mingw-w64-fxc2, mingw-w64-glsl-optimizer-git, mingw-w64-heimdall, mingw-w64-hlsl2glsl-git, mingw-w64-libbacktrace, mingw-w64-libfixmath, mingw-w64-libmpeg2-git, mingw-w64-libnatpmp, mingw-w64-libunifex, mingw-w64-libusb-compat-git, mingw-w64-libutp, mingw-w64-libvmime-git, mingw-w64-llama.cpp, mingw-w64-lldb-mi, mingw-w64-m68k-apple-macos-binutils-git, mingw-w64-mhook, mingw-w64-nanovg, mingw-w64-ntldd, mingw-w64-ogitor-git, mingw-w64-powerpc-apple-macos-binutils-git, mingw-w64-qoi, mingw-w64-rkdeveloptool, mingw-w64-stb, and mingw-w64-thor. None of which Git for Windows is threatening to shadow in its Pacman repository.

BTW mingw-w64-llama.cpp sticks out because it has the only "version number" that starts with a b, all the other non-numbers start with r.

For the record, I populated the list above using git grep 'pkgver="\?[^"0-9]' \*/PKGBUILD in MINGW-packages and MSYS2-packages, but in hindsight a slightly involved tar Oxvf /var/lib/pacman/sync/<db> | sed -e '<put-package-name-in-hold-then-print-it-for-every-package-version-that-does-not-start-with-a-digit>' invocation would have been a lot faster and a lot less manual.

[dscho]

in hindsight a slightly involved tar Oxvf /var/lib/pacman/sync/<db> | sed -e '<put-package-name-in-hold-then-print-it-for-every-package-version-that-does-not-start-with-a-digit>' invocation would have been a lot faster and a lot less manual.

Indeed, it only took me 5 minutes (as opposed to the 30 minutes before, but analysing it took a bit longer again because I was puzzled about llama.cpp, see below):

(
  printf '| Version | Package |\n| - | - |\n'
  for dbname in msys mingw64
  do
    tar Oxvf /var/lib/pacman/sync/$dbname.db 2>&1 |
    sed -n '/%NAME%/{n;x};/%VERSION%/{n;G;s/^/| /;s/\n/ | /g;s/$/ |/;/^| [^0-9]/p}'
  done
)

yields:

Version	Package
r8.9c1d2e6-2	colormake-git
r9.2cbf760-2	cygnative
r2173.a03d7413-3	gyp
r65.b5fcc0f-1	pactoys
r1680.70914b2d-2	mingw-w64-x86_64-breakpad-git
r390.a87e97f-2	mingw-w64-x86_64-cppcoro
r1505.52bbd43-2	mingw-w64-x86_64-daala-git
r540.f0adcda0-3	mingw-w64-x86_64-devcon-git
r19.807d26f-1	mingw-w64-x86_64-fxc2
r180.3997d5c-2	mingw-w64-x86_64-heimdall
r115.1db8564-1	mingw-w64-x86_64-libbacktrace
r1108.946bf4b-2	mingw-w64-x86_64-libmpeg2-git
r84.07004b9-2	mingw-w64-x86_64-libnatpmp
r564.d7d191e-2	mingw-w64-x86_64-libunifex
r76.b5db9d0-2	mingw-w64-x86_64-libusb-compat-git
r175.9cb9f9c-2	mingw-w64-x86_64-libutp
r1270.d03ad5f0-1	mingw-w64-x86_64-libvmime-git
r93.a6c8c66-2	mingw-w64-x86_64-lldb-mi
r808.8d32c583ae-2	mingw-w64-x86_64-m68k-apple-macos-binutils
r7.a159eed-2	mingw-w64-x86_64-mhook
r398.c35e80c-2	mingw-w64-x86_64-nanovg
r19.7fb9365-4	mingw-w64-x86_64-ntldd
r893.58e2cca-1	mingw-w64-x86_64-ogitor-git
r808.8d32c583ae-1	mingw-w64-x86_64-powerpc-apple-macos-binutils
r346.b0b926e-1	mingw-w64-x86_64-qoi
r70.21b25fd-1	mingw-w64-x86_64-rkdeveloptool
r2193.31707d1-1	mingw-w64-x86_64-stb
r271.3e320cb-2	mingw-w64-x86_64-thor

On the other hand, it missed mingw-w64-llama.cpp. That's because the package's filename contains the epoch 1 and therefore the version part does start with a digit (it's 1~b4458-1).

[jeremyd2019]

Huh. So what you're saying is that sed 's/\(-[^-]*\)\{3\}$//' would do the job, too? I guess that would work, but it would remove some of the sanity checking... I think I want to go with sed 's/\(-[^-]*\)\{3\}\.pkg\.tar\.\(xz\|zst\)$/-[0-9]*/' instead.

Note that the replacement -[0-9]* is a shell wildcard pattern designed to catch earlier package versions, and this still relies on version numbers starting with, ahem, numbers.

Also, this blob from makepkg

                        # clean up dangling symlinks to packages
                        for pkg in ${pkgname[@]}; do
                                for file in ${pkg}-*-*-*{${PKGEXT},${SRCEXT}}; do
                                        if [[ -h $file && ! -e $file ]]; then
                                                rm -f "$file"
                                        fi
                                done
                        done

[dscho]

@jeremyd2019 I'm just worried about things like mingw-w64-x86_64-git-*-*-*${PKGEXT} totally matching mingw-w64-x86_64-git-extra-1.1.646.cafecafe-1-any.pkg.tar.xz...

[jeremyd2019]

Another alternative I considered was using the GitHub Releases, but since Pacman repositories have to have a flat directory layout, meaning that the package archives' URLs are expected to be siblings of the package databases' URLs, that would have needed ugly work-arounds

For the record, I use the "single, honking big GitHub Release" method, and there is another issue with using a GitHub release that has to be worked around: certain characters are not allowed in the filenames, and get silently replaced with periods.
The github docs just have this to say about it:

GitHub renames asset filenames that have special characters, non-alphanumeric characters, and leading or trailing periods. The "List release assets" endpoint lists the renamed filenames. For more information and help, contact GitHub Support.

The only such character I have run into in practice is ~, for MSYS2 pacman repositories this is used for the version epoch separator. (I saw a comment from somebody trying to host an Arch pacman repository in a similar manner that reports : (their epoch separator) is also subject to this).

Note that this is silent, the upload won't fail, but attempts to fetch using the original name will fail. As a workaround, the files can be renamed before repo-add, the version recorded in the sync database comes from the .PKGINFO in the package, not from the filename, while the filename recorded in the database is used to fetch it.

[mjcheetham]

The resulting exclude file would be:

/git-for-windows.db*
git-for-windows.files*

..which means, exclude only files matching git-for-windows.db* from the root of the repository /, and exclude files matching git-for-windows.files* at any depth path.

[dscho]

@jeremyd2019 @mjcheetham I have addressed these concerns in https://github.com/git-for-windows/build-extra/compare/c527edf5be3204469a7b72caa8c101a261c4bba1..80a77f4e211ead3a0cfb245e1b5ebf79e9699892, I believe. Could you have a look?

[mjcheetham]

Could you have a look?

LGTM

[dscho]

Note that this PR will still need a git-for-windows-automation companion to generate and use an installation access token in the build-and-deploy workflow so that it can push the updates to git-for-windows/pacman-repo.

Well, here it is, that PR, at last: git-for-windows/git-for-windows-automation#113

[jeremyd2019]

Could you have a look?

LGTM

me too, fwiw

[dscho]

For the record, I needed a follow-up patch to make it actually work. The reason I did not spot that problem is that Git for Windows' automation has to play some games with gpg to make it work non-interactively, and hence gpg.program is set to gnupg-with-gpgkey.sh (which is designed to sign, but not to verify), and that in turn is called by the call_gpg function in pacman-helper.sh. To make it work in git-for-windows-automation, I therefore had to add e3a5828. But it looks as if that was the only follow-up patch that was required, which seems surprising to me given the complexity of what I added ;-)

[dscho]

Ah, I needed another follow-up patch, to accommodate for concurrent deployments: 1acb634.

Note: This work-around is only necessary because we still deploy to Azure Blobs. Without that, there would not have been any waiting for the lease, and no downloading the package database from Azure Blobs, and no up to date check that could go wrong 😉