Implement --matrix-extra for more matrix dimensions

[frasertweedale]

Sometimes you need to test your project along more dimensions than just GHC
version. This is particularly important for programs/libraries that use
FFI to bind to libraries - they might need to be tested against a range of
library versions.

In general, you want to test all the combinations of GHC versions and other
properties, i.e. the cartesian product. It is burdensome for maintainers
that need such a strategy to manually adjust the matrix after every
(re)generation of the CI script/spec. Better support for this scenario in
haskell-ci is warranted.

This commit implements a new --matrix-extra option, which adds additional
matrix dimensions. The option value format is:

--matrix-extra libfoo:2.6,3.0;libbar:0.1,0.2

haskell-ci adds all the combinations of GHC version and the
--matrix-extra fields to the matrix. Corresponding build/test steps can be
introduced via --github-patches (or --travis-patches).

This commit implements this feature for GitHub actions only. It can be
implemented for Travis in a subsequent commit, if desired.

[phadej]

Please don't add new packages to test-suite, if not strictly necessary. I wonder what notmuch is (it's far from obvious it is a library name).

[frasertweedale]

Please don't add new packages to test-suite, if not strictly necessary. I wonder what notmuch is (it's far from obvious it is a library name).

notmuch is a mail indexer. I maintain a Haskell binding. It's one of the projects this PR will help with - for testing with different versions of libnotmuch.

I wanted a .cabal with shorter Tested-With list, so the cartesian product is not huge. All existing .cabal files in the tests have long lists. Perhaps I can make a small bogus package .cabal to test with? (FWIW GitHub actions enforces a limit of 256 matrix legs.)

Are you comfortable with the general approach of this PR? Are there any other changes you would like to make? (naming, etc?) I'm about to crash for the night but I'll address all your feedback tomorrow.

[phadej]

I'm not sure. I feel that then someone would ask to have away to filter the cartesian product etc. I have to think about this properly.

[frasertweedale]

@phadej some final arguments in support of this feature:

• Right now, if you need to expand the matrix you are doomed to manually do it yourself every time you want to update the GHC versions. This PR solves the simple case of this problem (i.e. when you want the full cartesian product).
• If you want to filter, or add extra jobs to the matrix, it can be done with exclude and include specifications. These fields could be added via a --github-patches patch, so that even without explicit support in haskell-ci, the desired outcome can be achieved with minimal ongoing maintainer intervention.

I hope you will decide in favour of this feature. Let me know of any changes you would like.

[frasertweedale]

Rebased to resolve merge conflicts.

[frasertweedale]

-_- I'll sort out those failing tests on the weekend, hopefully!

[frasertweedale]

I am reviving this PR. The need still exists (and always will).

I changed the tests to avoid adding notmuch. The golden tests now use splitmix instead.

I also added a check (with corresponding test) that the matrix size does not exceed GitHub's limit of 256 jobs.
This test also uses splitmix. However, we may eventually need to use a different .cabal file with more constrained
Tested-With - there are currently 51 GHC versions. With matrix-extra of size 2 x 2 the size of the matrix is already
> 200 and approaching the limit.

[frasertweedale]

@phadej ping - could you please review and consider merging this PR?

[frasertweedale]

Annual ping... I have rebased and resolved conflicts (waiting to see if tests pass - will address them if not) and once again ask for a determination about this PR.

[frasertweedale]

Another rebase and test fixture update... @phadej would you consider merging this? There is a real need for it, especially for projects that use system binaries or libraries and support multiple versions thereof.

[phadej]

As you have to use patches anyway, you can do this with a small patch. See #785

     continue-on-error: ${{ matrix.allow-failure }}
     strategy:
       matrix:
+        compiler: [ghc-9.10.2,ghc-9.8.4,ghc-9.6.7,ghc-9.4.8,ghc-9.2.8,ghc-9.0.2,ghc-8.10.7,ghc-8.8.4]
+        animal: [cat, dog]
         include:
           - compiler: ghc-9.10.2
             compilerKind: ghc

It would make sense to make haskell-ci generate the compiler matrix-dimension automatically. So you would only need to add your dimensions with your patch.

https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/running-variations-of-jobs-in-a-workflow#expanding-or-adding-matrix-configurations

[frasertweedale]

@phadej I tried this approach before pivoting to the current proposal. It didn't work well because the patch will need updating every time the GHC list changes (otherwise conflicts), which defeats the purpose.

[phadej]

It would make sense to make haskell-ci generate the compiler matrix-dimension automatically. So you would only need to add your dimensions with your [script] patch.

The extra dimension could also be put somewhere where it doesn't conflict as easily (i.e. after include:, older GHCs don't change as much).

IMO, if you are using patches, conflicts are to be expected.

[frasertweedale]

The extra dimension could also be put somewhere where it doesn't conflict as easily (i.e. after include:, older GHCs don't change as much).

Until you drop support for old versions and stop testing them. You could reduce the number of context lines to slip it in ahead of fail-fast: false, but this seems brittle.

IMO, if you are using patches, conflicts are to be expected.

They do happen, but far less often than changes to the GHC versions I'm testing. In other words, haskell-ci regenerate Just Works most of the time (using a build containing my patch).

Genuinely, what is the hesitation to accepting this change? You previously mentioned that some people might want to filter the matrix further, but right now there's not even a robust way to produce the extra matrix dimension(s). It solves the problem well for me - much better than the alternative proposals. I've been carrying the patch and using it in anger for 4.5 years.

Let me know if there's anything I can do to make this patch acceptable :)

[phadej]

I don't like patch feature. It's currently a necessary evil. But it's bad. I don't like build-type: Custom in Cabal, I don't like patches in haskell-ci. And I'm not going to not put effort to make using them easier or/and "more applicable".

In both cases, there are missing features, i'd rather work on them.

[phadej]

I'll add compiler: list to haskell-ci generated GitHub workflow, but I don't think this is going in. I'm sorry.

[frasertweedale]

In both cases, there are missing features, i'd rather work on them.

Testing your project across multiple values related to non-Haskell dependencies is one such missing feature.

Here are some ideas for how --matrix-extra could work without patches:

1. Hook points for inject extra steps in the GitHub workflow, supplied via config. This seems unpleasant to implement, but easy to use.
2. Expose extra matrix dimensions to the test suite as environment variables, e.g. HASKELL_CI_MATRIX_key=value. Then the Haskell test program could read the env vars perform the additional steps. This seems easier to implement, but more awkward for users.

Do either of these approaches seem acceptable?