Use GHA build matrix to parallelize over Python versions?

[wjakob]

I had the following question regarding cibuildwheel: is there a builtin feature to use the GHA build matrix to compile different wheels for a specific platform in parallel? For example for mitsuba3, some of the CI launches take over five hours, which is hitting the limit of practicality.

Many thanks!
Wenzel

[burgholzer]

I am not @henryiii, but you might be interested in the following article: https://iscinumpy.dev/post/cibuildwheel-2-10-0/#only by him.

[Czaki]

You could limit set of wheels to be built on given action using CIBW_BUILD environment variable. And you could use it for control what will be build in a given action in matrix.

[wjakob]

That sounds like a viable option. The thing I was shying away from was that it would be necessary to enumerate quite a lot of configurations (AFAIK there isn't a way to say: build half of the versions in this runner, and half in the other. I will have to essentially hard-code all the configuration strings in CIBW_BUILD and update them every time a new Python version is released.)

[henryiii]

That's one of the reasons for the only method. :) But you can do cp3?-* on one runner and cp3??-* on anther, etc. There's a lot you can do with that string.

[henryiii]

It supports expressions like cp3{6,7,8}-* and cp3{9,1?}- too.

[henryiii]

You can either use CIBW_BUILD and manually chose some way to split it up (@Czaki's answer, see https://github.com/scikit-hep/boost-histogram/blob/develop/.github/workflows/wheels.yml, for some examples) or the newer only trick to dynamically generate them (@burgholzer's answer, see mypyc/mypy_mypyc-wheels#58, though I like only using jq like the linked article).