Conda build with ROOT 6.22 and python 3.8 is broken

[maxgalli]

The CI workflow that builds combine with ROOT 6.22 and python 3.8 in conda is broken. Spotted for the first time in this action.

[clelange]

I'm wondering whether we should still test using conda given that we have tests with the officially recommended software library versions.

[anigamova]

I looked at this briefly yesterday, and it looks like several packages (expat, icu) changed versions due to the GitHub runner updates ('2.326.0' -> '2.327.1'), in addition several other libraries get installed with the new runners. We could try to understand which one is responsible for the crashes, but I agree it is annoying to have to figure it out with every new GitHub runner version, considering that we have the CMSSW based tests for all of the root versions that we are testing with conda based installation at the moment. So I would be in favor of removing these tests. Tagging @nsmith- , since he wrote these tests specifically for inference model I think, which we can also add to the CMSSW jobs.

[nsmith-]

I would prefer we keep it since it guards against releases being incompatible with the conda package https://github.com/conda-forge/cms-combine-feedstock which people do use.

That said, ROOT 6.22 is very old. Do we even test that version in the CMSSW CI? We can just drop that version.

As for trying to resolve the issue, runner updates shouldn't affect this since conda build is in an entirely separate software stack from the runner base image. Looks to me like ROOT's genreflex is having trouble with some change to the conda solve. Here's the diff in packages compared to a previous run that worked

41c41
< expat                                   2.7.1  hecca717_0            conda-forge      141kB
---
> expat                                   2.7.0  h5888daf_0            conda-forge      140kB
58,59c58,59
< gettext                                0.25.1  h3f43e3d_1            conda-forge      541kB
< gettext-tools                          0.25.1  h3f43e3d_1            conda-forge        4MB
---
> gettext                                0.25.1  h5888daf_0            conda-forge      538kB
> gettext-tools                          0.25.1  h5888daf_0            conda-forge        4MB
67c67
< graphite2                              1.3.14  hecca717_1            conda-forge       99kB
---
> graphite2                              1.3.14  h5888daf_0            conda-forge       98kB
82d81
< icu                                      67.1  he1b5a44_0            conda-forge       13MB
108c107
< kernel-headers_linux-64                5.14.0  he073ed8_2            conda-forge        1MB
---
> kernel-headers_linux-64                3.10.0  he073ed8_18           conda-forge      943kB
110,111c109,110
< libasprintf                            0.25.1  h3f43e3d_1            conda-forge       54kB
< libasprintf-devel                      0.25.1  h3f43e3d_1            conda-forge       35kB
---
> libasprintf                            0.25.1  h8e693c7_0            conda-forge       53kB
> libasprintf-devel                      0.25.1  h8e693c7_0            conda-forge       35kB
119c118
< libexpat                                2.7.1  hecca717_0            conda-forge       75kB
---
> libexpat                                2.7.0  h5888daf_0            conda-forge       74kB
121,122c120,121
< libgettextpo                           0.25.1  h3f43e3d_1            conda-forge      188kB
< libgettextpo-devel                     0.25.1  h3f43e3d_1            conda-forge       37kB
---
> libgettextpo                           0.25.1  h5888daf_0            conda-forge      178kB
> libgettextpo-devel                     0.25.1  h5888daf_0            conda-forge       37kB
125c124
< libgfortran5                           15.1.0  hcea5267_4            conda-forge        2MB
---
> libgfortran5                           15.1.0  hcea5267_3            conda-forge        2MB
140d138
< libstdcxx                              15.1.0  h8f9b012_4            conda-forge        4MB
142d139
< libstdcxx-ng                           15.1.0  h4852527_4            conda-forge       29kB
172c169
< pixman                                 0.46.4  h537e5f6_0            conda-forge      410kB
---
> pixman                                 0.46.2  h29eaf8c_0            conda-forge      402kB
210c207
< sysroot_linux-64                         2.34  h087de78_2            conda-forge       40MB
---
> sysroot_linux-64                         2.17  h0157908_18           conda-forge       15MB
245c242
< yaml                                    0.2.5  h280c20c_3            conda-forge       85kB
---
> yaml                                    0.2.5  h7f98852_2            conda-forge       89kB

First (of many) compile errors in genreflex step is:

2025-08-05T15:03:45.5820994Z genreflex src/classes.h -s src/classes_def.xml -o build/obj/a/HiggsAnalysisCombinedLimit_xr.cc --deep --fail_on_warnings --rootmap=build/obj/a/HiggsAnalysisCombinedLimit_xr.rootmap --rootmap-lib=libHiggsAnalysisCombinedLimit.so -Isrc -I/home/runner/work/HiggsAnalysis-CombinedLimit/HiggsAnalysis-CombinedLimit/HiggsAnalysis/CombinedLimit/../../
2025-08-05T15:03:45.6979128Z In file included from input_line_3:2:
2025-08-05T15:03:45.6984949Z In file included from /usr/share/miniconda3/envs/combine/bin/../lib/gcc/x86_64-conda-linux-gnu/7.5.0/../../../../x86_64-conda-linux-gnu/include/c++/7.5.0/string:40:
2025-08-05T15:03:45.6986695Z In file included from /usr/share/miniconda3/envs/combine/bin/../lib/gcc/x86_64-conda-linux-gnu/7.5.0/../../../../x86_64-conda-linux-gnu/include/c++/7.5.0/bits/char_traits.h:40:
2025-08-05T15:03:45.6988581Z In file included from /usr/share/miniconda3/envs/combine/bin/../lib/gcc/x86_64-conda-linux-gnu/7.5.0/../../../../x86_64-conda-linux-gnu/include/c++/7.5.0/bits/postypes.h:40:
2025-08-05T15:03:45.6990389Z In file included from /usr/share/miniconda3/envs/combine/bin/../lib/gcc/x86_64-conda-linux-gnu/7.5.0/../../../../x86_64-conda-linux-gnu/include/c++/7.5.0/cwchar:44:
2025-08-05T15:03:45.6992573Z /usr/share/miniconda3/envs/combine/etc//cling/lib/clang/5.0.0/include/wchar.h:95:3: error: typedef redefinition with different types ('struct __mbstate_t' vs 'struct __mbstate_t')
2025-08-05T15:03:45.6993824Z } __mbstate_t;
2025-08-05T15:03:45.6994118Z   ^
2025-08-05T15:03:45.6995047Z /usr/share/miniconda3/envs/combine/x86_64-conda-linux-gnu/sysroot/usr/include/bits/types/__mbstate_t.h:21:3: note: previous definition is here
2025-08-05T15:03:45.6996114Z } __mbstate_t;
2025-08-05T15:03:45.6996401Z   ^

@guitargeek any idea here?

[maxgalli]

I would prefer we keep it since it guards against releases being incompatible with the conda package https://github.com/conda-forge/cms-combine-feedstock which people do use.

That said, ROOT 6.22 is very old. Do we even test that version in the CMSSW CI? We can just drop that version.

As for trying to resolve the issue, runner updates shouldn't affect this since conda build is in an entirely separate software stack from the runner base image. Looks to me like ROOT's genreflex is having trouble with some change to the conda solve. Here's the diff in packages compared to a previous run that worked

41c41
< expat 2.7.1 hecca717_0 conda-forge 141kB

expat 2.7.0 h5888daf_0 conda-forge 140kB
58,59c58,59
< gettext 0.25.1 h3f43e3d_1 conda-forge 541kB
< gettext-tools 0.25.1 h3f43e3d_1 conda-forge 4MB

---

gettext 0.25.1 h5888daf_0 conda-forge 538kB
gettext-tools 0.25.1 h5888daf_0 conda-forge 4MB
67c67
< graphite2 1.3.14 hecca717_1 conda-forge 99kB

---

graphite2 1.3.14 h5888daf_0 conda-forge 98kB
82d81
< icu 67.1 he1b5a44_0 conda-forge 13MB
108c107
< kernel-headers_linux-64 5.14.0 he073ed8_2 conda-forge 1MB

---

kernel-headers_linux-64 3.10.0 he073ed8_18 conda-forge 943kB
110,111c109,110
< libasprintf 0.25.1 h3f43e3d_1 conda-forge 54kB
< libasprintf-devel 0.25.1 h3f43e3d_1 conda-forge 35kB

---

libasprintf 0.25.1 h8e693c7_0 conda-forge 53kB
libasprintf-devel 0.25.1 h8e693c7_0 conda-forge 35kB
119c118
< libexpat 2.7.1 hecca717_0 conda-forge 75kB

---

libexpat 2.7.0 h5888daf_0 conda-forge 74kB
121,122c120,121
< libgettextpo 0.25.1 h3f43e3d_1 conda-forge 188kB
< libgettextpo-devel 0.25.1 h3f43e3d_1 conda-forge 37kB

---

libgettextpo 0.25.1 h5888daf_0 conda-forge 178kB
libgettextpo-devel 0.25.1 h5888daf_0 conda-forge 37kB
125c124
< libgfortran5 15.1.0 hcea5267_4 conda-forge 2MB

---

libgfortran5 15.1.0 hcea5267_3 conda-forge 2MB
140d138
< libstdcxx 15.1.0 h8f9b012_4 conda-forge 4MB
142d139
< libstdcxx-ng 15.1.0 h4852527_4 conda-forge 29kB
172c169
< pixman 0.46.4 h537e5f6_0 conda-forge 410kB

---

pixman 0.46.2 h29eaf8c_0 conda-forge 402kB
210c207
< sysroot_linux-64 2.34 h087de78_2 conda-forge 40MB

---

sysroot_linux-64 2.17 h0157908_18 conda-forge 15MB
245c242
< yaml 0.2.5 h280c20c_3 conda-forge 85kB

---

yaml 0.2.5 h7f98852_2 conda-forge 89kB
First (of many) compile errors in genreflex step is:

2025-08-05T15:03:45.5820994Z genreflex src/classes.h -s src/classes_def.xml -o build/obj/a/HiggsAnalysisCombinedLimit_xr.cc --deep --fail_on_warnings --rootmap=build/obj/a/HiggsAnalysisCombinedLimit_xr.rootmap --rootmap-lib=libHiggsAnalysisCombinedLimit.so -Isrc -I/home/runner/work/HiggsAnalysis-CombinedLimit/HiggsAnalysis-CombinedLimit/HiggsAnalysis/CombinedLimit/../../
2025-08-05T15:03:45.6979128Z In file included from input_line_3:2:
2025-08-05T15:03:45.6984949Z In file included from /usr/share/miniconda3/envs/combine/bin/../lib/gcc/x86_64-conda-linux-gnu/7.5.0/../../../../x86_64-conda-linux-gnu/include/c++/7.5.0/string:40:
2025-08-05T15:03:45.6986695Z In file included from /usr/share/miniconda3/envs/combine/bin/../lib/gcc/x86_64-conda-linux-gnu/7.5.0/../../../../x86_64-conda-linux-gnu/include/c++/7.5.0/bits/char_traits.h:40:
2025-08-05T15:03:45.6988581Z In file included from /usr/share/miniconda3/envs/combine/bin/../lib/gcc/x86_64-conda-linux-gnu/7.5.0/../../../../x86_64-conda-linux-gnu/include/c++/7.5.0/bits/postypes.h:40:
2025-08-05T15:03:45.6990389Z In file included from /usr/share/miniconda3/envs/combine/bin/../lib/gcc/x86_64-conda-linux-gnu/7.5.0/../../../../x86_64-conda-linux-gnu/include/c++/7.5.0/cwchar:44:
2025-08-05T15:03:45.6992573Z /usr/share/miniconda3/envs/combine/etc//cling/lib/clang/5.0.0/include/wchar.h:95:3: error: typedef redefinition with different types ('struct __mbstate_t' vs 'struct __mbstate_t')
2025-08-05T15:03:45.6993824Z } __mbstate_t;
2025-08-05T15:03:45.6994118Z   ^
2025-08-05T15:03:45.6995047Z /usr/share/miniconda3/envs/combine/x86_64-conda-linux-gnu/sysroot/usr/include/bits/types/__mbstate_t.h:21:3: note: previous definition is here
2025-08-05T15:03:45.6996114Z } __mbstate_t;
2025-08-05T15:03:45.6996401Z   ^

@guitargeek any idea here?

I'm afraid we still test it in CMSSW, here