Notes on building stack for NVIDIA Grace & Grace/Hopper

[trz42]

Issue for keeping track of what to build, in which order (coordinate among builders) and document potential issues. In general, we will follow efforts done by @bedroge building a new stack for Sapphire Rapids.

Current approach

• use the most recent EasyBuild version before v5.0.0 (to avoid being affected by breaking changes)
• collapse easystack files into one using v4.9.4
• remove from-pr for PRs being included with v4.9.4 or earlier
• replace from-pr with from-commit

Outdated (by 2025-03-26)

NOTE, we need to be careful with from-pr. It might be that the ec being used is not what we expect. It seems it is much better to use the latest commit of the PR.

Idea to approach the from-pr issue:

• for software that was originally built with EB < 4.9.2 we will nevertheless use EB 4.9.2
• we will always add from-commit and include-easyblocks-from-commit
• if neither from-pr or include-easyblocks-from-pr was used we use the commit hashes corresponding to the release of the EB version originally being used
• if *from-pr was used we will replace it with *from-commit and the most recent commit in the PR

---

Attempt 1 (removed 2025-03-25)

Kickstarting the stack

• add EasyBuild (grace PR: {2023.06}[system,grace] EasyBuild 4.8.2, 4.9.0, 4.9.1, 4.9.2, 4.9.3, 4.9.4 #968; sapphirerapids PR: {2023.06}[system,sapphirerapids] EasyBuild 4.8.2, 4.9.0, 4.9.1, 4.9.2, 4.9.3, 4.9.4 #921)
  • we did not use --from-pr with {2023.06}[system,grace] EasyBuild 4.8.2, 4.9.0, 4.9.1, 4.9.2, 4.9.3, 4.9.4 #968

Below we list further installations with the toolchain system:

• add ReFrame 4.6.2, Java 11 & NextFlow (grace PR {2023.06}[system,grace] NextFlow 23.10.0, ReFrame 4.6.2 #975; sapphirerapids PR: {2023.06}[system,sapphirerapids] Nextflow 23.10.0, ReFrame 4.3.3 + 4.6.2 #928)
  • we skipped ReFrame/4.3.3 for now, because it failed due to relying on pip as install method but we are not going to use it (we will use ReFrame/4.6.2 or newer); also see {2023.06}[system,grace] NextFlow 23.10.0, ReFrame 4.6.2 #975 (comment)
  • we used from-pr: 19172 for Nextflow, however the easyconfig available from /cvmfs is identical to the one on GitHub in PR 19172 -> no need to rebuild Nextflow

Attempt 2

• add EasyBuild 4.8.2, 4.9.{0,1,2,3,4}, 5.0.0; ReFrame 4.3.3, 4.6.2, Java 11 & NextFlow 23.10.0 (grace PR: {2023.06}[system,grace] EasyBuild 4.8.2, 4.9.0, 4.9.1, 4.9.2, 4.9.3, 4.9.4, NextFlow 23.10.0, ReFrame 4.3.3, ReFrame 4.6.2 #981; sapphirerapids PRs: {2023.06}[system,sapphirerapids] EasyBuild 4.8.2, 4.9.0, 4.9.1, 4.9.2, 4.9.3, 4.9.4 #921 & 928)

---

We will continue with the toolchains foss/2023b and foss/2023a and document progress in comments progress with foss/2023b and progress with foss/2023a, respectively. (later we may also look into foss/2022b)

[TopRichard]

Reference: https://gitlab.com/eessi/support/-/wikis/building_add_new_cpu_target

[bedroge]

EESSI/docs#414

Grace needs to be excluded from the software overview page v

[trz42]

EESSI/docs#414

Grace needs to be excluded from the software overview page v

See EESSI/docs#415

[trz42]

Comment for installation of toolchain foss/2023b and all packages built with it.

Need to be careful when building the following packages because rebuilding doesn't work on jureca currently:

• scikit-build-core-0.9.3-GCCcore-13.2.0.eb (see {2023.06}[2023b] Rebuild scikit-build-core to include additional extension #957)

Attempt 1 (removed on 2025-03-25)

• add foss/2023b (grace PR: {2023.06}[2023b,grace] foss 2023b #977, sapphirerapids PR: {2023.06}[2023b,sapphirerapids] foss 2023b #926)
  • Post-ingestion analysis of from-pr vs from-commit
    • GCCcore/13.2.0 was built with from-pr: 19974, however the ec used includes two additional patches

      'GCCcore-12.x_riscv_multiarch_support.patch',
      'GCCcore-13.2.0_fix-unguarded-is_convertible-builtin.patch',
      

      The first patch only applies to RISC-V, while the latter applies to ARM. The latter is very small though, so it is considered a minimal risk to not rebuild that one.
    • OpenMPI/4.1.6 was built with from-pr:19940, however the ec hasn't changed since then.
• NOPE (starting over with attempt 2) before continuing with {2023.06}[2023b,grace] Remaining apps originally built with EB 4.9.0 #978, we need to rebuild foss/2023b to ensure we don't use a more recent easyconfig version
• NOPE (starting over with attempt 2) add remaining apps built with EB 4.9.0 (grace PR: {2023.06}[2023b,grace] Remaining apps originally built with EB 4.9.0 #978, sapphirerapids: {2023.06}[2023b,sapphirerapids] Remaining EB 4.9.0 2023b apps #931)

Attempt 2

• add foss/2023b (grace PR: {2023.06}[2023b,grace] foss/2023b #982, sapphirerapids PR: {2023.06}[2023b,sapphirerapids] foss 2023b #926)
• remaining apps originally built with EB 4.9.0 (grace PR: {2023.06}[2023b,grace] Remaining apps originally built with EB 4.9.0 #978, sapphirerapids PR: {2023.06}[2023b,sapphirerapids] Remaining EB 4.9.0 2023b apps #931)
• apps originally built with EB 4.9.1 (grace PR: {2023.06}[2023b,grace] Apps originally built with EB 4.9.1 #987, sapphirerapids PR: {2023.06}[2023b,sapphirerapids] Add 2023b easystack for EB 4.9.1 #936)
  • for scikit-build-core-0.9.3-GCCcore-13.2.0.eb we used a new from-commit to avoid a need to rebuild it (see {2023.06}[2023b] Rebuild scikit-build-core to include additional extension #957)
• apps originally built with EB 4.9.2 (grace PR: {2023.06}[2023b,grace] apps originally built with EB 4.9.2 + changes to bot/build.sh and eessi_container.sh #989, sapphirerapids PR: {2023.06}[2023b,sapphirerapids] Add EB 4.9.2 easystack for 2023b #939)
• apps originally built with EB 4.9.3 (grace PR: {2023.06}[2023b,grace] apps originally built with EB 4.9.3 #992, sapphirerapids PR: {2023.06}[2023b,sapphirerapids] Add EB 4.9.3 easystack for 2023b #946)
• a couple of apps built with EB 4.9.4 (grace PR: {2023.06}[2023b,grace] a couple of apps built with EB 4.9.4 #993, sapphirerapids PR: {2023.06}[2023b,sapphirerapids] EB 4.9.4 easystack for 2023b #947)
• a few more apps built with EB 4.9.4 (grace PR: {2023.06}[system,2023b,grace] a few remaining apps for the system and 2023b toolchains #996, no sapphirerapids PR)
• CUDA, UCX-CUDA, UCC-CUDA, OSU benchmarks with CUDA (grace PR: {2023.06}[2023b,CUDA,grace] CUDA/12.4, UCX-CUDA/1.15.0, UCC-CUDA/1.2.0, OSU-Micro-Benchmarks-7.5 w/ CUDA #997, no sapphirerapids PR)
• GROMACS 2024.4 with CUDA (grace PR: {2023.06}[2023b,grace,CUDA] GROMACS 2024.4 with CUDA 12.4.0 #999, no sapphirerapids PR)

[TopRichard]

Comment for installation of toolchain foss/2023a and all packages built with it.

To avoid re-building software within the 2023a stack, we have to be careful to use from-commitwhile building:

SciPy-bundle-2023.07-gfbf-2023a
R-bundle-CRAN-2023.12-foss-2023a
R-bundle-Bioconductor-3.18-foss-2023a-R-4.3.2

For now we have skipped LAMMPS-2Aug2023_update2-foss-2023a-kokkos-CUDA-12.1.1.eb as it fails to build with error:
#1012 (comment)
Different versions(stable + feature) have been tested, but none supports ararch64 with CUDA GPUs yet

Apps from EB 4.8.2 2023a easystack

• add GCCcore/12.3.0 grace PR:  {2023.06}[2023a,grace] GCCcore 12.3.0 #979 , sapphirerapids PR: {2023.06}[2023a,sapphirerapids] foss 2023a #925
• add foss/2023a grace PR:  {2023.06}[2023a,grace] foss 2023a, Rust 1.70.0 #983 , sapphirerapids PR: {2023.06}[2023a,sapphirerapids] foss 2023a #925
• add apps grace PR:  {2023.06}[2023a,grace] Apps from EB 4.8.2 2023a easystack #985 , {2023.06}[2023a,grace] Remaining apps from EB 4.8.2 2023a easystack #986 sapphirerapids PR: {2023.06}[2023a,sapphirerapids] Remaining apps for 2023a with EB 4.8.2 #929
  - added from-commit to SciPy-bundle 2023.07
  - added from-commit to Boost-1.82.0 and Boost.Python-1.82.0

Apps from EB 4.9.0 2023a easystack

• add apps grace PR:  {2023.06}[2023a,grace] Apps from EB 4.9.0 2023a easystack #990 , sapphirerapids PR: {2023.06}[2023a,sapphirerapids] Batch of EB 4.9.0 2023a apps #932
• add apps grace PR: {2023.06}[2023a,grace] Further apps from EB 4.9.0 2023a easystack #991 + {2023.06}[2023a,grace] Remaining apps from EB 4.9.0 2023a easystack #995   ,sapphirerapids PR: {2023.06}[2023a,sapphirerapids] Add remaining EB 490 2023a apps #934
  - added from-commit to libxc-6.2.2
  - added from-commit to ParMETIS-4.0.3

Apps from EB 4.9.1 2023a easystack

• add apps grace PR:  {2023.06}[2023a,grace] Apps from EB 4.9.1 2023a easystack #998, {2023.06}[2023a,grace] rebuild R-bundle-Bioconductor-3.18 in 2023a easystack #1007 , sapphirerapids PR: {2023.06}[2023a,sapphirerapids] R-bundle-Bioconductor 3.18 #944
  - added from-commit to R-bundle-Bioconductor/3.18

Apps from EB 4.9.2 2023a easystack

• add apps grace PR:  {2023.06}[2023a,grace] Apps from EB 4.9.2 2023a easystack #1002 , sapphirerapids PR: {2023.06}[2023a,sapphirerapids] OpenFOAM 10 and 11 #942 {2023.06}[2023a,sapphirerapids] Batch of apps for EB 4.9.2 / 2023a #945 {2023.06}[2023a,sapphirerapids] Remaining apps for EB 4.9.2 2023a easystack #954

Apps from EB 4.9.3 2023a easystack

• add apps grace PR:  {2023.06}[2023a,grace] Apps from EB 4.9.3 2023a easystack #1004 , sapphirerapids PR: {2023.06}[2023a,sapphirerapids] Add EB 4.9.3 easystack for 2023a #952

Apps from EB 4.9.4 2023a easystack

• add apps grace PR:  {2023.06}[2023a,grace] Apps from EB 4.9.4 2023a easystack #1006 , sapphirerapids PR: {2023.06}[foss/2023a] waLBerla 6.1 #779, {2023.06}[2023a] mpl-ascii v0.10.0 #791, {2023.06}[2023a] jedi 0.19.0 #794, {2023.06}[foss/2023a] Cassiopeia v2.0.0 #806, {2023.06}[2023a] OpenFOAM v2406 #826, {2023.06}[foss/2023a] LightGBM v4.5.0 #829, {2023.06}[2023a] Paraver 4.11.4 #570, {2023.06}[foss/2023a] Tombo v1.5.1 #838, {2023.06}[2023a] elfx86exts 0.6.2 #846, {2023.06}[2023a] archspec 0.2.5 #905, {2023.06}[2023a] OpenCV 4.8.1 #959, {2023.06}[2023a] Add FALL3D 9.0.1 #970, {2023.06}[foss/2023a] add 'lit' a new missing dependency for R-bundle-CRAN/2023.12-foss-2023a #976

GDRCopy-2.3.1 ,Perl-bundle-CPAN-5.36.1 and Z3/4.12.2-GCCcore-12.3.0-Python-3.11.3 2023a easystack

• add GDRCopy-2.3.1-GCCcore-12.3.0 grace PR: {2023.06}[2023a,grace] add GDRCopy/2.3.1 in 2023a easystack #1015, sapphirerapids PR: {2023.06}[2023a] Explicitly add missing GDRCopy and Z3 versions to EB 4.9.0 easystack (and build them for zen4+sapphirerapids) #960
• add Perl-bundle-CPAN-5.36.1-GCCcore-12.3.0 grace PR: {2023.06}[2023a,grace] add Perl-bundle-CPAN/5.36.1 in 2023a easystack #1016, no sapphirerapids PR
  In the latest easyconfig, Time::HiRes extension has been removed, we will use EB 4.9.2 to build Perl-bundle-CPAN-5.36.1-GCCcore-12.3.0 and stay in sync with builds for other architectures in EESSI, Perl-bundle-CPAN is a dependency of BioPerl
• add Z3/4.12.2-GCCcore-12.3.0-Python-3.11.3 PR: {2023.06}[2023a,grace] Z3 v4.12.2 #1045, sapphirerapids PR: {2023.06}[2023a] Explicitly add missing GDRCopy and Z3 versions to EB 4.9.0 easystack (and build them for zen4+sapphirerapids) #960

CUDA + apps

• add CUDA apps grace/hopper PR: {2023.06}[2023a,CUDA,grace] CUDA/12.1.1, cuDNN/8.9.2.26, CUDA-Samples/12.1 + apps with CUDA #1012, no sapphirerapids PR

Keeping up with EESSI stack:

• add apps recently added to EESSI with EB 4.9.4 PR: {2023.06}[2023a,grace] sync with EESSI EB 4.9.4 easystack  #1018

• add Siesta/5.2.2 grace with EB 5.0.0 PR: {2023.06}[2023a,grace] Rebuild Siesta/5.2.2 in 2023a easystack #1020

• Register known issues for 2023a on NVIDIA/Grace {2023.06}[2023a,grace] Register known issues for 2023a on NVIDIA/Grace #1021

[bedroge]

@TopRichard and @trz42 want to restart from scratch, so I've wiped the existing nvidia/grace stack.

[trz42]

Comment for installation of toolchain foss/2022b and all packages built with it.

Sapphire Rapids PRs to consider (incl rebuilds):

• {2023.06}[2022b,sapphirerapids] foss 2022b #923
  • includes Python-3.10.8-GCCcore-12.2.0-bare: check if EB 4.9.4 includes updated easyblock from PR 3352
• {2023.06}[2022b,sapphirerapids] HarfBuzz 5.3.1, Qt5 5.15.7, QuantumESPRESSO 7.2 #927
  • includes Python-3.10.8-GCCcore-12.2.0: check if EB 4.9.4 includes updated easyblock from PR 3352
• {2023.06}[2022b,sapphirerapids] Easystack for 2022b with EB 4.9.0 #930
  • probably need to use from commit for 2 Boost easyconfigs
• {2023.06}[2022b,sapphirerapids] Add 2022b easystack for EB 4.9.1 #935
• Rebuild Python with ctypes fix for Sapphire Rapids #938
  • make sure that we don't need to rebuild

  - Python-3.10.8-GCCcore-12.2.0-bare:
      options:
        # See https://github.com/easybuilders/easybuild-easyblocks/pull/3352
        include-easyblocks-from-commit: 1ee17c0f7726c69e97442f53c65c5f041d65c94f
  - Python-3.10.8-GCCcore-12.2.0:
      options:
        # See https://github.com/easybuilders/easybuild-easyblocks/pull/3352
        include-easyblocks-from-commit: 1ee17c0f7726c69e97442f53c65c5f041d65c94f
  

• {2023.06}[2022b,sapphirerapids] Add EB 4.9.2 easystack for 2022b #940

Need to be careful when building the following packages because rebuilding doesn't work on jureca currently:

• Python-3.10.8-GCCcore-12.2.0-bare (see Rebuild Python with ctypes fix for Sapphire Rapids #938)
• Python-3.10.8-GCCcore-12.2.0 (see Rebuild Python with ctypes fix for Sapphire Rapids #938)

Apps originally built with EB 4.8.2

• foss/2022b, HarfBuzz 5.3.1 (grace PR: {2023.06}[2022b,grace] foss 2022b, HarfBuzz 5.3.1 #1003, sapphirerapids PRs: {2023.06}[2022b,sapphirerapids] foss 2022b #923, {2023.06}[2022b,sapphirerapids] HarfBuzz 5.3.1, Qt5 5.15.7, QuantumESPRESSO 7.2 #927)
• QuantumESPRESSO 7.2 (grace PR: {2023.06}[2022b,grace] QuantumESPRESSO 7.2 #1008, sapphirerapids PRs: {2023.06}[2022b,sapphirerapids] HarfBuzz 5.3.1, Qt5 5.15.7, QuantumESPRESSO 7.2 #927)

Apps originally built with EB 4.8.2, and also built here with EB 4.8.2

• Qt5 5.15.7 (grace PR: {2023.06}[2022b,grace] Qt5 5.15.7 #1005, sapphirerapids PRs: {2023.06}[2022b,sapphirerapids] HarfBuzz 5.3.1, Qt5 5.15.7, QuantumESPRESSO 7.2 #927)

Apps originally built with EB 4.9.0

• a bunch of apps (grace PR: {2023.06}[2022b,grace] apps originally built with EB 4.9.0 #1013, sapphirerapids PR: {2023.06}[2022b,sapphirerapids] Easystack for 2022b with EB 4.9.0 #930)
  • update for SitePackage.lua so module load SciPy-bundle/2023.02-gfbf-2022b (included in PR 1013) prints a message about a unusually high number of failing unit tests (general PR add Lmod hook that prints warning for SciPy-bundle on NVIDIA Grace #1022); the PR also fixes ReFrame test issues on Grace nodes

Apps originally built with EB 4.9.1

• another set of apps (grace PR: {2023.06}[2022b,grace] apps originally built with EB 4.9.1 #1026, sapphirerapids PR: {2023.06}[2022b,sapphirerapids] Add 2022b easystack for EB 4.9.1 #935)

Apps originally built with EB 4.9.2_

• part 1: 47 of 70 apps (grace PR: {2023.06}[2022b,grace] apps originally built with EB 4.9.2 #1029; sapphirerapids PR: {2023.06}[2022b,sapphirerapids] Add EB 4.9.2 easystack for 2022b #940)
• part 2: 23 of 70 apps (grace PR: {2023.06}[2022b,grace] remaining apps originally built with EB 4.9.2 #1031; sapphirerapids PR: {2023.06}[2022b,sapphirerapids] Add EB 4.9.2 easystack for 2022b #940)