Version 2.1 Release

[gdicker1]

Version 2.1 Functional Release

Version 2.1 of the EarthWorks Modeling System is a patch release that addresses some issues in the November 2023 Version 2.0 release. In particular, Version 2.1:

• Patches aerosol_optics_cam.F90 for use with nvhpc compiler suite. It is critical for EarthWorks Software to stay synchronized with the evolution of the CESM code base and simultaneously maintain the ability to compile with the NVIDIA compiler (nvhpc) for GPU offload. This patch lifts the software version in the CESM externals from CAM6.3 tag 124 to 145 level.
• Fixes an ifx (Intel OneAPI) compiler error with MPAS-Seaice file mpas_seaice_core_interface.F.
• First release of MPAS-7x GPU-offload. At present, testing of this capability is confined to the FHS94 and F2000Climo compsets and 120 km, 60 km, 30 km resolutions.

NB: All of these configurations should be considered functional releases, i.e. that they have not been scientifically verified.

Description of Model Configurations (Compsets)

See EarthWorks Supported Configurations in the GitHub wiki for more details.

Testing

Tested Systems

Tests were performed on NCAR’s Derecho supercomputer.

CPU-only tests: Derecho’s CPU-only nodes consisted of dual, 64-core, 3rd Gen AMD EPYC™ 7763 Milan processors with 256 GB of DDR4 memory.
CPU/GPU hybrid tests: Derecho has GPU nodes consisting of a single 3rd Gen AMD EPYC™ 7763 Milan processor with 512 GB of DDR4 memory plus 4 NVIDIA A100 GPUs each with 40 GB of onboard memory.

Tested Software Stacks

Compiler Versions

Derecho:

• ifort (Intel Classic compiler version 2021.8)
• ifx (Intel OneAPI version 2023.0)
• Nvfortran (NVHPC fortran compiler version 23.5)
• Gnufortran (compiler version 12.2)

Libraries

Derecho:

• ESMF (8.6)
• PIO2 (2.6.2)
• MPI (Cray MPICH (8.1.25))

Testing Results

Known issues by compset/compiler/resolution (CPU-only):

Supported resolution/level combinations:

• 120km_L32

• 60km_L32

• 30km_L32

• 15km_L58

• NVHPC compilers (tested nvfortran version 23.X): Initializing from restart fails.

  • Additional details: Any configuration (tested with QPC6, F2000Climo, Fully-Coupled) that attempts to restart from a previous run will fail in CAM subroutine dyn_init.
  • Resolutions affected: all supported resolution/level combinations.
  • Work around: Run without restart.

• NVHPC compiler requires patched version of aerosol_optics_cam.F90 module: Module will not build or run correctly without patched version.

  • Additional Details: The nvhpc compiler functionality with tag 145 software was accomplished by making two patches to the aerosol_optics_cam.F90 module. The first patch addresses an internal compiler error encountered in the nvhpc 23.x compilers related to the use of Fortran complex number accessors (see ESCOMP/CAM Issue #881). The second is a runtime error addressed (for the time being) by removing dynamic dispatch polymorphism in the F2003 code in one place in this module (ESCOMP/Cam Issue #945).The team is working with NVIDIA to permanently correct these issues in a future version of the compiler.
  • Resolutions affected: All supported resolution/level combinations involving the atmospheric component (CAM).
  • Work around: Use the patched version of aerosol_optics_cam.F90 distributed in release version 2.1.

• FullyCoupled compset (any compiler): Due to changes in the mpas-framework directory to match with MPAS-7.x OpenACC framework, runs fail.

  • Example error message: ERROR: shr_reprosum_calc ERROR: NaNs or INFs in input

Known issues by compset/compiler/resolution (Hybrid CPU-GPU)

• F2000Climo-GPU-Offload:
  • Configuration notes: Includes PUMAS and MPAS-7.x GPU offload with OpenACC directives. Does not work with nvfortran 24.1 compiler due to an compilation issue with the CICE model. .
  • Supported resolution/level combinations:
    • 120km_L32
    • 60km_L32
    • 30 km_L32
  • Tested on:
    • Derecho GPU-partition
  • Compiler Issues:
    • See: Known NVHPC compilation issues above.

FHS94-GPU-Offload:

• Configuration notes: CAM-Physics not included (e.g. PUMAS GPU offload). So only MPAS-7.x is offloaded to GPUs.
• Supported resolution/level combinations:
  • 120km_L32
  • 60km_L32
  • 30km_L32
• Tested on:
  • Derecho GPU-partition

FullyCoupled Compset GPU-Offload:

• Configuration notes: Builds fail due to redefinition of of routines mpas_dmpar.F.
• Example error message: nvlink error : Multiple definition of 'mpas_dmpar_mpas_dmpar_exch_halo_1d_real_acc_6481_gpu' in '${CASEDIR}/bld/lib/libocn.a:mpas_dmpar.o', first defined in '${CASEDIR}/bld/lib/libice.a:mpas_dmpar.o'

---

This discussion was created from the release Version 2.1 Release.

[areanddee]

I would prioritize the work ahead of us in the following way:

1. Get QPC6 compiling and running on Perlmutter under nvhpc 24.1/OpenMPI. This is needed for the compiler group aerosol_optics_mod reproducer. Even compiling QPC6 is good as it will get NVIDIA started on ESCOMP/CAM #881. Running is needed however for ESCOMP/CAM #945. It is not necessary to run this with GPU flags enabled for the aerosol reproducer.

Note: I think we may have run issues on Perlmutter across the board due to OpenMPI. If so, then 4 becomes a higher priority.

2. I think we need to design out way out of the collisions happening between MPAS-infrastructure files in atm, ocean and sea ice that affect all compilers. This is a pretty high priority, but not as high as the aerosol reproducer, IMO, because we need to be able to build fully coupled configurations - and we can't right now with 2.1 with any compiler.

3. Finding a reproducer for the CICE compiler failure under 24.1. This is blocking issue to maintaining synchronization with CESM tags for the nvhpc compiler unless we can get a reproducer for this issue. Of course, getting to 24.1 is important for enabling EOSHIFT fix in RRTMGP and that will be an issue for upcoming releases.

4. F2000climo should not depend on CICE in the first place: we are not really running F2000Climo as an EarthWorks configuration, as MPAS-{atm,ocn, sea ice} is a defining characteristic for EW software. Assuming a fix to 2), we need to make F2000climo consistent with this principle and with Fully Coupled.

5. We need to sort out, in a broader sense, our issues on Perlmutter. GPU code does not run on Perlmutter, and we have MPI issues even on simple CPU-only configurations on Perlmutter like FHS94. I think we need to make sure our work on the dycore is kept current with the release software, and avoid working within obsolescent sandboxes.

6. Fix run after restart failures. This is a major functionality issue that will prevent running long experiments with the code base. I hate that this is the 5th ranked issue, but we has so many other issues that seem more pressing.

[dazlich]

I've created a PR in mpas-seaice that implements a prescribed seaice cover mode that should address points 3 and 4 above. It is subject to mpas-framework issues which I understand still continue on GPUs even though I've fixed it for CPUs.

The branch is seaice-prescribed-mode.

Any way the MPASSI version of F2000climo for create_newcase is:
--compset 2000_CAM60_CLM50%SP_MPASSI%PRES_DOCN%DOM_MOSART_CISM2%NOEVOLVE_SWAV