Version 2.5 Functional Release

[gdicker1]

This release features a few improvements and changes:

• EarthWorks is now based on the cesm3_0_beta06 tag from ESCOMP/CESM
  • As a result, CHAOS2000dev and F2000devEW now use the CLM60%SP mode. Conflicts occur with CAM70 and CLM50 now.
  • Configuring a case for GPUs is now done entirely through xmlchange commands to set GPU\_TYPE, OPENACC\_GPU\_OFFLOAD, OVERSUBSCRIBE\_GPU, and NGPUS\_PER\_NODE.
• Updates to MPAS-O, MPAS-SI
  • Changes and input data for running at resolutions 15km and above.
  • Reduced answer differences when restarting cases containing MPAS-O and MPAS-SI.
  • MPAS-O is now able to run on GPUs though the results differ from the CPU runs.
• Smaller fixes
  • Caught a bug for cases using MPAS-A with DEBUG=true and OpenACC.
  • Added a workaround for running cases with any of the MPAS cores with NVHPC+OpenMPIv5.x (observed on Narval and Vista clusters).

What’s Changed

• Update 15km ocean file defaults by @gdicker1 in Update 15km ocean file defaults #94
• BFB restartability for mpas-ocean and -seaice by @dazlich in BFB restartability for mpas-ocean and -seaice #99
• Update externals based on ESCOMP/CESM tag 'cesm3_0_beta05' by @gdicker1 in Update externals based on ESCOMP/CESM tag 'cesm3_0_beta05' #106
• Fix loop bounds in EWOrg/MPAS-Model for DEBUG=true runs by @gdicker1 in Fix loop bounds in EWOrg/MPAS-Model for DEBUG=true runs #108
• merge branch Update/ocean3p75 to develop by @dazlich in merge branch Update/ocean3p75 to develop #110
• Update EW compsets, use CLM60 in F2000dev-like compsets by @gdicker1 in Update EW compsets, use CLM60 in F2000dev-like compsets #117
• Better align EWOrg/CAM with upstream tag cam6_4_048 by @gdicker1 in Better align EWOrg/CAM with upstream tag cam6_4_048 #118
• Workaround MPAS block decomp reads with OpenMPI v5.x by @gdicker1 in Workaround MPAS block decomp reads with OpenMPI v5.x #119
• Update externals based on ESCOMP/CESM tag 'cesm3_0_beta06' by @gdicker1 in Update externals based on ESCOMP/CESM tag 'cesm3_0_beta06' #115
• Create EarthWorks v2.5 Release by @gdicker1 in Create EarthWorks v2.5 Release #120

Full Changelog: release-ew2.4.001...release-ew2.5

Description of Model Configurations (Compsets)

See EarthWorks Supported Configurations in the GitHub wiki for more details.

Testing

Tested Systems

NSF NCAR’s Derecho Supercomputer

The tests in this release occurred on Derecho.

CPU-only hardware: Derecho’s CPU-only nodes consisted of dual-socket, 64-core, 3rd Gen AMD EPYC™ 7763 Milan processors with 256 GB of DDR4 memory.

CPU/GPU hybrid hardware: Derecho has GPU nodes consisting of single-socket, 64-core, 3rd Gen AMD EPYC™ 7763 Milan processor with 512 GB of DDR4 memory plus 4 NVIDIA A100 GPUs each with 40 GB of onboard memory.

Tested Software Stacks

Compiler Versions

Derecho:

• ifx (Intel OneAPI version 2024.2.1)
• Nvfortran (NVHPC fortran compiler version 24.3)
• Gnufortran (compiler version 12.2.0)

Libraries

Derecho:

Intel

• MPI (Cray MPICH version 8.1.29)
• Parallel-NetCDF (version 1.14.0)
• PIO2 (version 2.6.4)
• ESMF (version 8.8.0)

NVHPC, and GNU compilers

• MPI (Cray MPICH version 8.1.27)
• Parallel-NetCDF (version 1.12.3)
• PIO2 (version 2.6.2)
• ESMF (version 8.6.0)

Testing Results

Derecho create_test Results

To test this release, tests were carried out on Derecho using the ew-rel, ew-pr, and (new) ew-pr-gpu categories as described in the v2.2 release.

ew-rel Results

11 day exact restart tests, CPU-only

• 120km, 32L CHAOS2000dev with NVHPC (Overall: FAIL)

• 120km, 58L CHAOS2000dev with NVHPC (Overall: FAIL)

• 60km CHAOS2000dev with NVHPC (Overall: FAIL)

• 15km, 58L CHAOS2000dev with NVHPC (Overall: FAIL)

  • These tests fail to restart since they use CAM-MPAS with NVHPC/24.3.

• 120km, 32L CHAOS2000dev with Intel (Overall: DIFF)

• 120km, 58L CHAOS2000dev with Intel (Overall: DIFF)

• 60km CHAOS2000dev with Intel (Overall: DIFF)

• 15km, 58L CHAOS2000dev with Intel (Overall: DIFF)

  • These tests all ran without issues but reported differences compared to the results from release-ew2.4.

• 30km CHAOS2000dev with NVHPC and Intel (Overall: FAIL)

  • This case has a conflict with its input data. A fix is coming soon.

ew-pr Results

11 day exact restart tests, CPU-only

• 120km FHS94 with NVHPC (Overall: FAIL)
• 120km CHAOS2000dev with NVHPC (Overall: FAIL)
  • These two tests fail to restart since they use CAM-MPAS with NVHPC/24.3.
• 120km FHS94 with Intel, and GNU (Overall: DIFF)
  • Reports ‘FIELDLIST field lists differ (otherwise bit-for-bit)’.
• 120km CHAOS2000dev with Intel, and GNU (Overall: DIFF)
  • Results changed from last release.

5 day smoke tests, CPU-only

• 120km FKESSLER with NVHPC (Overall: DIFF)
• 120km QPC6 with NVHPC (Overall: DIFF)
• 120km F2000climoEW with NVHPC (Overall: DIFF)
• 120km FullyCoupledEW with NVHPC (Overall: DIFF)
• 120km CHAOS2000dev with NVHPC, Intel, and GNU (Overall: DIFF)
  • Results changed from last release.

ew-pr-gpu Results

These new tests check some important functionality for GPU compsets. Currently this separately checks a GPU CAM-MPAS dycore-only run and another GPU CAM run that uses the Spectral Element dycore with the GPU physics suites.

• 5 day smoke test, 120km FKESSLER with NVHPC on GPU (Overall: DIFF)
  • Reports ‘FIELDLIST field lists differ (otherwise bit-for-bit)’.
• 9 step restart test, 1-deg SE-dycore F2000dev with NVHPC on GPU (Overall: DIFF)
  • Results changed from last release.

Known Issues

• Memory leak for atm+lnd compsets on Derecho with NVHPC/24.9 stacks or newer
  • Additional details: Investigation showed increasing memory usage per simulated day in cases that had active atmosphere and land components when using nvhpc/24.9 and nvhpc/24.11 software stacks. Testing on Derecho has shown stable memory usage with intel-based software stacks and nvhpc/24.3 software stacks.
  • Resolutions affected: all supported resolution/level combinations.
  • See also: EarthWorksOrg/EarthWorks Issue #90
• Compsets using MPAS-A on GPUs are only supported for simple physics in this release.
  • Additional details: Compsets like FHS94 and FKESSLER are supported but non-simple physics compsets like F2000dev currently abort during run-time when using MPAS-A OpenACC due to the results becoming unrealistic.
  • Work around: Don’t use the MPAS-A dycore in CAM with non-simple physics and compiling to run on GPUs.
  • See also: EarthWorksOrg/EarthWorks Issue #82
• Compsets involving MPAS-O and MPAS-SI fail to match between default runs and restart runs
  • Additional details: Runs involving the MPAS-O and MPAS-SI components in EarthWorks that begin from restart files of a previous run do not produce the same results. For example, the results are different when comparing a ‘base’ run of 11 simulated days (that writes a restart on day 6) to a ‘restart’ run from day 6 to day 11.
  • Work around: None currently
  • See also: EarthWorksOrg/EarthWorks Issue #66
• NVHPC compilers (tested nvfortran version 24.3, continues from previous releases): Initializing CAM-MPAS from restart fails.
  • Additional details: Any configuration using MPAS-A dycore (tested with QPC6, F2000ClimoEW, FullyCoupledEW) that attempts to restart from a previous run will fail in CAM subroutine dyn_init.
  • Resolutions affected: all supported resolution/level combinations.
  • Work around: Run without restart.
  • See also: EarthWorksOrg/EarthWorks Issue #21

---

This discussion was created from the release Version 2.5 Functional Release.