Build and Run on OLCF Crusher

Download

git clone -b clang13_rocm452 git@github.com:SCOREC/omega_h.git

Create envCrusherRocm520.sh with the following contents:

module load PrgEnv-cray
module load cmake/3.23.2
module load craype-accel-amd-gfx90a
module load rocm/5.2.0

## HIP and ROCm env
hipcc=`which hipcc`
rocm=${hipcc%%bin/hipcc}
export HIP_PATH=$rocm/hip
export CMAKE_PREFIX_PATH=$rocm:$CMAKE_PREFIX_PATH

Build for MI250x GPUs using hipcc

srcpath=/path/to/omegah/source
source envCrusherRocm520.sh
mkdir build-omega-rocm520
cd $_
hipcc=`which hipcc`
rocm=${hipcc%%bin/hipcc}
cmake $srcpath \
  -DCMAKE_INSTALL_PREFIX=$PWD/install \
  -DBUILD_SHARED_LIBS=OFF \
  -DOmega_h_USE_HIP=ON \
  -DOmega_h_USE_Kokkos=OFF \
  -DCMAKE_CXX_FLAGS="-O2" \
  -DHIP_PATH=${rocm}/hip \
  -DOmega_h_USE_MPI=OFF \
  -DCMAKE_CXX_COMPILER=hipcc \
  -DBUILD_TESTING=ON

make -j4

Run tests

salloc -N 1 -t 10  -A <ProjectId>
# wait for node to be allocated
source envCrusherRocm520.sh
cd /path/to/build-omega-rocm520
ctest  # all tests should pass

Profiling

To run the fun3d delta case omega_h needs to rebuilt with libmeshb support.

libmeshb is here:

https://github.com/LoicMarechal/libMeshb

Installation requires CMake and a C (and Fortran) compiler.

Libmeshb install

Note, the use of the hipcc compiler is required to avoid undefined symbol errors (__cray_sset_detect) when linking Omega_h executables.

module load PrgEnv-cray
module load cmake/3.23.2
module load craype-accel-amd-gfx90a
module load rocm/5.2.0

git clone https://github.com/LoicMarechal/libMeshb

mkdir build-libmeshb
cd $_

cmake ../libMeshb/ \
-DCMAKE_INSTALL_PREFIX=/path/to/buildLibMesh/install \
-DCMAKE_C_COMPILER=hipcc

make -j2 install

Rebuild Omega_h after installing libmeshb

• append the path to the libmeshb install directory to your CMAKE_PREFIX_PATH,
• rebuild Omega_h with -DOmega_h_USE_libMeshb=on passed to cmake.

Download test data

Clone the repo with the fun3d delta case. The repo is 527MB.

module load git-lfs
git lfs install
git clone git@github.com:UGAWG/parallel-adapt-results.git

Run

Create a file named runDeltaOshTime.sh with the following contents; edit the paths for bin and delta.

#!/bin/bash
bin=/path/to/omegah/build/directory
delta=/path/to/parallel-adapt/delta-wing/fun3d-fv-lp2
mesh=$delta/delta50k.meshb
mach=$delta/delta50k-mach.solb 
for case in 50k 500k; do
  metric=$delta/scaled-metric/delta${case}-metric.solb
  [ "$case" == "50k" ] && metric=$delta/delta${case}-metric.solb
  for opt in noTimePool time pool timePool; do
    arg=""
    [ "$opt" == "time" ] && arg="--osh-time" && export HIP_LAUNCH_BLOCKING=1
    [ "$opt" == "pool" ] && arg="--osh-pool" && unset HIP_LAUNCH_BLOCKING
    [ "$opt" == "timePool" ] && arg="--osh-time --osh-pool" && export HIP_LAUNCH_BLOCKING=1 
    echo $case $arg
    set -x
    # c64 is needed to allocate all the cpus for the jobstep
    # map_cpu selects a core that is in the same numa domain as gpu 0 (48-55 or 112-119)
    # gpu binding is not required as the code calls hipSetDevice(0)
    # see https://docs.olcf.ornl.gov/_images/Crusher_Node_Diagram.jpg
    srun -n1 -c64 --cpu-bind=verbose,map_cpu:48 --gpus-per-task=1 $bin/ugawg_hsc $arg $mesh $mach $metric $case &> ${case}-${opt}.log
    set +x
  done
done

Make it executable:

chmod +x runDeltaOshTime.sh

Allocate a node and run:

salloc -N 1 -t 10  -A <project>
source /path/to/envCrusherRocm520.sh
cd /path/to/run/dir
./runDeltaOshTime.sh

Appended to the end of the runs using --osh-time will be a top-down and bottom-up tree of functions sorted by time in descending order.

The --osh-pool argument enables use of an internal memory pool instead of device runtime allocation calls.