Building on NERSC Perlmutter

The following was tested with SCOREC/omega_h reducedThrust@2745b6b4 and Kokkos master@1a3ea28f6.

environment setup

module load PrgEnv-gnu
module load cudatoolkit/11.5
module load cmake/3.24.3
#cuda 11.7 is loaded/available by default
#craype-accel-nvidia80 is loaded by default

build

libmeshb - if running deltawing adapt benchmark

git clone git@github.com:LoicMarechal/libMeshb.git
bdir=build-libMeshb-perlGcc11
cmake -S libMeshb -B $bdir \
  -DCMAKE_C_COMPILER=gcc \
  -DBUILD_SHARED_LIBS=on \
  -DCMAKE_Fortran_COMPILER=gfortran \
  -DCMAKE_INSTALL_PREFIX=$bdir/install
cmake --build $bdir --target install -j8

Kokkos

bdir=build-kokkos-perlGcc11
cmake -S kokkos -B $bdir \
  -DBUILD_SHARED_LIBS=ON \
  -DCRAYPE_LINK_TYPE=dynamic \
  -DCMAKE_CXX_COMPILER=$PWD/kokkos/bin/nvcc_wrapper \
  -DKokkos_ARCH_AMPERE80=ON \
  -DKokkos_ENABLE_SERIAL=ON \
  -DKokkos_ENABLE_OPENMP=off \
  -DKokkos_ENABLE_CUDA=on \
  -DKokkos_ENABLE_CUDA_LAMBDA=on \
  -DKokkos_ENABLE_DEBUG=off \
  -DCMAKE_INSTALL_PREFIX=$bdir/install

Omega_h - MPI off

#!/bin/bash -ex

usage="Usage: $0  <mpi=on|off> <libmeshb=on|off>"
[[ $# -ne 2 ]] && echo $usage && exit 1

mpi=$1
[[ $mpi != "on" && $mpi != "off" ]] && echo $usage && exit 1

libmeshb=$2
[[ $libmeshb != "on" && $libmeshb != "off" ]] && echo $usage && exit 1

bdir=build-omegah-perlGcc11
cmake -S omega_h -B $bdir \
  -DCMAKE_INSTALL_PREFIX=$PWD/install \
  -DBUILD_SHARED_LIBS=on \
  -DOmega_h_USE_Kokkos=on \
  -DOmega_h_CUDA_ARCH=80 \
  -DOmega_h_USE_MPI=$mpi \
  -DOmega_h_USE_libMeshb=$libmeshb \
  -DBUILD_TESTING=on \
  -DCMAKE_CXX_COMPILER=CC

cmake --build $bdir -j8

delta wing benchmark

clone repo of inputs:

git lfs install #there will be some errors about this not being a git repo - ignore them /flamesuiton
git clone git@github.com:UGAWG/parallel-adapt-results.git 

create slurm run script runDelta.sh and submit with sbatch runDelta.sh. Note, the directory of inputs, the binary ugawg_hsc, and the slurm run script are all expected to be in the $PSCRATCH/omegahDeltaWingAdapt directory in perlmutter scratch.

#!/bin/bash
#SBATCH -A <YourProject>
#SBATCH -C gpu
#SBATCH -q debug
#SBATCH -t 00:10:00
#SBATCH -n 1
#SBATCH -c 32
#SBATCH --gpus-per-task 1
#SBATCH -J delta
#SBATCH -o delta.%J
#SBATCH -e delta.%J

wdir=$PSCRATCH/omegahDeltaWingAdapt/
source $wdir/envPerlGnu112.sh
bin=$wdir/
delta=$wdir/parallel-adapt-results/delta-wing/fun3d-fv-lp2
mesh=$delta/delta50k.meshb
mach=$delta/delta50k-mach.solb
run="srun"
for case in 50k 500k; do
  metric=$delta/scaled-metric/delta${case}-metric.solb
  [ "$case" == "50k" ] && metric=$delta/delta${case}-metric.solb
  for opt in time pool timePool; do
    arg=""
    [ "$opt" == "time" ] && arg="--osh-time" && export CUDA_LAUNCH_BLOCKING=1
    [ "$opt" == "pool" ] && arg="--osh-pool" && unset CUDA_LAUNCH_BLOCKING
    [ "$opt" == "timePool" ] && arg="--osh-time --osh-pool" && export CUDA_LAUNCH_BLOCKING=1 
    echo $case $arg
    $run $bin/ugawg_hsc $arg $mesh $mach $metric $case &> ${case}-${opt}.log
  done
done

Issues

Building with PrgEnv-nvidia/8.2.0 (Nvidia HPC SDK 21.7) and the default cuda/11.3 results in a hang during the compilation of Omega_h_adj.cpp. Switching the cuda module to cuda/11.1.1 or cuda/11.0.3 results in the following compatibility errors:

In file included from /global/homes/c/cwsmith/omegahHackathon/omega_h/src/Omega_h_library.cpp:31:0:
/opt/nvidia/hpc_sdk/Linux_x86_64/21.7/compilers/include/xmmintrin.h:25:2: error: #error "This version of <xmmintrin.h> does not work with this compiler."
 #error "This version of <xmmintrin.h> does not work with this compiler."