Building on NERSC Perlmutter

The following was tested with reducedThrust@2745b6b4.

environment setup

module load PrgEnv-gnu
module load cudatoolkit/11.5
module load cmake/3.24.3
#cuda 11.7 is loaded/available by default
#craype-accel-nvidia80 is loaded by default

build and test

Kokkos

bdir=build-kokkos-perlGcc11
cmake -S kokkos -B $bdir \
  -DBUILD_SHARED_LIBS=ON \
  -DCRAYPE_LINK_TYPE=dynamic \
  -DCMAKE_CXX_COMPILER=$PWD/kokkos/bin/nvcc_wrapper \
  -DKokkos_ARCH_AMPERE80=ON \
  -DKokkos_ENABLE_SERIAL=ON \
  -DKokkos_ENABLE_OPENMP=off \
  -DKokkos_ENABLE_CUDA=on \
  -DKokkos_ENABLE_CUDA_LAMBDA=on \
  -DKokkos_ENABLE_DEBUG=off \
  -DCMAKE_INSTALL_PREFIX=$bdir/install

Omega_h - MPI off

#!/bin/bash -ex

usage="Usage: $0  <mpi=on|off>"
[[ $# -ne 1 ]] && echo $usage && exit 1

mpi=$1
[[ $mpi != "on" && $mpi != "off" ]] && echo $usage && exit 1

bdir=build-omegah-perlGcc11
cmake -S omega_h -B $bdir \
  -DCMAKE_INSTALL_PREFIX=$PWD/install \
  -DBUILD_SHARED_LIBS=on \
  -DOmega_h_USE_Kokkos=on \
  -DOmega_h_CUDA_ARCH=80 \
  -DOmega_h_USE_MPI=$mpi \
  -DOmega_h_USE_libMeshb=on \
  -DBUILD_TESTING=on \
  -DCMAKE_CXX_COMPILER=CC

cmake --build $bdir -j8

Issues

Building with PrgEnv-nvidia/8.2.0 (Nvidia HPC SDK 21.7) and the default cuda/11.3 results in a hang during the compilation of Omega_h_adj.cpp. Switching the cuda module to cuda/11.1.1 or cuda/11.0.3 results in the following compatibility errors:

In file included from /global/homes/c/cwsmith/omegahHackathon/omega_h/src/Omega_h_library.cpp:31:0:
/opt/nvidia/hpc_sdk/Linux_x86_64/21.7/compilers/include/xmmintrin.h:25:2: error: #error "This version of <xmmintrin.h> does not work with this compiler."
 #error "This version of <xmmintrin.h> does not work with this compiler."