Build and Run on SCOREC RHEL9 Workstation

environment

module use /opt/scorec/spack/rhel9/v0201_4/lmod/linux-rhel9-x86_64/Core/
module load \
gcc/12.3.0-iil3lno \
cmake/3.26.3-2duxfcd \
mpich/4.1.1-xpoyz4t \
cuda/12.1.1-zxa4msk 

build kokkos

Note, the -DKokkos_ARCH_<ARCH> flag should be set to match the capability of your GPU. See https://kokkos.org/kokkos-core-wiki/keywords.html#architectures for the available Kokkos CMake options. The following pages provide the architecture and the compute capability of NVIDIA GPUs:

• https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units#Desktop_GPUs
• https://developer.nvidia.com/cuda-gpus

git clone -b develop git@github.com:Kokkos/kokkos.git
bdir=buildKokkosCuda
cmake -S kokkos -B $bdir \
  -DCMAKE_CXX_COMPILER=g++ \
  -DBUILD_SHARED_LIBS=on \
  -DKokkos_ARCH_AMPERE86=on \
  -DKokkos_ENABLE_SERIAL=ON \
  -DKokkos_ENABLE_OPENMP=off \
  -DKokkos_ENABLE_CUDA=on \
  -DKokkos_ENABLE_CUDA_LAMBDA=on \
  -DKokkos_ENABLE_DEBUG=off \
  -DCMAKE_INSTALL_PREFIX=$bdir/install
cmake --build $bdir -j 24 --target install

build and test omegah

As done for Kokkos, the -DOmega_h_CUDA_ARCH flag should be set to match the compute capability of your GPU.

git clone git@github.com:SCOREC/omega_h.git
bdir=buildOmegahCuda
cmake -S omega_h -B $bdir \
  -DCMAKE_INSTALL_PREFIX=$PWD/$bdir/install \
  -DOmega_h_USE_Kokkos=on \
  -DKokkos_PREFIX=$PWD/buildKokkosCuda/install \
  -DOmega_h_CUDA_ARCH=75 \
  -DOmega_h_USE_MPI=on \
  -DBUILD_TESTING=on \
  -DCMAKE_CXX_COMPILER=g++
cmake --build $bdir -j 24
ctest --test-dir $bdir