Build and Run on CCI NGH

Build and Run on CCI NGH

This document is a guide to building Omega_h with Kokkos with support for shared memory spaces on CCI's NVIDIA GraceHopper nodes. This documents assumes you are already logged into a node. All of the following commands are run from the same working directory, unless otherwise specified.

Environment Setup

First you must load the NVIDIA HPC Software Development Kit (SDK):

module use /opt/nvidia/hpc_sdk/modulefiles
module spider
module load nvhpc

Obtaining Sources

To clone the sources with Git, you must enable the HTTP proxy:

export http_proxy=http://proxy:8888
export https_proxy=$http_proxy

You can then clone Kokkos and Omega_h with Git respectively:

git clone https://github.com/kokkos/kokkos.git
git clone https://github.com/SCOREC/omega_h.git

Configuring, Building and Installing Kokkos

You can use CMake to configure Kokkos with CUDA:

cmake                                       \
    -S kokkos                               \
    -B kokkos-build                         \
    -DCMAKE_CXX_COMPILER=g++                \
    -DCMAKE_INSTALL_PREFIX=kokkos-install   \
    -DKokkos_ENABLE_CUDA=ON                 \
    -DBUILD_SHARED_LIBS=OFF

Then build and install Kokkos with:

cmake --build kokkos-build --target install -j8

Configuring, Building and Testing Omega_h

The following configures Omega_h with support for shared memory spaces and enabled the memory pool for the tests.

cmake                               \
    -S omega_h/                     \
    -B omega_h-build                \
    -DCMAKE_CXX_COMPILER=g++        \
    -DOmega_h_USE_Kokkos=ON         \
    -DKokkos_PREFIX=kokkos-install  \
    -DOMEGA_H_MEM_SPACE_SHARED=ON   \
    -DBUILD_TESTING=ON              \
    -DENABLE_CTEST_MEMPOOL=ON       \
    -DBUILD_SHARED_LIBS=OFF         \
    -DCMAKE_CXX_EXTENSIONS=OFF

Build Omega_h with:

cmake --build omega_h-build -j8

Switch to the omega_h-build directory and run CTest:

cd omega_h-build && ctest