Build and Run on CCI NGH

This document is a guide to building Omega_h with Kokkos with support for shared memory spaces on CCI's NVIDIA GraceHopper nodes. This documents assumes you are already logged into a node. All of the following commands are run from the same working directory, unless otherwise specified.

Without libMeshb

Environment Setup

First you must load the NVIDIA HPC Software Development Kit (SDK):

module use /opt/nvidia/hpc_sdk/modulefiles
module spider
module load nvhpc

Obtaining Sources

To clone the sources with Git, you must enable the HTTP proxy:

export http_proxy=http://proxy:8888
export https_proxy=$http_proxy

You can then clone Kokkos and Omega_h with Git respectively:

git clone https://github.com/kokkos/kokkos.git
git clone https://github.com/SCOREC/omega_h.git

Configuring, Building and Installing Kokkos

You can use CMake to configure Kokkos with CUDA:

cmake                                       \
    -S kokkos                               \
    -B kokkos-build                         \
    -DCMAKE_CXX_COMPILER=g++                \
    -DCMAKE_INSTALL_PREFIX=kokkos-install   \
    -DKokkos_ENABLE_CUDA=ON                 \
    -DBUILD_SHARED_LIBS=OFF

Then build and install Kokkos with:

cmake --build kokkos-build --target install -j8

Configuring, Building and Testing Omega_h

The following configures Omega_h with support for shared memory spaces and enabled the memory pool for the tests.

cmake                               \
    -S omega_h/                     \
    -B omega_h-build                \
    -DCMAKE_CXX_COMPILER=g++        \
    -DOmega_h_USE_Kokkos=ON         \
    -DKokkos_PREFIX=kokkos-install  \
    -DOMEGA_H_MEM_SPACE_SHARED=ON   \
    -DBUILD_TESTING=ON              \
    -DENABLE_CTEST_MEMPOOL=ON       \
    -DBUILD_SHARED_LIBS=OFF         \
    -DCMAKE_CXX_EXTENSIONS=OFF

Build Omega_h with:

cmake --build omega_h-build -j8

Switch to the omega_h-build directory and run CTest:

cd omega_h-build && ctest

With libMeshb

Configuring, Building and Installing libMeshb

The environment setup is the same, and the HTTP proxy must be enabled if cloning with Git. This assumes you've already built and installed Kokkos from above. The following clones libMeshb with Git:

git clone https://github.com/LoicMarechal/libMeshb.git

You can configure libMeshb with:

cmake -S libMeshb/ -B libMeshb-build -DCMAKE_INSTALL_PREFIX=libMeshb-install -DCMAKE_C_COMPILER=gcc

Then, build and install libMeshb with:

cmake --build libMeshb-build/ --target install -j8

Configuring, Building and Testing Omega_h

The following configures Omega_h similarly to before, but with support for libMeshb:

cmake                                  \
    -S omega_h/                        \
    -B omega_h-build-libmeshb          \
    -DCMAKE_CXX_COMPILER=g++           \
    -DOmega_h_USE_Kokkos=ON            \
    -DKokkos_PREFIX=kokkos-install     \
    -DOMEGA_H_MEM_SPACE_SHARED=ON      \
    -DOmega_h_USE_libMeshb=ON          \
    -DlibMeshb_PREFIX=libMeshb-install \
    -DBUILD_TESTING=ON                 \
    -DENABLE_CTEST_MEMPOOL=ON          \
    -DBUILD_SHARED_LIBS=OFF            \
    -DCMAKE_CXX_EXTENSIONS=OFF

Install Git LFS and the parallel-adapt-wing case

Download and copy the latest Git LFS archive for Linux arm64 to your node. At the time this was written, I used git-lfs-linux-arm64-v3.5.1.tar.gz:

curl -LO https://github.com/git-lfs/git-lfs/releases/download/v3.5.1/git-lfs-linux-arm64-v3.5.1.tar.gz

Unarchive with:

tar xvf git-lfs-linux-arm64-v3.5.1.tar.gz

Replace git-lfs-linux-arm64-v3.5.1.tar.gz with your archive name if using a different version.

Add the git-lfs binary to your path:

cd git-lfs-3.5.1/ # Or whichever your resulting directory is
PATH="$PATH:$(pwd)" # Add git-lfs to PATH
cd - # Return to the previous working directory

Initialize Git LFS:

git lfs install

Finally, clone the parallel-adapt-wing case:

git clone https://github.com/UGAWG/parallel-adapt-results.git

The following script is adapted from Build and Run on Summit which runs the test cases with 50k or 500k tetrahedrons, and with --osh-pool and/or --osh-time:

#!/bin/bash -x

bin=omega_h-build-libmeshb/src
delta=parallel-adapt-results/delta-wing/fun3d-fv-lp2
mesh=$delta/delta50k.meshb
mach=$delta/delta50k-mach.solb
for case in 50k 500k; do
    metric=$delta/scaled-metric/delta${case}-metric.solb
    [ "$case" == "50k" ] && metric=$delta/delta${case}-metric.solb
    for opt in time pool timePool; do
        arg=""
        [ "$opt" == "time" ] && arg="--osh-time" && export CUDA_LAUNCH_BLOCKING=1
        [ "$opt" == "pool" ] && arg="--osh-pool" && unset CUDA_LAUNCH_BLOCKING
        [ "$opt" == "timePool" ] && arg="--osh-time --osh-pool" && export CUDA_LAUNCH_BLOCKING=1
        echo $case $arg
        $bin/ugawg_hsc $arg $mesh $mach $metric $case &> ${case}-${opt}.log
    done
done