Caltech central cluster

Sample sbatch script

THIS PAGE IS OUTDATED Please, update it!

Here is a sbatch script template for setting up simulations using multiple CPUs on Caltech central cluster. Customize the SBATCH instructions if you need more memory/GPUs.

#!/bin/bash
#SBATCH --ntasks=32
#SBATCH --job-name=[YOUR_JOB_NAME]
#SBATCH --time=2:00:00

module purge
module load julia/1.10.0 cuda/12.2 ucx/1.14.1_cuda-12.2 openmpi/4.1.5_cuda-12.2 nsight-systems/2023.3.1

CA_PATH=[YOUR_CLIMAATMOS_PATH]
CA_EXAMPLE=$CA_PATH'examples/'
DRIVER=$CA_EXAMPLE'hybrid/driver.jl'
CONFIG_FILE=[YOUR_CONFIG_FILE]

export OPENBLAS_NUM_THREADS=1
export JULIA_NVTX_CALLBACKS=gc
export OMPI_MCA_opal_warn_on_missing_libcuda=0
export JULIA_MAX_NUM_PRECOMPILE_FILES=100
export JULIA_LOAD_PATH="${JULIA_LOAD_PATH}:${CA_PATH}.buildkite"
export CLIMACORE_DISTRIBUTED="MPI"
export SLURM_KILL_BAD_EXIT=1
export MPITRAMPOLINE_LIB="/groups/esm/software/MPIwrapper/ompi4.1.5_cuda-12.2/lib64/libmpiwrapper.so"
export MPITRAMPOLINE_MPIEXEC="/groups/esm/software/MPIwrapper/ompi4.1.5_cuda-12.2/bin/mpiwrapperexec"

julia --project=$CA_EXAMPLE -e 'using Pkg; Pkg.instantiate()'
julia --project=$CA_EXAMPLE -e 'using Pkg; Pkg.precompile()'
julia --project=$CA_EXAMPLE -e 'using Pkg; Pkg.status()'

srun julia --project=$CA_EXAMPLE $DRIVER --config_file $CONFIG_FILE

If you have any LocalPreferences.toml file, change it so that the preferences for MPIPreferences are:

[preferences.MPIPreferences]
_format = "1.0"
binary = "MPItrampoline_jll"

This file might be in your repository, or in your julia environment.