Legate MPI wrapper CMakeLists.txt

[saroj-lbl]

Hello,

I am following https://docs.nvidia.com/legate/latest/installation.html#installation-of-the-legate-mpi-wrapper to install MPI wrapper on legate 24.11 install and get the following error when running install.bash

CMake Error: The source directory ".../legate/share/legate/mpi_wrapper" does not appear to contain CMakeLists.txt.

Is the CMakeLists.txt file missing from the legate package?

Thank you

[manopapad]

Yes, you're right, since this documentation was written we've separated out this CMakeLists.txt from the main package, and the documentation was unfortunately not updated.

Please try also installing in the same environment the package legate-mpi-wrapper like so (note the different channel, -c legate/label/gex instead of -c legate):

conda install -c legate/label/gex legate-mpi-wrapper

then re-activate your environment, build the wrapper, and reactivate again to make sure everything is loaded properly:

conda deactivate
conda activate <myenv>
$CONDA_PREFIX/mpi-wrapper/build-mpi-wrapper.sh
conda deactivate
conda activate <myenv>

It might also be a good idea to remove the conda openmpi package, to make sure Legate doesn't get confused and loads that instead of your system's MPI:

conda uninstall --force openmpi

This is all planned to become much easier after some package restructuring, but this is the situation as it stands today.

If the above fix works for you, I'll try to push the fixes to the docs ASAP, but it may take a while, as a lot of people are out of the office right now.

[marcinz]

Actually, there are two sets of instructions. The instructions at https://docs.nvidia.com/legate/latest/installation.html#installation-of-the-legate-mpi-wrapper should still be correct, but they assume a source install. The instructions for GASNet-based installation are here: https://docs.nvidia.com/legate/latest/gasnet.html. This may be a little confusing, so maybe we should make it clear that the instructions in the first link are for a source build.

[saroj-lbl]

Thanks,

I tried legate-mpi-wrapper from legate/label/gex. I had to make a small change to build-mpi-wrapper.sh

• Line 6: compiler="CC" to compiler="$CC"

Additionally, I found that errexit is being set when I activate the conda environment for legate leading to closing my ssh connection to the cluster on tab completion.

After changing to $CC, I was able to install the wrapper but the location of the wrapper installed went to $PREFIX/$PREFIX/lib64 rather than $PREFIX/lib64. I will try to see if things work out by just copying the files (likely after the holidays).

[manopapad]

I had to make a small change to build-mpi-wrapper.sh

• Line 6: compiler="CC" to compiler="$CC"

@marcinz is this by design? I can imagine we're trying to do the right thing for Cray, and they might be using the actual alias CC (rather than $CC).

Additionally, I found that errexit is being set when I activate the conda environment for legate leading to closing my ssh connection to the cluster on tab completion.

Yes, this is a known issue, see nv-legate/legate#965

After changing to $CC, I was able to install the wrapper but the location of the wrapper installed went to $PREFIX/$PREFIX/lib64 rather than $PREFIX/lib64.

Could you please share the output of the build script? Also @marcinz to advise.

[saroj-lbl]

The output of the build script is as follows:

(legate) sadhikari@sadhikari-thelio:~$ $CONDA_PREFIX/mpi-wrapper/build-mpi-wrapper.sh 
Building Legate MPI wrapper:
  Installation directory: /home/sadhikari/anaconda3/envs/legate/lib
  Compiler: 
-- The CXX compiler identification is GNU 13.2.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Using build type: Release
-- Building shared library: ON
-- Generating src install rules: ON
-- Found MPI_CXX: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so (found version "3.1")
-- Found MPI: TRUE (found version "3.1") found components: CXX
-- Configuring done (1.1s)
-- Generating done (0.0s)
-- Build files have been written to: /home/sadhikari/anaconda3/envs/legate/mpi-wrapper/build
[ 50%] Building CXX object CMakeFiles/mpi_wrapper.dir/src/legate_mpi_wrapper/mpi_wrapper.cc.o
[100%] Linking CXX shared library lib/liblegate_mpi_wrapper.so
[100%] Built target mpi_wrapper
-- Install configuration: "Release"
-- Installing: /home/sadhikari/anaconda3/envs/legate/mpi-wrapper/home/sadhikari/anaconda3/envs/legate/mpi-wrapper/lib/liblegate_mpi_wrapper.so.1
-- Set non-toolchain portion of runtime path of "/home/sadhikari/anaconda3/envs/legate/mpi-wrapper/home/sadhikari/anaconda3/envs/legate/mpi-wrapper/lib/liblegate_mpi_wrapper.so.1" to ""
-- Up-to-date: /home/sadhikari/anaconda3/envs/legate/mpi-wrapper/home/sadhikari/anaconda3/envs/legate/mpi-wrapper/lib/liblegate_mpi_wrapper.so
-- Up-to-date: /home/sadhikari/anaconda3/envs/legate/mpi-wrapper/home/sadhikari/anaconda3/envs/legate/mpi-wrapper/include/legate_mpi_wrapper/legate_mpi_wrapper/mpi_wrapper.h
-- Up-to-date: /home/sadhikari/anaconda3/envs/legate/mpi-wrapper/home/sadhikari/anaconda3/envs/legate/mpi-wrapper/include/legate_mpi_wrapper/legate_mpi_wrapper/mpi_wrapper_types.h
-- Installing: /home/sadhikari/anaconda3/envs/legate/mpi-wrapper/home/sadhikari/anaconda3/envs/legate/mpi-wrapper/lib/cmake/legate_mpi_wrapper/mpi_wrapperTargets.cmake
-- Installing: /home/sadhikari/anaconda3/envs/legate/mpi-wrapper/home/sadhikari/anaconda3/envs/legate/mpi-wrapper/lib/cmake/legate_mpi_wrapper/mpi_wrapperTargets-release.cmake
-- Installing: /home/sadhikari/anaconda3/envs/legate/mpi-wrapper/home/sadhikari/anaconda3/envs/legate/mpi-wrapper/lib/cmake/legate_mpi_wrapper/mpi_wrapperConfig.cmake
-- Installing: /home/sadhikari/anaconda3/envs/legate/mpi-wrapper/home/sadhikari/anaconda3/envs/legate/mpi-wrapper/lib/cmake/legate_mpi_wrapper/mpi_wrapperConfigVersion.cmake

Reactivate the conda environment to set the necessary environment variables:

$ conda deactivate
$ conda activate legate

Notice the directory structure for installation is: $CONDA_PREFIX/mpi-wrapper/$CONDA_PREFIX/mpi-wrapper/lib

[marcinz]

@marcinz is this by design? I can imagine we're trying to do the right thing for Cray, and they might be using the actual alias CC (rather than $CC).

Yes, this is assuming the HPE Cray Programming Environment (CPE) suite to be active and one of the environments to be activated with the compiler wrappers (cc and CC) available. The Cray compiler wrappers do "the magic," making everything that is available through CPE automatically available without any options. E.g., on Cray, we use libfabric through the wrapper. This is only the default though, and the build-mpi-wrapper.sh script from the packge has a -c option to specify the compiler (also the -h option to display the help message).

Could you please share the output of the build script? Also @marcinz to advise.

This is an artifact of setting an install prefix for CMake. It puts the lib directory into the prefix we give it, which is the directory in which the build script is located. Assuming this is all used through our packages, the activation script for the package actually performs a search for the built library, so it does not really matter where the library is installed. If this is used "manually," outside of the package setup, the path to the wrapper has to be in the LEGATE_MPI_WRAPPER environment variable, and Legate will find it.

Yes, this is a known issue, see nv-legate/legate#965

I will take a look at this.