Unable to run legate-boost on multi-node Slurm job (multi‑GPU single‑node works fine)

[NihaalChowdary]

I successfully installed legate‑boost on Perlmutter and was able to run workloads on multiple GPUs (single node) and on CPU. However, when attempting to run a multi-node job using Slurm, it fails immediately with an OMPI/PMIx error (see full logs below). I could not find any guidance in the documentation for multi-node setup or troubleshooting.

Documentation includes multi-node flags (e.g. --nodes, --launcher) (docs.nvidia.com) and UCX/GASNet requirements, but does not explicitly address errors related to missing PMIx support in MPI build.

Request for Help

• Could you clarify the recommended launcher and dependencies for running multi-node, multi-GPU jobs with Legate Boost on Slurm clusters (specifically Perlmutter)?
• Does conda-based installation via the gex channel (as above) support multi-node out-of-the-box? Or is a custom build with install.py --network gasnet1 required?
• Any instructions or examples for multi-node scheduling would be extremely helpful.

Thank you for your time and support!

Installation steps
I used the same environment setup commands I had used successfully for other Legate tasks.

$ module load conda
$ conda create -n legate‑boost‑env python=3.10
$ conda activate legate‑boost‑env
$ conda install \
  -c legate/label/gex‑experimental \
  -c legate/label/experimental \
  -c legate \
  -c conda‑forge \
  -c nvidia \
  legate=*=*gex* \
  cupynumeric \
  legate‑mpi‑wrapper \
  realm‑gex‑wrapper
$ conda install -c conda‑forge cmake>=3.22.1
$ /global/homes/n/ngraddon/.conda/envs/legate‑boost‑env/mpi‑wrapper/build‑mpi‑wrapper.sh
$ /global/homes/n/ngraddon/.conda/envs/legate‑boost‑env/gex‑wrapper/build‑gex‑wrapper.sh

$ conda install \
  -c legate \
  -c conda‑forge \
  -c nvidia \
  legate‑boost

working commands:
$ legate --cpu 1 --gpus 0 ./housing.py
$ legate --gpus 4 ./housing.py

Multi-Node Error:
$ legate --nodes 2 --launcher srun --gpus 4 --ranks‑per‑node 1 ./housing.py

Error message:

[nid001217:1593825] OPAL ERROR: Unreachable in file pmix3x_client.c at line 111
The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM's PMI support and therefore cannot
execute. There are several options for building PMI support under
SLURM, depending upon the SLURM version you are using:
  version 16.05 or later: you can use SLURM's PMIx support. This
  requires that you configure and build SLURM --with-pmix
  Versions earlier than 16.05: you must use either SLURM's PMI-1 or
  PMI-2 support. SLURM builds PMI-1 by default, or you can manually
  install PMI-2. You must then build Open MPI using --with-pmi pointing
  to the SLURM PMI library location.
Please configure as appropriate and try again.
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[nid001217:1593825] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: nid001217: task 0: Exited with exit code 1
srun: Terminating StepId=40531224.0
[nid001220:1156306] OPAL ERROR: Unreachable in file pmix3x_client.c at line 111
--------------------------------------------------------------------------
The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM's PMI support and therefore cannot
execute. There are several options for building PMI support under
SLURM, depending upon the SLURM version you are using:
  version 16.05 or later: you can use SLURM's PMIx suppor. This
  requires that you configure and build SLURM --with-pmix
  Versions earlier than 16.05: you must use either SLURM's PMI-1 or
  PMI-2 support. SLURM builds PMI-1 by default, or you can manually
  install PMI-2. You must then build Open MPI using --with-pmi pointing
  to the SLURM PMI library location.
Please configure as appropriate and try again.
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[nid001220:1156306] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: nid001220: task 1: Exited with exit code 1
(legate-boost-env) nihaal@nid001217:~> exit
exit
srun: error: nid001217: task 0: Exited with exit code 1
srun: Terminating StepId=40531224.interactive
salloc: Relinquishing job allocation 40531224

[seberg]

I am not familiar with Perlmutter (or --launcher=srun, but if you get this far that seems good).
However this error is typical if you don't use --mpi=pmix on srun. I'll ping someone more familiar, but I expect what you need to add there is:

legate --nodes 2 --launcher srun --launcher-extra="--mpi=pmix" ...

Maybe the error legate gives here could also be improved, since in practice it seems more likely you need to pass the argument to slurm, rather than ensure that it is compiled with it.

(If we want such improvements, should move the issue to https://github.com/nv-legate/Legate, but we can do that.)

[seberg]

@NihaalChowdary have you been able to get things to run or tried this? I think this is an issue with the MPI layer on the cluster, be we can move it to https://github.com/nv-legate/Legate to get you help (maybe even someone familiar with this cluster).

[NihaalChowdary]

@seberg
I have tried using the suggested flag (--launcher-extra="--mpi=pmix"). It helped resolve the missing PMI support error by enabling PMIx for MPI launch. However, it introduced a new issue where MPI detected unexpected IP addresses during node communication, causing connection failures.

"--------------------------------------------------------------------------
Open MPI detected an inbound MPI TCP connection request from a peer
that appears to be part of this MPI job (i.e., it identified itself as
part of this Open MPI job), but it is from an IP address that is
unexpected.  This is highly unusual.

The inbound connection has been dropped, and the peer should simply
try again with a different IP interface (i.e., the job should
hopefully be able to continue).

  Local host:          nid001180
  Local PID:           1128405
  Peer hostname:       nid001181 ([[7976,3478],1])
  Source IP of socket: 10.249.5.85
  Known IPs of peer:   
	10.100.0.108
	128.55.69.22
	10.249.5.86
	10.249.25.134
	10.249.23.134

"

Thanks a lot for following up on this issue—I really appreciate your help!

[seberg]

Really not sure what's going on there. This in OpenMPI still and I found this issue which seems like it is the same: open-mpi/ompi#12232

Maybe check your OpenMPI version Perlmutter OpenMPI docs:

Only Open MPI 5.0.0 and newer are supported on Perlmutter

and ask briefly locally? This seems like familiarity with Perlmutter may be good and I do not have that (but others here may).

[seberg]

Closing for now, I hope you figured it out.

[dongb]

Please keep it open for a moment. @seberg , could you help to reopen to avoid confusion?

I don't know how to move this one to Legate repo. So create another one in Legate repo.
nv-legate/legate#1005
This issue should be marked as open and block by 1005.

[manopapad]

@XiaLuNV, @marcinz you recently ran on Perlmutter, do we have any flags in our configuration that could address @NihaalChowdary's issues? Alternatively, do you see any other issue in his configuration?

[XiaLuNV]

We had the same issue for about 3 weeks when run the cuPyNumeric tests on Perlmutter, see bug 5406684.
The issue was resolved with the 25.08.00.rc1 packages. You may try this new version, which is available under the -c legate/label/rc channel.

[NihaalChowdary]

Thanks for your input, @XiaLuNV. However, I ran into an issue updating because my current environment uses legate-boost 25.07.00 (CUDA 12, Python 3.12), which doesn’t have a compatible build for legate 25.08.00.rc1. Without a matching legate-boost build for 25.08.00.rc1, Conda is unable to resolve dependencies for the upgrade.

For reference, here are some of the legate-boost dependencies:

dependencies:
build:
common:
- output_types: [conda]
packages:
- cuda-cudart-dev
- libcublas-dev
- openblas
- output_types: [conda, pyproject, requirements]
packages:
- cmake>=3.26.4,!=3.30.0
- &legate legate==25.07.*,>=0.0.0.dev0

[manopapad]

@RAMitchell @seberg could you please kick off some 25.08.00rc builds for Legate-Dataframe and Legate-Boost, for Nihaal to check them against legate 25.08.00.rc?

[seberg]

@RAMitchell can you check out my PR #236 to see what's failing? We need to put that in to get the new packages.
(Since it's been a bit: One test is failing isclose and I have not tried to figure out yet what is causing it, in principle it could be even the span changes, although I don't see an obvious mistake.)

I had fixed that glibc issue that was preventing local runs for us and I am pretty sure that was before the rc came out.

[RAMitchell]

Yes I will take a look today.