Skip to content

Commit 340f8f0

Browse files
committed
docs: Add info about building accelerators as DSOs
Add information about building the accelerator-related components as DSOs, and an explanation of why pacakgers might want to do so. We need these docs for v5.0.0; if some of these specifics change later in the v5.0.x series (e.g., if we starting building these components as DSOs by default), this documentation can be updated to match. Signed-off-by: Jeff Squyres <jeff@squyres.com>
1 parent 59afe11 commit 340f8f0

File tree

1 file changed

+49
-16
lines changed

1 file changed

+49
-16
lines changed

docs/installing-open-mpi/packagers.rst

Lines changed: 49 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -198,22 +198,9 @@ using ``--enable-mca-dso`` to selectively build some components as
198198
DSOs and leave the others included in their respective Open MPI
199199
libraries.
200200

201-
.. code:: sh
202-
203-
# Build all the "accelerator" components as DSOs (all other
204-
# components will default to being built in their respective
205-
# libraries)
206-
shell$ ./configure --enable-mca-dso=accelerator ...
207-
208-
This allows packaging ``$libdir`` as part of the "main" Open MPI
209-
binary package, but then packaging
210-
``$libdir/openmpi/mca_accelerator_*.so`` as sub-packages. These
211-
sub-packages may inherit dependencies on the CUDA and/or ROCM
212-
packages, for example. User can always install the "main" Open MPI
213-
binary package, and can install the additional "accelerator" Open MPI
214-
binary sub-package if they actually have accelerator hardware
215-
installed (which will cause the installation of additional
216-
dependencies).
201+
:ref:`See the section on building accelerator support
202+
<label-install-packagers-building-accelerator-support-as-dsos>` for a
203+
practical example where this can be useful.
217204

218205
.. _label-install-packagers-gnu-libtool-dependency-flattening:
219206

@@ -283,3 +270,49 @@ these flattened dependencies, use either of the following mechanisms:
283270
utilizes compiler and linker flags if they are
284271
*needed*. All other flags should be the user's /
285272
packager's choice.
273+
274+
.. _label-install-packagers-building-accelerator-support-as-dsos:
275+
276+
Building accelerator support as DSOs
277+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
278+
279+
If you are building a package that includes support for one or more
280+
accelerators, it may be desirable to build accelerator-related
281+
components as DSOs (see the :ref:`static or DSO?
282+
<label-install-packagers-dso-or-not>` section for details).
283+
284+
.. admonition:: Rationale
285+
:class: tip
286+
287+
Accelerator hardware is expensive, and may only be present on some
288+
compute nodes in an HPC cluster. Specifically: there may not be
289+
any accelerator hardware on "head" or compile nodes in an HPC
290+
cluster. As such, invoking Open MPI commands on a "head" node with
291+
an MPI that was built with static accelerator support but no
292+
accelerator hardware may fail to launch because of run-time linker
293+
issues (because the accelerator hardware support libraries are
294+
likely not present).
295+
296+
Building Open MPI's accelerator-related components as DSOs allows
297+
Open MPI to *try* opening the accelerator components, but proceed
298+
if those DSOs fail to open due to the lack of support libraries.
299+
300+
Use the ``--enable-mca-dso`` command line parameter to Open MPI's
301+
``configure`` command can allow packagers to build all
302+
accelerator-related components as DSO. For example:
303+
304+
.. code:: sh
305+
306+
# Build all the accelerator-related components as DSOs (all other
307+
# components will default to being built in their respective
308+
# libraries)
309+
shell$ ./configure --enable-mca-dso=btl-smcuda,rcache-rgpusm,rcache-gpusm,accelerator
310+
311+
Per the example above, this allows packaging ``$libdir`` as part of
312+
the "main" Open MPI binary package, but then packaging
313+
``$libdir/openmpi/mca_accelerator_*.so`` and the other named
314+
components as sub-packages. These sub-packages may inherit
315+
dependencies on the CUDA and/or ROCM packages, for example. The
316+
"main" package can be installed on all nodes, and the
317+
accelerator-specific subpackage can be installed on only the nodes
318+
with accelerator hardware and support libraries.

0 commit comments

Comments
 (0)