Skip to content

Commit 8c87d63

Browse files
jsquyresjsquyres
committed
docs: update runpath/rpath docs
Add a bunch of clarifications / explanations of Open MPI's runpath/rpath behavior. Include a link to some preview slides of a paid course about Linux shared libraries and the linker with explicit permission from the author. Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
1 parent 28524e0 commit 8c87d63

File tree

3 files changed

+147
-32
lines changed

3 files changed

+147
-32
lines changed

‎docs/installing-open-mpi/configure-cli-options/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,3 +19,4 @@ below.
1919
mpi
2020
oshmem
2121
misc
22+
rpath-and-runpath

‎docs/installing-open-mpi/configure-cli-options/installation.rst

Lines changed: 82 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,3 @@
1-
.. This file is included by building-open-mpi.rst
2-
31
Installation options
42
^^^^^^^^^^^^^^^^^^^^
53

@@ -32,48 +30,100 @@ be used with ``configure``:
3230
Be sure to read the description of ``--without-memory-manager``,
3331
below; it may have some effect on ``--enable-static``.
3432

35-
* ``--disable-wrapper-rpath``:
36-
By default, the wrapper compilers (e.g., ``mpicc``) will enable
37-
"rpath" support in generated executables on systems that support it.
38-
That is, they will include a file reference to the location of Open
39-
MPI's libraries in the application executable itself. This means
40-
that the user does not have to set ``LD_LIBRARY_PATH`` to find Open
41-
MPI's libraries (e.g., if they are installed in a location that the
42-
run-time linker does not search by default).
43-
44-
On systems that utilize the GNU ``ld`` linker, recent enough versions
45-
will actually utilize "runpath" functionality, not "rpath". There
46-
is an important difference between the two:
47-
48-
#. "rpath": the location of the Open MPI libraries is hard-coded into
49-
the MPI/OpenSHMEM application and cannot be overridden at
50-
run-time.
51-
#. "runpath": the location of the Open MPI libraries is hard-coded into
52-
the MPI/OpenSHMEM application, but can be overridden at run-time
53-
by setting the ``LD_LIBRARY_PATH`` environment variable.
33+
* ``--disable-wrapper-runpath`` / ``--disable-wrapper-rpath``: By
34+
default, the wrapper compilers (e.g., ``mpicc``) will enable
35+
"runpath" or "rpath" support in generated executables on systems
36+
that support it. That is, they will include a filesystem path
37+
reference to the location of Open MPI's libraries in the application
38+
executable itself. This means that the user does not have to set
39+
``LD_LIBRARY_PATH`` to find Open MPI's libraries, which can be
40+
helpful if they are installed in a location that the run-time linker
41+
does not search by default.
42+
43+
.. note:: By default, the wrapper compilers prefer "runpath"
44+
behavior over "rpath" behavior.
45+
46+
* Using ``--disable-wrapper-runpath`` alters this
47+
preference: "runpath" behavior will never be used.
48+
Instead, the wrapper compilers will use "rpath" behavior
49+
if it is supported.
50+
* Using ``--disable-wrapper-rpath`` will disable *both*
51+
"runpath" and "rpath" behavior in the wrapper compilers.
52+
53+
.. important:: Note that the ``--disable-wrapper-runpath`` and
54+
``--disable-wrapper-rpath`` CLI options *only* affect
55+
the flags that the wrapper compilers use when
56+
building MPI/OpenSHMEM applications. These options
57+
do not affect how Open MPI or OpenSHMEM are built (to
58+
include the wrapper compilers themselves).
59+
60+
See the :ref:`Linker "rpath" and "runpath"
61+
functionality
62+
<building-ompi-cli-options-rpath-and-runpath-label>`
63+
section for details on how "rpath" and "runpath"
64+
affect the building and linking of Open MPI itself.
65+
66+
When either of "runpath" or "rpath" behaviors are enabled, the
67+
wrapper compilers will hard-code the filesystem path location of the
68+
Open MPI libraries into the Open MPI/OpenSHMEM application. The
69+
most notable differences between "runpath" and "rpath" behavior are:
70+
71+
* runpath
72+
73+
#. The run-time linker first searches the paths in the
74+
``LD_LIBRARY_PATH`` environment variable for the relevant Open
75+
MPI/OpenSHMEM libraries.
76+
#. If not found there, the run-time linker falls back to checking
77+
the hard-coded location for the relevant Open MPI/OpenSHMEM
78+
libraries.
79+
80+
* rpath
81+
82+
#. The run-time linker first checks the hard-coded location for
83+
the relevant Open MPI/OpenSHMEM libraries.
84+
#. If not found there, the run-time linker falls back to
85+
searching the paths in the ``LD_LIBRARY_PATH`` environment
86+
variable for the relevant Open MPI/OpenSHMEM libraries.
87+
88+
.. warning:: There are other, subtle differences between "runpath"
89+
and "rpath" which are out of scope for this
90+
documentation. You may wish to consult other sources
91+
for more information.
92+
93+
For example, a decent set of explanations can be found
94+
in the slides for a Linux course entitled "Building and
95+
Using Shared Libraries on Linux // `Shared Libraries:
96+
The Dynamic Linker
97+
<https://man7.org/training/download/shlib_dynlinker_slides.pdf>`_".
5498

5599
For example, consider that you install Open MPI vA.B.0 and
56100
compile/link your MPI/OpenSHMEM application against it. Later, you
57101
install Open MPI vA.B.1 to a different installation prefix (e.g.,
58-
``/opt/openmpi/A.B.1`` vs. ``/opt/openmpi/A.B.0``), and you leave the old
59-
installation intact.
102+
``/opt/openmpi/A.B.1`` vs. ``/opt/openmpi/A.B.0``), and you leave
103+
the old installation intact.
104+
105+
In the runpath case, you can set the ``LD_LIBRARY_PATH`` environment
106+
variable to point to the A.B.1 installation, and then your MPI
107+
application will use those libraries, since the runtime will search
108+
the paths in ``LD_LIBRARY_PATH`` first.
60109

61-
In the rpath case, your MPI application will always use the
62-
libraries from your A.B.0 installation. In the runpath case, you
63-
can set the ``LD_LIBRARY_PATH`` environment variable to point to the
64-
A.B.1 installation, and then your MPI application will use those
65-
libraries.
110+
In the rpath case, since the run-time linker searches the
111+
``/opt/openmpi/A.B.0`` location that is hard-coded in your MPI
112+
application first, your application will use the libraries from your
113+
A.B.0 installation (regardless of the value of the
114+
``LD_LIBRARY_PATH`` environment variable).
66115

67116
Note that in both cases, however, if you remove the original A.B.0
68117
installation and set ``LD_LIBRARY_PATH`` to point to the A.B.1
69118
installation, your application will use the A.B.1 libraries.
70119

71-
This rpath/runpath behavior can be disabled via
120+
As noted above, both runpath/rpath behaviors can be disabled via
72121
``--disable-wrapper-rpath``.
73122

74-
If you would like to keep the rpath option, but not enable runpath
75-
a different ``configure`` option is avalabile
76-
``--disable-wrapper-runpath``.
123+
.. note:: You can also :ref:`customize the compiler/linker flags
124+
that are used by the wrapper compilers
125+
<label-customizing-wrapper-compiler>` to build Open
126+
MPI/OpenSHMEM applications.
77127

78128
* ``--enable-dlopen``: Enable Open MPI to load components as
79129
standalone Dynamic Shared Objects (DSOs) at run-time. This option

‎docs/installing-open-mpi/configure-cli-options/rpath-and-runpath.rst

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
.. _building-ompi-cli-options-rpath-and-runpath-label:
2+
3+
Linker "rpath" and "runpath" functionality
4+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
5+
6+
Open MPI and OpenSHMEM |ompi_ver| are both composed of multiple
7+
libraries that depend on each other.
8+
9+
When built from official distribution tarballs (from `the official
10+
Open MPI download site <https://www.open-mpi.org/software/ompi/>`_),
11+
Open MPI and OpenSHMEM are built with the following versions of the
12+
GNU Autotools:
13+
14+
* Autoconf |autoconf_min_version|
15+
* Automake |automake_min_version|
16+
* Libtool |libtool_min_version|
17+
18+
This set of GNU Autotools invokes the ``libtool`` executable to build
19+
Open MPI's libraries and executables with the `-rpath CLI option
20+
<https://www.gnu.org/software/libtool/manual/html_node/Link-mode.html#Link-mode>`_.
21+
22+
The behavior of ``libtool -rpath ...`` is, unfortunately, highly
23+
system-dependent. In conjunction with compiler-, linker-, and other
24+
system-level settings, the end result may be to effect "rpath"
25+
behavior, "runpath" behavior, or possibly even neither behavior.
26+
27+
Although the specific behavior of ``libtool -rpath ..`` is outside the
28+
scope of this documentation, you can run commands such as ``readelf -d
29+
...`` to find out with which behavior your Open MPI was built.
30+
31+
For example:
32+
33+
.. code::
34+
35+
shell$ ./configure --prefix=/opt/openmpi/ ...
36+
...
37+
shell$ make -j 32 all && make install
38+
...
39+
shell$ readelf -d /opt/openmpi/lib/libmpi.so | egrep -i 'rpath|runpath'
40+
0x000000000000001d (RUNPATH) Library runpath: [/opt/openmpi/lib]
41+
42+
The above output indicates that ``libmpi.so`` was built with "runpath"
43+
support, whereas output like this:
44+
45+
.. code::
46+
47+
shell$ readelf -d /opt/openmpi/lib/libmpi.so | egrep -i 'rpath|runpath'
48+
0x000000000000000f (RPATH) Library rpath: [/opt/openmpi/lib]
49+
50+
indicates that ``libmpi.so`` was built with "rpath" support.
51+
52+
.. note:: If you want to utilize additional compiler or linker flags
53+
(such as runpath flags) when building Open MPI/OpenSHMEM,
54+
you can :ref:`specify these flags on the configure command
55+
line <install-configure-compilers-and-flags-label>`.
56+
57+
For example, if invoking ``libtool -rpath ...`` on your
58+
system actually effects "rpath" behavior, and you wish to
59+
effect "runpath" behavior, you could set ``LDFLAGS`` when
60+
invoking ``configure``, like this:
61+
62+
.. code-block:: sh
63+
64+
shell$ ./configure LDFLAGS=--enable-new-dtags ...

0 commit comments

Comments
 (0)