Move debugging section of OpenMPI faq to new section in user documentation

Cleanup some text in debugging topics

Fix review comments

Signed-off-by: David Wootton <dwootton@us.ibm.com>
Co-authored-by: Jeff Squyres <jsquyres@cisco.com>

Author: David Wootton

docs/Makefile.am

@@ -35,6 +35,7 @@ IMAGE_SOURCE_FILES = \
         $(srcdir)/installing-open-mpi/required-support-libraries-dependency-graph.png
 RST_SOURCE_FILES   = \
         $(srcdir)/*.rst \
+        $(srcdir)/app-debug/*.rst \
         $(srcdir)/building-apps/*.rst \
         $(srcdir)/developers/*.rst \
         $(srcdir)/faq/*.rst \

docs/app-debug/debug-options.rst

@@ -0,0 +1,61 @@
+Open MPI Runtime Debugging Options
+==================================
+
+Open MPI has a series of MCA parameters for the MPI layer
+itself that are designed to help with debugging.
+These parameters :ref:`can be set <label-running-setting-mca-param-values>`
+in the usual ways.  MPI-level MCA parameters can be
+displayed by invoking the following command:
+
+.. code-block:: sh
+
+   # Use "--level 9" to see all the MCA parameters
+   # (the default is "--level 1"):
+   shell$ ompi_info --param mpi all --level 9
+
+Here is a summary of the debugging parameters for the MPI layer:
+
+* ``mpi_param_check``: If set to true (any positive value), and when
+  Open MPI is compiled with parameter checking enabled (the default),
+  the parameters to each MPI function can be passed through a series
+  of correctness checks.  Problems such as passing illegal values
+  (e.g., NULL or ``MPI_DATATYPE_NULL`` or other "bad" values) will be
+  discovered at run time and an MPI exception will be invoked (the
+  default of which is to print a short message and abort the entire
+  MPI job).  If set to false, these checks are disabled, slightly
+  increasing performance.
+
+* ``mpi_show_handle_leaks``: If set to true (any positive value),
+  Open MPI will display lists of any MPI handles that were not freed before
+  :ref:`MPI_Finalize(3) <mpi_finalize>`  (e.g., communicators,
+  datatypes, requests, etc.)
+
+* ``mpi_no_free_handles``: If set to true (any positive value), do not
+  actually free MPI objects when their corresponding MPI "free"
+  function is invoked (e.g., do not free communicators when
+  :ref:`MPI_Comm_free(3) <mpi_comm_free>`.  This can be helpful in tracking down
+  applications that accidentally continue to use MPI handles after
+  they have been freed.
+
+* ``mpi_show_mca_params``: If set to true (any positive value), show a
+  list of all MCA parameters and their values when MPI is initialized.
+  This can be quite helpful for reproducibility of MPI applications.
+
+* ``mpi_show_mca_params_file``: If set to a non-empty value, and if
+  the value of ``mpi_show_mca_params`` is true, then output the list
+  of MCA parameters to the filename value.  If this parameter is an
+  empty value, the list is sent to ``stderr``.
+
+* ``mpi_abort_delay``: If nonzero, print out an identifying message
+  when :ref:`MPI_Abort(3) <mpi_abort>` is invoked showing the hostname and PID of the
+  process that invoked :ref:`MPI_Abort(3) <mpi_abort>`, and then delay that many seconds
+  before exiting.  A negative value means to delay indefinitely.  This
+  allows a user to manually come in and attach a debugger when an
+  error occurs.  Remember that the default MPI error handler |mdash|
+  ``MPI_ERRORS_ABORT`` |mdash| invokes :ref:`MPI_Abort(3) <mpi_abort>`, so this
+  parameter can be useful to discover problems identified by
+  ``mpi_param_check``.
+
+* ``mpi_abort_print_stack``: If nonzero, print out a stack trace (on
+  supported systems) when :ref:`MPI_Abort(3) <mpi_abort>` is invoked.
+

docs/app-debug/debug-tools.rst

@@ -0,0 +1,19 @@
+Parallel Debugging Tools
+========================
+
+There are two main categories of tools that can aid in
+parallel debugging:
+
+* **Debuggers:** Both serial and parallel debuggers are useful.  Serial
+  debuggers are what most programmers are used to (e.g.,
+  the GNU debugger, ``gdb``), while
+  parallel debuggers can attach to all the individual processes in an
+  MPI job simultaneously, treating the MPI application as a single
+  entity.  This can be an extremely powerful abstraction, allowing the
+  user to control every aspect of the MPI job, manually replicate race
+  conditions, etc.
+
+* **Profilers:** Tools that analyze your usage of MPI and display
+  statistics and meta information about your application's run.  Some
+  tools present the information "live" (as it occurs), while others
+  collect the information and display it in a post mortem analysis.

docs/app-debug/index.rst

@@ -0,0 +1,29 @@
+.. Open MPI Application Debugging
+
+Debugging Open MPI Parallel Applications
+========================================
+
+Debugging a serial applications includes solving problems like
+logic errors, uninitialized variables, storage overlays and timing
+problems.
+
+Debugging a parallel application can be further complicated
+by problems that can include additional race conditions and aysynchronous
+events, as well as understanding execution of multiple application
+processes running simultaneously.
+
+This section of the documentation describes some techniques that can
+be useful for parallel debugging. This section also describes some
+tools that can be useful as well as some Open MPI runtime options
+that can aid debugging.
+
+.. toctree::
+   :maxdepth: 1
+
+   debug-tools
+   debug-options
+   serial-debug
+   lost-output
+   memchecker
+   valgrind
+   mpir-tools

docs/app-debug/lost-output.rst

@@ -0,0 +1,34 @@
+Application Output Lost with Abnormal Termination
+=================================================
+
+There many be many reasons for application output to be lost when
+an application abnormally terminates. The Open MPI Team strongly
+encourages the use of tools (such as debuggers) whenever possible.
+
+One of the reasons, however, may come from inside Open MPI itself.  If
+your application fails due to memory corruption, Open MPI may
+subsequently fail to output an error message before terminating.
+Open MPI attempts to aggregate error
+messages from multiple processes in an attempt to show unique error
+messages only once (vs. once for each MPI process |mdash| which can be
+unwieldy, especially when running large MPI jobs).
+
+However, this aggregation process requires allocating memory in the
+MPI process when it displays the error message.  If the process's
+memory is already corrupted, Open MPI's attempt to allocate memory may
+fail and the process will simply terminate, possibly silently.  When Open
+MPI does not attempt to aggregate error messages, most of its setup
+work is done when the MPI library is initiaized  and no memory is allocated
+during the "print the error" routine.  It therefore almost always successfully
+outputs error messages in real time |mdash| but at the expense that you'll
+potentially see the same error message for *each* MPI process that
+encountered the error.
+
+Hence, the error message aggregation is *usually* a good thing, but
+sometimes it can mask a real error.  You can disable Open MPI's error
+message aggregation with the ``opal_base_help_aggregate`` MCA
+parameter.  For example:
+
+.. code-block:: sh
+
+   shell$ mpirun --mca opal_base_help_aggregate 0 ...

docs/app-debug/memchecker.rst

@@ -0,0 +1,193 @@
+Using Memchecker
+================
+
+The Memchecker functionality in Open MPI provides MPI semantic
+checking for your application (as well as internals of Open MPI), with
+the help of memory checking tools such as the ``memcheck`` component of
+`the Valgrind suite <https://www.valgrind.org/>`_.
+
+/////////////////////////////////////////////////////////////////////////
+
+Types of Errors Detected by Memchecker
+--------------------------------------
+
+Open MPI's Memchecker is based on the ``memcheck`` tool included with
+Valgrind, so it takes all the advantages from it. Firstly, it checks
+all reads and writes of memory, and intercepts calls to
+``malloc(3)``/``free(3)`` and C++'s ``new``/``delete`` operators.
+Most importantly, Memchecker is able to detect
+the user buffer errors in both non-blocking and one-sided
+communications, e.g. reading or writing to buffers of active
+non-blocking receive operations and writing to buffers of active
+non-blocking send operations.
+
+Here are some example problems that Memchecker can detect:
+
+Accessing buffer under control of non-blocking communication:
+
+.. code-block:: c
+
+   int buf;
+   MPI_Irecv(&buf, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, &req);
+   // The following line will produce a memchecker warning
+   buf = 4711;
+   MPI_Wait (&req, &status);
+
+Wrong input parameters, e.g., wrong-sized send buffers:
+
+.. code-block:: c
+
+   char *send_buffer;
+   send_buffer = malloc(5);
+   memset(send_buffer, 0, 5);
+   // The following line will produce a memchecker warning
+   MPI_Send(send_buffer, 10, MPI_CHAR, 1, 0, MPI_COMM_WORLD);
+
+Accessing a window in a one-sided communication:
+
+.. code-block:: c
+
+   MPI_Get(A, 10, MPI_INT, 1, 0, 1, MPI_INT, win);
+   A[0] = 4711;
+   MPI_Win_fence(0, win);
+
+Uninitialized input buffers:
+
+.. code-block:: c
+
+   char *buffer;
+   buffer = malloc(10);
+   // The following line will produce a memchecker warning
+   MPI_Send(buffer, 10, MPI_INT, 1, 0, MPI_COMM_WORLD);
+
+Usage of the uninitialized ``MPI_Status`` field in ``MPI_ERROR``
+structure: (the MPI-1 standard defines the ``MPI ERROR`` field to be
+undefined for single-completion calls such as :ref:`MPI_Wait(3) <mpi_wait>` or
+:ref:`MPI_Test(3) <mpi_test>`, see MPI-1 p. 22):
+
+.. code-block:: c
+
+   MPI_Wait(&request, &status);
+   // The following line will produce a memchecker warning
+   if (status.MPI_ERROR != MPI_SUCCESS)
+       return ERROR;
+
+/////////////////////////////////////////////////////////////////////////
+
+Building Open MPI with Memchecker Support
+-----------------------------------------
+
+To use Memchecker, you need Valgrind 3.2.0 or later, and have an Open
+MPI that was configured with the ``--enable-memchecker`` and
+``--enable-debug`` flags.
+
+.. note:: The Memchecker functionality is off by default, because it
+          incurs a performance penalty.
+
+When ``--enable-memchecker`` is specified, ``configure`` will check
+for a recent-enable valgrind distribution.  If found, Open MPI will
+build Memchecker support.
+
+For example:
+
+.. code-block:: sh
+
+   shell$ ./configure --prefix=/path/to/openmpi --enable-debug \
+       --enable-memchecker --with-valgrind=/path/to/valgrind
+
+You can check that Open MPI was built with Memchecker support by using
+the :ref:`ompi_info(1) <man1-ompi_info>` command.
+
+.. code-block:: sh
+
+   # The exact version numbers shown may be different for your Open
+   # MPI installation
+   shell$ ompi_info | grep memchecker
+   MCA memchecker: valgrind (MCA v1.0, API v1.0, Component v1.3)
+
+If you do not see the "MCA memchecker: valgrind" line, you probably
+did not configure and install Open MPI correctly.
+
+/////////////////////////////////////////////////////////////////////////
+
+Running an Open MPI Application with Memchecker
+-----------------------------------------------
+
+After Open MPI was built and installed with Memchecker support, 
+simply run your application with Valgrind, e.g.:
+
+.. code-block:: sh
+
+   shell$ mpirun -n 2 valgrind ./my_app
+
+If you enabled Memchecker, but you don't want to check the
+application at this time, then just run your application as
+usual. E.g.:
+
+.. code-block:: sh
+
+   shell$ mpirun -n 2 ./my_app
+
+/////////////////////////////////////////////////////////////////////////
+
+Application Performance Impacts Using Memchecker
+------------------------------------------------
+
+The configure option ``--enable-memchecker`` (together with
+``--enable-debug``) *does* cause performance degradation, even if not
+running under Valgrind.  The following explains the mechanism and may
+help in making the decision whether to provide a cluster-wide
+installation with ``--enable-memchecker``.
+
+There are two cases:
+
+#. If run without Valgrind, the Valgrind ClientRequests (assembler
+   instructions added to the normal execution path for checking) do
+   not affect overall MPI performance. Valgrind ClientRequests are
+   explained in detail `in Valgrind's documentation
+   <https://valgrind.org/docs/manual/manual-core-adv.html>`_.
+   In the case of x86-64, ClientRequests boil down to the following
+   four rotate-left (ROL) and one xchange (XCHG) assembler instructions
+   from ``valgrind.h``:
+
+   .. code-block:: c
+
+      #define __SPECIAL_INSTRUCTION_PREAMBLE                      \
+                     "rolq \$3,  %%rdi; rolq \$13, %%rdi\\n\\t"   \
+                     "rolq \$61, %%rdi; rolq \$51, %%rdi\\n\\t"
+
+   and
+
+   .. We do not make the code block below as "c" because the Sphinx C
+      syntax highlighter fails to parse it as C and emits a warning.
+      So we might as well just leave it as a plan verbatim block
+      (i.e., not syntax highlighted).
+
+   .. code-block::
+
+      __asm__ volatile(__SPECIAL_INSTRUCTION_PREAMBLE               \
+                     /* %RDX = client_request ( %RAX ) */           \
+                     "xchgq %%rbx,%%rbx"                            \
+                     : "=d" (_zzq_result)                           \
+                     : "a" (& _zzq_args``0``), "0" (_zzq_default)   \
+                     : "cc", "memory"                               \
+                    );
+
+   for every single ClientRequest.  In the case of not running
+   Valgrind, these ClientRequest instructions do not change the
+   arithmetic outcome (rotating a 64-bit register left by 128-Bits,
+   exchanging a register with itself), except for the carry flag.
+
+   The first request is checking whether we're running under Valgrind.
+   In case we're not running under Valgrind subsequent checks (a.k.a.
+   ClientRequests) are not done.
+
+#. If the application is run under Valgrind, performance is naturally reduced due
+   to the Valgrind JIT and the checking tool employed.
+   For costs and overheads of Valgrind's Memcheck tool on the SPEC 2000 Benchmark,
+   please see the excellent paper
+   `Valgrind: A Framework for Heavyweight Dynamic Binary Instrumentation
+   <https://valgrind.org/docs/valgrind2007.pdf>`_.
+   For an evaluation of various internal implementation alternatives of Shadow Memory, please see
+   `Building Workload Characterization Tools with Valgrind
+   <https://valgrind.org/docs/iiswc2006.pdf>`_.