Add section on external libraries (#713)

Fixes #702

Co-authored-by: Michael Schlottke-Lakemper <michael@sloede.com>
Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com>

Author: 3 people

docs/make.jl

@@ -57,6 +57,7 @@ makedocs(
         "index.md",
         "configuration.md",
         "usage.md",
+        "external.md",
         "knownissues.md",
         "Examples" => EXAMPLES,
         "Reference" => [

docs/src/configuration.md

@@ -23,7 +23,7 @@ julia --project -e 'using Pkg; Pkg.add("MPIPreferences")'
     See [Migration from MPI.jl v0.19 or earlier](@ref) for more information on
     how to migrate your configuration from earlier MPI.jl versions.
 
-## Using a system-provided MPI backend
+## [Using a system-provided MPI backend](@id using_system_mpi)
 
 ### Requirements
 
@@ -206,5 +206,3 @@ unknown MPI ABIs is not supported anymore. See also
 Removed without replacement. Automatic generation of a constants file for
 unknown MPI ABIs is not supported anymore. See also
 [#574](https://github.com/JuliaParallel/MPI.jl/issues/574).
-
-

docs/src/external.md

@@ -0,0 +1,62 @@
+# External libraries and packages
+
+Other libraries and packages may also make use of MPI. There are several concerns to ensure things are set up correctly.
+
+## Binary requirements
+
+You need to ensure that external libraries are built correctly. In particular, if you are [using a system-provided MPI backend](@ref using_system_mpi) in Julia, you also need to use the *same* system-provided binary for all packages and external libraries you use.
+
+## Passing MPI handles via `ccall`
+
+When passing MPI.jl handle objects ([`MPI.Comm`](@ref), [`MPI.Info`](@ref), etc) to C/C++ functions [via `ccall`](https://docs.julialang.org/en/v1/manual/calling-c-and-fortran-code/), you should pass the object directly as an argument, and specify the argument type as either the underlying handle type (`MPI.MPI_Comm`, `MPI.MPI_Info`, etc.), or a pointer (`Ptr{MPI.MPI_Comm}`, `Ptr{MPI.MPI_Info}`, etc.). This will internally handle the unwrapping, but ensure that a reference is kept to avoid premature garbage collection.
+
+For example the C function signatures
+```C
+int cfunc1(MPI_Comm comm);
+int cfunc2(MPI_Comm * comm);
+```
+would be called as
+```julia
+ccall((:cfunc1, lib), Cint, (MPI.MPI_Comm,), comm)
+ccall((:cfunc2, lib), Cint, (Ptr{MPI.MPI_Comm},), comm)
+```
+
+## Object finalizers and `MPI.Finalize`
+
+External libraries may allocate their own MPI handles (e.g., create or duplicate MPI communictors), which need to be cleaned up before MPI is finalized. If these are attached to [object finalizers](https://docs.julialang.org/en/v1/base/base/#Base.finalizer), they may not be guaranteed to be called before `MPI.Finalize`, which can result in an error upon program exit. (By default, MPI.jl will install an [`atexit`](https://docs.julialang.org/en/v1/base/base/#Base.atexit) hook that calls `MPI.Finalize` if it hasn't already been invoked.)
+
+There are two typical solutions to this problem:
+
+1. Gate the clean up functions behind an [`MPI.Finalized`](@ref) call, e.g.
+
+   ```julia
+   finalizer(obj) do obj
+       if !MPI.Finalized
+           # call clean up function
+       end
+   end
+   ```
+
+2. Keep track of all such objects, clean them up via [`MPI.add_finalize_hook!`](@ref), e.g.
+
+   ```julia
+   finalizer(obj) do obj
+       # call clean up function
+   end
+   MPI.add_finalize_hook!(() -> finalize(obj))
+   ```
+   A variant of this is to keep track of all such objects, for example, using a [`WeakKeyDict`](https://docs.julialang.org/en/v1/base/collections/#Base.WeakKeyDict), and use a hook to clean them all:
+   ```julia
+   const REFS = WeakKeyDict{ObjType, Nothing}()
+   MPI.add_finalize_hook!() do
+       for obj in keys(REFS)
+           finalize(obj)
+       end
+   end
+
+   # for each object `obj`
+   finalizer(obj) do obj
+       # call clean up function
+   end
+   REFS[obj] = nothing
+   ```

docs/src/reference/comm.md

@@ -12,8 +12,6 @@ each process a unique *rank* (see [`MPI.Comm_rank`](@ref)) taking an integer val
 MPI.Comm
 ```
 
-If you need to pass a Julia `MPI.Comm` object to an external C/C++ library ([via `ccall`](https://docs.julialang.org/en/v1/manual/calling-c-and-fortran-code/)) that expects an `MPI_Comm` argument, you should declare the corresponding `ccall` argument type `MPI.API.MPI_Comm` and the correct conversion will be performed.
-
 ## Constants
 
 ```@docs

docs/src/reference/environment.md

@@ -23,6 +23,8 @@ MPI.Is_thread_main
 MPI.Initialized
 MPI.Finalize
 MPI.Finalized
+MPI.add_init_hook!
+MPI.add_finalize_hook!
 ```
 
 ## Errors

src/environment.jl

@@ -42,6 +42,13 @@ end
 
 
 const mpi_init_hooks = Any[]
+
+"""
+    MPI.add_init_hook!(f)
+
+Register a function `f` that will be called as `f()` when `MPI.Init` is
+called. These are invoked in a first-in, first-out (FIFO) order.
+"""
 add_init_hook!(f) = push!(mpi_init_hooks, f)
 function run_init_hooks()
     while !isempty(mpi_init_hooks)
@@ -52,6 +59,13 @@ function run_init_hooks()
 end
 
 const mpi_finalize_hooks = Any[]
+
+"""
+    MPI.add_finalize_hook!(f)
+
+Register a function `f` that will be called as `f()` when `MPI.Finalizer` is
+called. These are invoked in a last-in, first-out (LIFO) order.
+"""
 add_finalize_hook!(f) = push!(mpi_finalize_hooks, f)
 function run_finalize_hooks()
     while !isempty(mpi_finalize_hooks)