Unable to locate `LocalPreferences.toml` outside of MPI.jl

[luraess]

I observe a strange behaviour using MPI.jl#master (0.20-dev) while running the MPI.use_system_binary() in a project outside of the MPI.jl project. Namely, the system MPI implementation seems to be correctly selected but the LocalPreferences.toml file that should be generated in the project is not there (only Project.toml and Manifest.toml are present). Do I miss something?

Reproducer:

[lraess@ault dev]$ mkdir tmp_test
[lraess@ault dev]$ cd tmp_test/
[lraess@ault dev]$ julia --project
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.2 (2022-02-06)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia>

(@v1.7) pkg> activate .
  Activating new project at `/scratch/lraess/dev/tmp_test`

(tmp_test) pkg> add MPI#master

(tmp_test) pkg> st
      Status `/scratch/lraess/dev/tmp_test/Project.toml`
  [da04e1cc] MPI v0.20.0-dev `https://github.com/JuliaParallel/MPI.jl.git#master`

julia> using MPI

julia> MPI.use_system_binary()
┌ Info: MPI implementation
│   libmpi = "libmpi"
│   version_string = "Open MPI v4.0.6rc4, package: Open MPI root@ault08.cscs.ch Distribution, ident: 4.0.6rc4, repo rev: v4.0.6-33-gd71f8c2ba8, Unreleased developer copy\0"
│   impl = "OpenMPI"
│   version = v"4.0.6-rc4"
└   abi = "OpenMPI"
┌ Warning: The underlying MPI implementation has changed. You will need to restart Julia for this change to take effect
│   libmpi = "libmpi"
│   abi = "OpenMPI"
│   mpiexec = "mpiexec"
└ @ MPIPreferences ~/.julia/packages/MPIPreferences/uArzO/src/MPIPreferences.jl:119

shell> ls
Manifest.toml  Project.toml

shell> cat Project.toml
[deps]
MPI = "da04e1cc-30fd-572f-bb4f-1f8673147195"

julia> exit()

[lraess@ault dev]$ julia --project
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.2 (2022-02-06)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using MPI

julia> MPI.MPI_LIBRARY_VERSION_STRING
"Open MPI v4.0.6rc4, package: Open MPI root@ault08.cscs.ch Distribution, ident: 4.0.6rc4, repo rev: v4.0.6-33-gd71f8c2ba8, Unreleased developer copy\0"

[vchuravy]

You must listen MPIPreferences in the [extras] section of you Project.toml

You can copy LocalPreferences.toml to a different project folder, but you must list MPIPreferences in the [extras] section of the Project.toml for the settings to take effect.

https://juliaparallel.org/MPI.jl/latest/configuration/#Configuration-2

[luraess]

Thanks for pointing to the latest doc @vchuravy. To me it is currently unclear what behaviour to expect. You're suggesting one should manually add MPIPreferences to an [extra] section in the project-local Project.toml. From the doc, I however understand that (with exception of Julia<1.6.5 and 1.7.1) the LocalPreferences.toml file would be created "adjacent to the current Project.toml":

To use the system MPI library, run MPI.use_system_binary(). This will attempt to locate and to identify any available MPI implementation, and create a file called LocalPreferences.toml adjacent to the current Project.toml

Also, using MPI.use_system_binary(export_prefs=true) does not seem to populate the project-local Project.toml with any MPI related prefs.

[vchuravy]

Ah I see I missed that you did indeed ran the command in your Project. Will investigate

[luraess]

Note that indeed adding manually MPIPreferences to [extra] fixes the issue but may not be the intended action.

[vchuravy]

~/builds/julia-1.7.2/bin/julia --project=repr
(repr) pkg> add MPI#master
 ...
  [da04e1cc] ~ MPI v0.19.2 ⇒ v0.20.0-dev `https://github.com/JuliaParallel/MPI.jl.git#master`
  [3da0fdf6] + MPIPreferences v0.1.1
...

julia> using MPI

julia> MPI.use_system_binary()
┌ Info: MPI implementation
│   libmpi = "libmpi"
│   version_string = "Open MPI v4.1.2, package: Open MPI builduser@arojas Distribution, ident: 4.1.2, repo rev: v4.1.2, Nov 24, 2021\0"
│   impl = "OpenMPI"
│   version = v"4.1.2"
└   abi = "OpenMPI"
┌ Warning: The underlying MPI implementation has changed. You will need to restart Julia for this change to take effect
│   libmpi = "libmpi"
│   abi = "OpenMPI"
│   mpiexec = "mpiexec"
└ @ MPIPreferences ~/.julia/packages/MPIPreferences/uArzO/src/MPIPreferences.jl:119

shell> ls repr/
LocalPreferences.toml  Manifest.toml  Project.toml

shell> cat repr/Project.toml
[deps]
MPI = "da04e1cc-30fd-572f-bb4f-1f8673147195"

[extras]
MPIPreferences = "3da0fdf6-3ccc-4f1b-acd9-58baa6c99267"

shell> cat repr/LocalPreferences.toml
[MPIPreferences]
abi = "OpenMPI"
binary = "system"
libmpi = "libmpi"
mpiexec = "mpiexec"

[vchuravy]

oO

@staticfloat is this intended:

➜  ~ ls mpiprefs
LocalPreferences.toml  Project.toml

➜  ~ ls repr
Manifest.toml  Project.toml

cat mpiprefs/LocalPreferences.toml
[MPIPreferences]
abi = "OpenMPI"
binary = "system"
libmpi = "libmpi"
mpiexec = "mpiexec"

JULIA_LOAD_PATH=":$HOME/mpiprefs" ~/builds/julia-1.7.2/bin/julia --project=repr

julia> MPI.MPIPreferences.abi
"OpenMPI"

julia> MPI.use_jll_binary()

shell> ls repr
Manifest.toml  Project.toml

shell> cat mpiprefs/LocalPreferences.toml
[MPIPreferences]
__clear__ = ["libmpi", "abi", "mpiexec"]
binary = "MPICH_jll"

E.g. setting the preference is not done in the current Project, but a Project up in the load path?

[vchuravy]

The issue of course is:

https://github.com/JuliaPackaging/Preferences.jl/blob/80b550904ae5b5615ec4168a2135f87c397a92b8/src/Preferences.jl#L198

MPIPreferences = Base.UUID("3da0fdf6-3ccc-4f1b-acd9-58baa6c99267")

julia> Preferences.find_first_project_with_uuid(MPIPreferences)
("/home/vchuravy/mpiprefs/Project.toml", "MPIPreferences")

shell> cat mpiprefs/Project.toml
[extras]
MPIPreferences = "3da0fdf6-3ccc-4f1b-acd9-58baa6c99267"

[luraess]

Ah ha

julia> MPIPreferences = Base.UUID("3da0fdf6-3ccc-4f1b-acd9-58baa6c99267")
UUID("3da0fdf6-3ccc-4f1b-acd9-58baa6c99267")

julia> Preferences.find_first_project_with_uuid(MPIPreferences)
("/users/lraess/.julia/environments/v1.7/Project.toml", "MPIPreferences")

shell> cat /users/lraess/.julia/environments/v1.7/Project.toml
[deps]
AMDGPU = "21141c5a-9bdb-4563-92ae-f87d6854732e"
CUDA = "052768ef-5323-5732-b1bb-66c8b64840ba"
MPI = "da04e1cc-30fd-572f-bb4f-1f8673147195"
MPIPreferences = "3da0fdf6-3ccc-4f1b-acd9-58baa6c99267"
Revise = "295af30f-e4ad-537b-8983-00126c2a3abe"

[extras]
CPUSummary = "2a0fbf3d-bb9c-48f3-b0a9-814d99fd7ab9"

[preferences.MPIPreferences]
abi = "OpenMPI"
binary = "system"
libmpi = "libmpi"
mpiexec = "mpiexec"