Julia is not using my local OpenMPI installation but asks to install MPICH

New to Julia
1

I am new to Julia but not new to building from source code.
After building a Julia ( which passes its make test ) I am trying to use
MPIPreferences to set use my installed OpenMPI. The following commands
seem to have worked to update Julia:
julia --project -e ‘using Pkg; Pkg.add(“MPIPreferences”)’
julia --project -e ‘using MPIPreferences; MPIPreferences.use_system_binary()’
( although I am getting message
Warning: could not download https://pkg.julialang.org/registries and
Error: curl_easy_getinfo: 43, 48, 48 )
The use system binary()
leaves output
┌ Info: MPI implementation identified
│ libmpi = “libmpi”
│ version_string = “Open MPI v4.1.3, package: Open MPI wadedl@sequoia.iera-lj.com Distribution, ident: 4.1.3, repo rev: v4.1.3, Mar 31, 2022\0”
│ impl = “OpenMPI”
│ version = v"4.1.3"
└ abi = “OpenMPI”
┌ Info: MPIPreferences changed
│ binary = “system”
│ libmpi = “libmpi”
│ abi = “OpenMPI”
└ mpiexec = “mpiexec”
and I seem to get a generated ~.,julia with a toml file with attributes these … …
However, paths to libmpi and mpiexec are /usr/local/lib/libmpi.so and
/usr/local/bin/mpiexecc …

Now when I do the following …
julia> using Pkg

julia> Pkg.instantiate()

julia> using MPI
│ Package MPI not found, but a package named MPI is available from a registry.
│ Install package?
│ (@v1.8) pkg> add MPI
└ (y/n/o) [y]:
I get this query do I want to install. Of course if I say no I’m dead in the water -
no MPI, if I say y
Julia attempts to build an Open MPI - which fails due to errors fetching the
repositories for such …
ERROR: Unable to automatically download/install artifact ‘OpenMPI’ from sources listed in ‘/home/wadedl/.julia/packages/OpenMPI_jll/AbWrR/Artifacts.toml’.
Sources attempted:

Of course I do not WANT to build a NEW Open MPI but to have
julia> using MPI just use the MPI I have installed, as identified in the
MPIPreferences use binary installed.

How do I set this up - what do I need to do to force julia to use my installed
open-mpi as the MPI package - the documentation available I have read over
now multiple times and tried adding Project.toml and LocalProject.toml and
followed guidance to do this from numerous other users but I am stumped.

Would like to get Julia into my projects at work ( I am consulting to USGS )
but have stumbled.

Please help

2

Welcome!

Good, but there’s nothing to compile from source here :slightly_smiling_face:

You’re running the command julia --project, so you’re activating a local environment?

Are you on an air-gapped system? That complicates things a bit.

Are you sure you typed mpiexecc correctly? That’s an unusual name. In any case, if you want to specify a different MPI launcher executable you can pass the mpiexec keyword argument to use_system_binary:

MPIPreferences.use_system_binary(; mpiexec="mpiexecc")

or

MPIPreferences.use_system_binary(; mpiexec="/usr/local/bin/mpiexecc")

if you really want to specify the absolute path of the executable, instead of relying on it being the first one to be found in PATH. Same for libmpi: libmpi is the correct basename to dlopen /usr/local/lib/libmpi.so, if that’s the first libmpi that the dynamical linker would find, but if you want to specify the absolute path to be absolutely clear you can do it:

MPIPreferences.use_system_binary(; library_names=["/usr/local/lib/libmpi.so"])

Note that nothing is building a new OpenMPI, what the package manager is trying to do is to download a Julia package which is a thin wrapper around OpenMPI, so your worry is ill-posed.

But now we’re back to the question above: do you have access to the Internet at all?

Side note, since you’re new here: it’d be extremely helpful if you quoted the code that you copy here with triple backquotes, like this:

```
put
some
code
here
```

which you can achieve also by selecting the code and clicking the </> button in the editing toolbar. You may also want to read Please read: make it easier to help you.

3 Likes
3

Thank you giordano - I typed mpiexecc but of course I meant mpiexec.
My OpenMPI is installed from source - I build into an environment which
may be somewhat unusual - but I build into a path identified for differing
purposes - so my parallel programming languages are all installed in
/usr/local/parallel/programming/library/tools/openmpi-4.1.3
All the proper folders/libraries/executables from bin etc include lib and share
are then symbolically linked into the proper place in /usr/local - i.e.
/usr/local/lib/libmpi.so.40.30.3 → …/parallel/programming/library/tools/openmpi-4.1.3/lib/libmpi.so.40.30.3
/usr/local/lib/libmpi.so → ./libmpi.so.40.30.3
Because each of the folders in /usr/local for different types of things can if need be mounted ( linux direct mounts ) for a different server ( i.e. a server ( or server partition just for programming languages ( i.e.
/usr/local/software/development/tools
is mounted from server_software_development_tools and mounts
/usr/local/software/development/tools/julia-1.8.5
/usr/local/software/development/tools/Python-3.10.4
/usr/local/software/development/tools/GCC_11.2.0
/usr/local/software/development/tools/cmake-3.23.1
and
server_parallel_programming_library_tools mounts
/usr/local/parallel/programming/library/tools/openmpi-4.1.3

Then folders and libraries and executables are symbolically linked back to where they would appear if installed directly in /usr/local/bin, /usr/local/lib, /usr/local/etc, /usr/local/share etc etc. This all dates back to a time when I
system administers and HPC lab where I could mount and unmount different
kinds of things for different departments - all before modules came along …

This is immaterial ( or should be ) to what Julia expects to see because when
MPIpreferences goes out to look it indicates it correctly has found the library
and executable in the right place.

Its just that when the Pkg.add(“MPI”) or using MPI tries to go out and fetch
as you say ) the wrapper around my OpenMPI the build aborts with the messages
Downloaded artifact: OpenMPI
ERROR: Unable to automatically download/install artifact ‘OpenMPI’ from sources listed in ‘/home/wadedl/.julia/packages/OpenMPI_jll/AbWrR/Artifacts.toml’.
Sources attempted:

I tired to edit the .toml file so it would pick up the gzipped tar from a local repository using the file://… … URL - but I’m not sure I put such in the right
plcae because the curl git kept trying to go out to the Web -

If you know or can find out how I can get access to the proper Julia repo site
( I am coming in from an ISP and have turned off my browser firewall but still get the errors ) I would truly appreciate.

I will continue to take to heart what you have said and hope to report success.

I am turning over this work to the sys admin people at the Advanced Research Computing group at USGS and on my system there I may have better luck - I’ve
been doing this for myself - for my own professional development.

THank You!

4

Here is the full stack trace of the errors I am getting when attempting to
‘using MPI’ after setting MPIPreferences as in:
julia --project -e ‘using MPIPreferences; MPIPreferences.use_jll_binary(“OpenMPI_jll”)’
‘’'julia> import Pkg;

julia> using Pkg;

julia> Pkg.instantiate()
Precompiling project…
1 dependency successfully precompiled in 1 seconds. 1 already precompiled.

julia> using MPI
│ Package MPI not found, but a package named MPI is available from a registry.
│ Install package?
│ (@v1.8) pkg> add MPI
└ (y/n/o) [y]: y
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
100 88 100 88 0 0 187 0 --:–:-- --:–:-- --:–:-- 187
┌ Warning: could not download https://pkg.julialang.org/registries
│ exception = Error status 200 while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry /var/tmp/julia/usr/share/julia/stdlib/v1.8/Pkg/src/Registry/Registry.jl:68
┌ Error: curl_easy_getinfo: 43
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
Updating registry at ~/.julia/registries/General
Updating git-repo https://github.com/JuliaRegistries/General.git
Resolving package versions…
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
100 88 100 88 0 0 523 0 --:–:-- --:–:-- --:–:-- 523
┌ Warning: could not download https://pkg.julialang.org/registries
│ exception = Error status 200 while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry /var/tmp/julia/usr/share/julia/stdlib/v1.8/Pkg/src/Registry/Registry.jl:68
┌ Error: curl_easy_getinfo: 43
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0 % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0 % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0 % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0 % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0 % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0 % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0 % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
0 0 0 3003 0 0 4469 0 --:–:-- --:–:-- --:–:-- 15973
┌ Error: curl_easy_getinfo: 43
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
0 0 0 7600 0 0 10755 0 --:–:-- --:–:-- --:–:-- 40860
┌ Error: curl_easy_getinfo: 43
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
0 0 0 8413 0 0 11607 0 --:–:-- --:–:-- --:–:-- 39497
┌ Error: curl_easy_getinfo: 43
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
0 0 0 29171 0 0 39710 0 --:–:-- --:–:-- --:–:-- 124k
┌ Error: curl_easy_getinfo: 43
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
0 0 0 7240 0 0 9819 0 --:–:-- --:–:-- --:–:-- 33518
┌ Error: curl_easy_getinfo: 43
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
0 0 0 8931 0 0 11668 0 --:–:-- --:–:-- --:–:-- 33701
Installed MPItrampoline_jll ─── v5.2.1+0
0 0 0 27477 0 0 34587 0 --:–:-- --:–:-- --:–:-- 99k
┌ Error: curl_easy_getinfo: 43
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
┌ Error: curl_easy_getinfo: 43
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
0 0 0 119k 0 0 150k 0 --:–:-- --:–:-- --:–:-- 940k
┌ Error: curl_easy_getinfo: 43
/utils.jl:29
Cloning [9237b28f-5490-5468-be7b-bb81f5f5e6cf] MicrosoftMPI_jll from GitHub - JuliaBinaryWrappers/MicrosoftMPI_jll.jl
Installed MicrosoftMPI_jll ──── v10.1.3+2
Cloning [ae029012-a4dd-5104-9daa-d747884805df] Requires from GitHub - JuliaPackaging/Requires.jl: Lazy code loading for Julia
Installed Requires ──────────── v1.3.0
Cloning [fe0851c0-eecd-5654-98d4-656369965a5c] OpenMPI_jll from GitHub - JuliaBinaryWrappers/OpenMPI_jll.jl
Installed OpenMPI_jll ───────── v4.1.5+0
hub.com/JuliaPackaging/JLLWrappers.jl.git
Installed JLLWrappers ───────── v1.4.1
Cloning [7cb0a576-ebde-5e09-9194-50597f1243b4] MPICH_jll from GitHub - JuliaBinaryWrappers/MPICH_jll.jl
Installed MPICH_jll ─────────── v4.1.1+0
Cloning [ffbed154-4ef7-542d-bbb7-c09d3a79fcae] DocStringExtensions from GitHub - JuliaDocs/DocStringExtensions.jl: Extensions for Julia's docsystem.
Installed DocStringExtensions ─ v0.9.3
Cloning [da04e1cc-30fd-572f-bb4f-1f8673147195] MPI from GitHub - JuliaParallel/MPI.jl: MPI wrappers for Julia
Installed MPI ───────────────── v0.20.8
Downloading artifact: OpenMPI
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
100 4812k 100 4812k 0 0 5443k 0 --:–:-- --:–:-- --:–:-- 7915k
┌ Error: curl_easy_getinfo: 43
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl Downloaded artifact: OpenMPI
Downloading artifact: OpenMPI
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:29
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
100 4809k 100 4809k 0 0 4202k 0 0:00:01 0:00:01 --:–:-- 7949k
┌ Error: curl_easy_getinfo: 43
└ @ Downloads.Curl /var/tmp/julia/usr/share/julia/stdlib/v1.8/Downloads/src/Curl Downloaded artifact: OpenMPI
ERROR: Unable to automatically download/install artifact ‘OpenMPI’ from sources listed in ‘/home/wadedl/.julia/packages/OpenMPI_jll/AbWrR/Artifacts.toml’.
Sources attempted:

Stacktrace:
[1] error(s::String)
@ Base ./error.jl:35
[2] ensure_artifact_installed(name::String, meta::Dict{String, Any}, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, verbose::Bool, quiet_download::Bool, io::Base.TTY)
@ Pkg.Artifacts /var/tmp/julia/usr/share/julia/stdlib/v1.8/Pkg/src/Artifacts.jl:468
[3] download_artifacts(env::Pkg.Types.EnvCache; platform::Base.BinaryPlatforms.Platform, julia_version::VersionNumber, verbose::Bool, io::Base.TTY)
@ Pkg.Operations /var/tmp/julia/usr/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:665
[4] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
@ Pkg.Operations /var/tmp/julia/usr/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1282
[5] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
@ Pkg.API /var/tmp/julia/usr/share/julia/stdlib/v1.8/Pkg/src/API.jl:275
[6] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Pkg.API /var/tmp/julia/usr/share/julia/stdlib/v1.8/Pkg/src/API.jl:156
[7] add(pkgs::Vector{Pkg.Types.PackageSpec})
@ Pkg.API /var/tmp/julia/usr/share/julia/stdlib/v1.8/Pkg/src/API.jl:145
[8] #add#27
@ /var/tmp/julia/usr/share/julia/stdlib/v1.8/Pkg/src/API.jl:144 [inlined]
[9] add
@ /var/tmp/julia/usr/share/julia/stdlib/v1.8/Pkg/src/API.jl:144 [inlined]
[10] try_prompt_pkg_add(pkgs::Vector{Symbol})
@ Pkg.REPLMode /var/tmp/julia/usr/share/julia/stdlib/v1.8/Pkg/src/REPLMode/REPLMode.jl:717
[11] #invokelatest#2
@ ./essentials.jl:729 [inlined]
[12] invokelatest
@ ./essentials.jl:726 [inlined]
[13] check_for_missing_packages_and_run_hooks(ast::Any)
@ REPL /var/tmp/julia/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:176
[14] eval_user_input(ast::Any, backend::REPL.REPLBackend)
@ REPL /var/tmp/julia/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:146
[15] repl_backend_loop(backend::REPL.REPLBackend)
@ REPL /var/tmp/julia/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:247
[16] start_repl_backend(backend::REPL.REPLBackend, consumer::Any)
@ REPL /var/tmp/julia/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:232
[17] run_repl(repl::REPL.AbstractREPL, consumer::Any; backend_on_current_task::Bool)
@ REPL /var/tmp/julia/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:369
[18] run_repl(repl::REPL.AbstractREPL, consumer::Any)
@ REPL /var/tmp/julia/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:355
[19] (::Base.var"#967#969"{Bool, Bool, Bool})(REPL::Module)
@ Base ./client.jl:419
[20] #invokelatest#2
@ ./essentials.jl:729 [inlined]
[21] invokelatest
@ ./essentials.jl:726 [inlined]
[22] run_main_repl(interactive::Bool, quiet::Bool, banner::Bool, history_file::Bool, color_set::Bool)
@ Base ./client.jl:404
[23] exec_options(opts::Base.JLOptions)
@ Base ./client.jl:318
[24] _start()
@ Base ./client.jl:522

julia> f9744710560ba3ddc00cd9df62ac7dfcd18c8649

[wadedl@sequoia 3]$
‘’’
I can myself download the gzipped wrapper files from github. If I put them somewhere, how would I tell the
/home/wadedl/.julia/packages/OpenMPI_jll/AbWrR/Artifacts.toml
file where to find them? ( i.e URL is http://file: … … … )
Thank you for your attention to this.