I am trying to use HDF5.jl with the system-provided libhdf5 (in order to use MPI), following the instructions here: Home · HDF5.jl.
I get an error about ‘failing to find libhdf5’, even though the library is present in the directory pointed to by JULIA_HDF5_PATH - see below (ls at the top shows library exists, then the error message is at the bottom). Any suggestions to track down the problem?
$ ls /usr/lib/x86_64-linux-gnu/hdf5/openmpi/
include libhdf5_cpp.a libhdf5_fortran.so libhdf5_hl_cpp.so libhdf5_hl.so
lib libhdf5_cpp.so libhdf5_hl.a libhdf5hl_fortran.a libhdf5.settings
libhdf5.a libhdf5_fortran.a libhdf5_hl_cpp.a libhdf5hl_fortran.so libhdf5.so
$ julia --project
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.8.2 (2022-09-29)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using MPI
julia> ENV["JULIA_HDF5_PATH"] = "/usr/lib/x86_64-linux-gnu/hdf5/openmpi/"
"/usr/lib/x86_64-linux-gnu/hdf5/openmpi/"
(moment_kinetics) pkg> add HDF5
Updating registry at `~/.julia/registries/General.toml`
Resolving package versions...
Updating `~/physics/moment_kinetics-test/Project.toml`
[f67ccb44] + HDF5 v0.16.13
Updating `~/physics/moment_kinetics-test/Manifest.toml`
[f67ccb44] + HDF5 v0.16.13
(moment_kinetics) pkg> build
Building Conda ─→ `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/6e47d11ea2776bc5627421d59cdcc1296c058071/build.log`
Building IJulia → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/98ab633acb0fe071b671f6c1785c46cd70bb86bd/build.log`
Building PyCall → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/53b8b07b721b77144a0fbbbc2675222ebf40a02d/build.log`
Building HDF5 ──→ `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/b5df7c3cab3a00c33c2e09c6bd23982a75e2fbb2/build.log`
ERROR: Error building `HDF5`:
[ Info: using system HDF5
ERROR: LoadError: libhdf5 could not be found
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:35
[2] top-level scope
@ ~/.julia/packages/HDF5/TcavY/deps/build.jl:49
[3] include(fname::String)
@ Base.MainInclude ./client.jl:476
[4] top-level scope
@ none:5
in expression starting at /home/john/.julia/packages/HDF5/TcavY/deps/build.jl:32
What happens if you try to manually open the library:
using Libdl
l = joinpath(ENV["JULIA_HDF5_PATH"], "libhdf5")
dlopen(lib, RTLD_LAZY)
?
Thank you @stevengj, that looks like a more informative error message (although I don’t know what to do about it at first glance!)
julia> using Libdl
julia> lib = joinpath(ENV["JULIA_HDF5_PATH"], "libhdf5")
"/usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5"
julia> dlopen(lib, RTLD_LAZY)
ERROR: could not load library "/usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5"
/home/john/pkg/julia-1.8.2/bin/../lib/julia/libcurl.so: version `CURL_OPENSSL_4' not found (required by /usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so)
Stacktrace:
[1] dlopen(s::String, flags::UInt32; throw_error::Bool)
@ Base.Libc.Libdl ./libdl.jl:117
[2] dlopen(s::String, flags::UInt32)
@ Base.Libc.Libdl ./libdl.jl:116
[3] top-level scope
@ REPL[12]:1
From the look of the error message, you have entered dependency hell: your openmpi/libhdf5.so is linked to a newer version of libcurl than the one linked to julia, and hence it is refusing to load.
You can check what libcurl version it wants by running ldd/usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so, and then force Julia to use this libcurl (which is hopefully backward compatible) by using LD_PRELOAD when launching Julia:
LD_PRELOAD=/path/to/lib/libcurl.so julia
(In the longer term, hopefully the library versions will get in sync.)
Unfortunately, LD_PRELOAD does not seem to be helping me 
$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libcurl.so julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.8.2 (2022-09-29)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> curllib = "/usr/lib/x86_64-linux-gnu/libcurl.so"
"/usr/lib/x86_64-linux-gnu/libcurl.so"
julia> using Libdl
julia> dlopen(curllib, RTLD_LAZY)
Ptr{Nothing} @0x00007f1e79dc3700
julia> lib = joinpath(ENV["JULIA_HDF5_PATH"], "libhdf5"); dlopen(lib, RTLD_LAZY)
ERROR: could not load library "/usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5"
/home/john/pkg/julia-1.8.2/bin/../lib/julia/libcurl.so: version `CURL_OPENSSL_4' not found (required by /usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so)
Stacktrace:
[1] dlopen(s::String, flags::UInt32; throw_error::Bool)
@ Base.Libc.Libdl ./libdl.jl:117
[2] dlopen(s::String, flags::UInt32)
@ Base.Libc.Libdl ./libdl.jl:116
[3] top-level scope
@ REPL[4]:1
The only think I can think to try now is to compile hdf5 for myself… I could either link it to Julia’s libcurl.so or (fingers crossed) compile without libcurl at all…
Compiling the HDF5 library from source myself seems to have worked - my version does not depend on libcurl at all, so avoids the dependency conflict that seems to be the problem above.
I don’t understand why LD_PRELOAD didn’t work for me - would be interested to know, or get suggestions for further debugging!
I’ve seen that error in other contexts, namely involving the netCDF lib (that links to HDF). I found it too in GMT CI tests when running in ububtu-latest. Lowering the ubuntu version solve (so far) the problem.
@joa-quim I guess what I’m seeing tallies with that - my OS is Linux Mint 21, so Ubuntu 22.04 based.