MPI library mismatch erro HDF5, Netcdf with Oceananigans

General Usage
, , , ,
1

Dear all,

Im having strong issues with a specific package in Julia while trying to run it in MPI. Initially I define the following libraries:

export LD_LIBRARY_PATH=/sw/spack-levante/netcdf-c-4.8.1-7dq6g2/lib/:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/sw/spack-levante/hdf5-1.12.1-jmeuy3/lib/:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/sw/spack-levante/openmpi-4.1.5-godjja/lib/:$LD_LIBRARY_PATH
export JULIA_HDF5_PATH=/sw/spack-levante/hdf5-1.12.1-jmeuy3/lib/

Im not sure if all of them are necessary, but so it was defined. After these definitions i used MPIPreferences to set up the following in Julia:

using MPIPreferences
MPIPreferences.use_system_binary(mpiexec="srun"; library_names="/sw/spack-levante/openmpi-4.1.5-godjja/lib/libmpi.so")

This configuration runs fine if the only library im using is the MPI. However, if I import Oceananigans, which contains many dependencies, including JLD2, NCDatasets, etc, then the precompilation of such library lead to a chain of errors. Namely:

  3 dependencies successfully precompiled in 5 seconds. 23 already precompiled.
Precompiling project...
  ✗ HDF5_jll
  ✗ NetCDF_jll
  ✗ NCDatasets
  ✗ Oceananigans
  ✗ Oceananigans → OceananigansMakieExt
  315 dependencies successfully precompiled in 199 seconds. 41 already precompiled.
  5 dependencies errored.
  For a report of the errors see `julia> err`. To retry use `pkg> precompile`

julia> err
PkgPrecompileError: The following 3 direct dependencies failed to precompile:

OceananigansMakieExt 

Failed to precompile OceananigansMakieExt [8b7e02c2-18e1-5ade-af7b-cfb5875075c8] to "/home/m/m300992/.julia/compiled/v1.11/OceananigansMakieExt/jl_spGCGf".
ERROR: LoadError: InitError: could not load library "/home/m/m300992/.julia/artifacts/695bf0397a849eab58cd21bf28f8c4279cfa42ad/lib/libmpi.so"
/home/m/m300992/.julia/artifacts/695bf0397a849eab58cd21bf28f8c4279cfa42ad/lib/libopen-pal.so.80: undefined symbol: pmix_framework_names
Stacktrace:

[...]

during initialization of module OpenMPI_jll
in expression starting at /home/m/m300992/.julia/packages/HDF5_jll/NQMjC/src/wrappers/x86_64-linux-gnu-libgfortran5-cxx11-mpi+openmpi.jl:9
in expression starting at /home/m/m300992/.julia/packages/HDF5_jll/NQMjC/src/HDF5_jll.jl:2
in expression starting at stdin:6
ERROR: LoadError: Failed to precompile HDF5_jll [0234f1f7-429e-5d53-9886-15a909be8d59] to "/home/m/m300992/.julia/compiled/v1.11/HDF5_jll/jl_VuLzHv".
Stacktrace:

[...]

in expression starting at /home/m/m300992/.julia/packages/NetCDF_jll/f16pp/src/wrappers/x86_64-linux-gnu-mpi+openmpi.jl:6
in expression starting at /home/m/m300992/.julia/packages/NetCDF_jll/f16pp/src/NetCDF_jll.jl:2
in expression starting at stdin:6
ERROR: LoadError: Failed to precompile NetCDF_jll [7243133f-43d8-5620-bbf4-c2c921802cf3] to "/home/m/m300992/.julia/compiled/v1.11/NetCDF_jll/jl_5gNMYp".
Stacktrace:

[...]

in expression starting at /home/m/m300992/.julia/packages/NCDatasets/xJGTj/src/NCDatasets.jl:1
in expression starting at stdin:6
ERROR: LoadError: Failed to precompile NCDatasets [85f8d34a-cbdd-5861-8df4-14fed0d494ab] to "/home/m/m300992/.julia/compiled/v1.11/NCDatasets/jl_Z1FxmC".
Stacktrace:

[...]

in expression starting at /work/mh0033/m300992/cci_euler/Oceananigans.jl/src/OutputWriters/netcdf_output_writer.jl:1
in expression starting at /work/mh0033/m300992/cci_euler/Oceananigans.jl/src/OutputWriters/OutputWriters.jl:1
in expression starting at /work/mh0033/m300992/cci_euler/Oceananigans.jl/src/Oceananigans.jl:1
in expression starting at stdin:6
ERROR: LoadError: Failed to precompile Oceananigans [9e8cae18-63c1-5223-a75c-80ca9d6e9a09] to "/home/m/m300992/.julia/compiled/v1.11/Oceananigans/jl_08PDn5".
Stacktrace:

[...]

in expression starting at /work/mh0033/m300992/cci_euler/Oceananigans.jl/ext/OceananigansMakieExt.jl:1
in expression starting at stdin:6
NCDatasets 

Failed to precompile NCDatasets [85f8d34a-cbdd-5861-8df4-14fed0d494ab] to "/home/m/m300992/.julia/compiled/v1.11/NCDatasets/jl_T2tpqM".
ERROR: LoadError: InitError: could not load library "/home/m/m300992/.julia/artifacts/695bf0397a849eab58cd21bf28f8c4279cfa42ad/lib/libmpi.so"
/home/m/m300992/.julia/artifacts/695bf0397a849eab58cd21bf28f8c4279cfa42ad/lib/libopen-pal.so.80: undefined symbol: pmix_framework_names
Stacktrace:

[...]

during initialization of module OpenMPI_jll
in expression starting at /home/m/m300992/.julia/packages/HDF5_jll/NQMjC/src/wrappers/x86_64-linux-gnu-libgfortran5-cxx11-mpi+openmpi.jl:9
in expression starting at /home/m/m300992/.julia/packages/HDF5_jll/NQMjC/src/HDF5_jll.jl:2
in expression starting at stdin:6
ERROR: LoadError: Failed to precompile HDF5_jll [0234f1f7-429e-5d53-9886-15a909be8d59] to "/home/m/m300992/.julia/compiled/v1.11/HDF5_jll/jl_6OxoUC".
Stacktrace:

[...]

in expression starting at /home/m/m300992/.julia/packages/NetCDF_jll/f16pp/src/wrappers/x86_64-linux-gnu-mpi+openmpi.jl:6
in expression starting at /home/m/m300992/.julia/packages/NetCDF_jll/f16pp/src/NetCDF_jll.jl:2
in expression starting at stdin:6
ERROR: LoadError: Failed to precompile NetCDF_jll [7243133f-43d8-5620-bbf4-c2c921802cf3] to "/home/m/m300992/.julia/compiled/v1.11/NetCDF_jll/jl_JgVqvL".
Stacktrace:

[...]

in expression starting at /home/m/m300992/.julia/packages/NCDatasets/xJGTj/src/NCDatasets.jl:1
in expression starting at stdin:6

Oceananigans

Ive tried the help from the vendors of my institute HPC, but they couldn’t help. So, as a last resort, im coming here for help, as i exausted all my knowledge from these hardware specific libraries and all my motivation to keep digging into this problem. I already appreciate everyones attention.

Best,

Fabricio

2

See