HDF5 cannot compile in M1 pro mac

wzy · November 7, 2021, 2:12pm

The julia version is v1.7.0 rc-2, installed by official dmg.
OS: OSX Monterey

Hope to learn about how to resolve this problem, thanks!

The log:

julia> using HDF5
[ Info: Precompiling HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f]
ERROR: LoadError: UndefVarError: libhdf5 not defined
Stacktrace:
[1] top-level scope
@ ~/.julia/packages/HDF5/pIJra/src/api_types.jl:119
[2] include(mod::Module, _path::String)
@ Base ./Base.jl:420
[3] include(x::String)
@ HDF5 ~/.julia/packages/HDF5/pIJra/src/HDF5.jl:1
[4] top-level scope
@ ~/.julia/packages/HDF5/pIJra/src/HDF5.jl:51
[5] include
@ ./Base.jl:420 [inlined]
[6] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
@ Base ./loading.jl:1318
[7] top-level scope
@ none:1
[8] eval
@ ./boot.jl:373 [inlined]
[9] eval(x::Expr)
@ Base.MainInclude ./client.jl:453
[10] top-level scope
@ none:1
in expression starting at /Users/John/.julia/packages/HDF5/pIJra/src/api_types.jl:119
in expression starting at /Users/John/.julia/packages/HDF5/pIJra/src/HDF5.jl:1
ERROR: Failed to precompile HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f] to /Users/John/.julia/compiled/v1.7/HDF5/jl_Ku0anm.
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:33
[2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
@ Base ./loading.jl:1466
[3] compilecache(pkg::Base.PkgId, path::String)
@ Base ./loading.jl:1410
[4] _require(pkg::Base.PkgId)
@ Base ./loading.jl:1120
[5] require(uuidkey::Base.PkgId)
@ Base ./loading.jl:1013
[6] require(into::Module, mod::Symbol)
@ Base ./loading.jl:997

kylebeggs · November 24, 2021, 7:25pm

I have the same issue. I even try directly giving the path to libhdf5 but it still doesn’t work.

Edit - Got it to work. I removed the .julia folder in your home directory completely, upgraded to 1.7.0 rc-3, and in my terminal exported the path to libhdf5 like this export JULIA_HDF5_PATH=/opt/homebrew/Cellar/hdf5/1.12.1 which I installed with homebrew, obviously by the path. Then started Julia and ran these commands using Pkg; Pkd.add("HDF5"); Pkg.build("HDF5").

wzy · December 11, 2021, 11:25pm

It also worked for me. Many thanks for the solution!

Xavier_Gonzalez · May 6, 2022, 5:09pm

Thank you so much for this solution. I’m trying to follow these steps. I have julia version 1.7.2, which is what I think is recommended for M1 chip.

I used homebrew to install hdf5 as you recommend, and added to the path.

I then ran using Pkg; Pkg.add("HDF5") without any problem.

However, when I ran Pkg.build("HDF5") I got the following error:

Error building HDF5:

[ Info: using system HDF5

ERROR: LoadError: libhdf5 could not be found

Do you know what might be going wrong?

kylebeggs · May 6, 2022, 5:36pm

It looks like it’s not using the home brew hdf5. There is an environment variable within julia (not your OS) called JULIA_LIBHDF5 or something similar that you need to set to point to the brew hdf5 install. I am mobile right now but respond if you can’t get it and I’ll be able to check exactly what my settings are later tonight.

Xavier_Gonzalez · May 9, 2022, 8:19pm

Thank you so much! I really appreciate your saying this. yes, if you could check your settings I would be super grateful. I am trying to follow this website, which talks about environment variables in Julia.

Environment Variables · The Julia Language.

It talks about editing the file ~/.julia/config/startup.jl

But when I go into my .julia folder, my only options are artifacts, conda, logs, pluto_notebooks, registries, compiled, environments, packages, prefs, and scratchspaces

PetrKryslUCSD · May 9, 2022, 8:41pm

startup.jl is optional. If it isn’t there, create one.

Xavier_Gonzalez · May 9, 2022, 9:53pm

right, but there is also not even a config folder in my settings

mkitti · May 9, 2022, 9:54pm

@giordano Shouldn’t there be another way of doing this with HDF5_jll.jl and a Preferences.jl override?

I’m also confused because I thought we have

github.com

JuliaBinaryWrappers/HDF5_jll.jl/blob/main/src/wrappers/aarch64-apple-darwin.jl

# Autogenerated wrapper script for HDF5_jll for aarch64-apple-darwin
export libhdf5, libhdf5_hl

using Zlib_jll
using OpenSSL_jll
using LibCURL_jll
JLLWrappers.@generate_wrapper_header("HDF5")
JLLWrappers.@declare_library_product(libhdf5, "@rpath/libhdf5.200.dylib")
JLLWrappers.@declare_library_product(libhdf5_hl, "@rpath/libhdf5_hl.200.dylib")
function __init__()
    JLLWrappers.@generate_init_header(Zlib_jll, OpenSSL_jll, LibCURL_jll)
    JLLWrappers.@init_library_product(
        libhdf5,
        "lib/libhdf5.200.dylib",
        RTLD_LAZY | RTLD_DEEPBIND,
    )

    JLLWrappers.@init_library_product(
        libhdf5_hl,
        "lib/libhdf5_hl.200.dylib",

This file has been truncated. show original

giordano · May 9, 2022, 10:01pm

Depends on the version of HDF5_jll.jl they’re using. May not be the last one. You can see it with

]status -m HDF5_jll

kylebeggs · May 10, 2022, 12:01pm

@mkitti you still need to point Julia to the brew install of hdf5. @Xavier_Gonzalez Just put in your .zshrc or .bashrc or whatever shell you use export JULIA_HDF5_PATH=/opt/homebrew/Cellar/hdf5/1.12.1_1 make sure to match the version number at the end. you can check your version number by just doing ls -l /opt/homebrew/Cellar/hdf5/. also start a new shell and then a new Julia instance when you go to test. then do add HDF5, build HDF5 etc

mkitti · May 10, 2022, 1:54pm

@kylebeggs I’m uncertain why homebrew is still needed. It could certainly still be useful, but I’m not sure why it is necessary as of January 10th, 2022.

As I posted above, HDF5_jll now appears to provide support aarch64-apple-darwin and giordano is not objecting to that assertion.

kylebeggs · May 10, 2022, 3:38pm

Oh I see, yes it’s possible it’s not needed anymore. I was just saying what worked for me (which was before Jan 2022)