libnetcdf not defined on M1

On a M1 Macbook I get errors trying to use NetCDF from Julia. Any idea which libnetcdf it is looking for? Isn’t NetCDF in Julia self-sufficient? does not mention any external system package dependencies.

Thank you.

[~]᳅ julia
_ _ ()_ | Documentation:
() | () () |
_ _ | | __ _ | Type “?” for help, “]?” for Pkg help.
| | | | | | |/ ` | |
| | |
| | | | (
| | | Version 1.7.0 (2021-11-30)
/ |_|||_’_| | Official release
|__/ |

(@v1.7) pkg> add NetCDF
Updating registry at ~/.julia/registries/General.toml
Resolving package versions…
Updating ~/.julia/environments/v1.7/Project.toml
[30363a11] + NetCDF v0.11.4
Updating ~/.julia/environments/v1.7/Manifest.toml
[3c3547ce] + DiskArrays v0.3.0
[59287772] + Formatting v0.4.2
[692b3bcd] + JLLWrappers v1.4.0
[30363a11] + NetCDF v0.11.4
[21216c6a] + Preferences v1.2.3
[0234f1f7] + HDF5_jll v1.12.0+1
[7243133f] + NetCDF_jll v400.702.400+0
[458c3c95] + OpenSSL_jll v1.1.13+0

julia> using NetCDF

julia> ncinfo(“”)
ERROR: UndefVarError: libnetcdf not defined
[1] nc_open
@ ~/.julia/packages/NetCDF/CS38M/src/netcdf_c.jl:55 [inlined]
[2] nc_open(fname::String, omode::UInt16)
@ NetCDF ~/.julia/packages/NetCDF/CS38M/src/netcdf_helpers.jl:103
[3] open(fil::String; mode::UInt16, readdimvar::Bool, add_finalizer::Bool)
@ NetCDF ~/.julia/packages/NetCDF/CS38M/src/NetCDF.jl:913
[4] open(f::typeof(show), args::String; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ NetCDF ~/.julia/packages/NetCDF/CS38M/src/NetCDF.jl:999
[5] open
@ ~/.julia/packages/NetCDF/CS38M/src/NetCDF.jl:999 [inlined]
[6] ncinfo(fil::String)
@ NetCDF ~/.julia/packages/NetCDF/CS38M/src/NetCDF.jl:1101
[7] top-level scope
@ REPL[4]:1

It’s not yet supported (except as mentioned in the end). NetCDF.jl has a dependency on NetCDF_jll. JLL packages, are often written in C (which CAN possibly support M1, not a guarantee independently of Julia), to me JLL is “code” for C, but could be other languages.

When people say ARM, or M1, is supported by Julia, it means for pure Julia code (one reason to have that, I still like the ability, and no reason not to use with C, except security, and might not be supported), and Julia itself.

There you can see under Platforms, e.g.

  • macOS x86_64 ( x86_64-apple-darwin )

You probably can use x86 Julia (and then indirectly the Rosetta transpiler).

I’m not sure what it takes to get NetCDF_jll updated. It (and other JLLs, but HDF5_jll and OpenSSL_jll are already taken care of, so this seems to be your last hurdle for that package) need to compile too for M1. It’s clearly not automatic, but might be around the corner. Maybe you can help with (trying) compiling libnetcdf and/or investigating.

In that case I noticed the JLL dependency in the Projects.toml file, but it could have been a dependency of one of those… so I use JuliaHub to see all the dependencies usually, if I need to. It should work, but isn’t always updated with some info, so the Manifest file might be a better place to look…

Currently, the NetCDF JLL package (the compiled binary) isn’t built for M1:

Hopefully this is just a matter of updating the build script.

Looks like the limiting factor is HDF5, which doesn’t yet support cross-compiling:

However, it looks like HDF5 now has precompiled binaries for MacOS on ARM via conda-forge, so once the HDF5 build script is updated, then it should be possible to update NetCDF to build on MacOS/ARM as well.

It is already

It seems we/you, know why this didn’t work out of the box, AND the solution(s). I was going by:

HDF5_jll.jl is available for the following platforms:

  • macOS aarch64 ( aarch64-apple-darwin ) […]

Still, this is sub-optimal for new users. I’m thinking, did the user get a warning? NetCDF_jll was seemingly installed with no (working) support. Should it not have installed? I’m not sure that’s possible, at best to install, and provide a warning on install time. Then that package needs to implement it, or strictly something else (Yggrasil?) for all of them.

I was curious if we do better/worse than Python, it has the same problem:

In general, Python (and R) has a lot of C/otherlang or C++ (very common for R, less so for Python?) dependencies. That works very smoothly most of the time, and users may not even know that Python isn’t doing the heavy lifting. For new platforms, such as M1, maybe they have greater problems, more to fix.

Thank you all for your responses!

Any update?

I installed NetCDF on a M1 Mac Mini with Julia 1.8-rc3 but still couldn’t use it for the same reason. I got the following message with status --outdated -m:

⌅ [0234f1f7] HDF5_jll v1.12.0+1 (<v1.12.2+1): NetCDF_jll
⌅ [7243133f] NetCDF_jll v400.702.400+0 (<v400.802.103+0): NetCDF

The NetCDF.jl package needs to update their compat bounds for the JLL. Could you open an issue if there isn’t one already?

I don’t fully understand what is going on with NetCDF_jll but I think it has some problems now: NetCDF_jll broken on v1.8.0-rc1 on macOS and linux · Issue #182 · Alexander-Barth/NCDatasets.jl · GitHub

For NetCDF_jll basically we haven’t been able to build a version that works on all platforms for a while. The problems seem related to limited cross compilation support of both libnetcdf and libhdf5. That means we got to a state where users on different platforms will have to pin different NetCDF_jll versions for things to work, which we cannot really make easier since Pkg works platform independently. There are many threads and issues but NCDatasets.jl#164 and Yggdrasil#4511 have quite some discussion, and links to upstream issues. Our most recent attempt at updating HDF5_jll did unfortunately not help NetCDF_jll.

Interestingly since last week we now have much better options coming up for netCDF users that don’t want to use libnetcdf / netCDF-C but use pure Julia packages. NetCDF file format 3 is not based on HDF5, and therefore simpler to implement. So @Alexander-Barth has been working on GitHub - Alexander-Barth/NetCDF3.jl for the past week. NetCDF file format 4 is based on HDF5. @JonasIsensee has been doing really great work in read compat for many hdf5 files by JonasIsensee · Pull Request #388 · JuliaIO/JLD2.jl · GitHub, which means that most netCDF 4 files can now be read using that branch, which we are discussing in JLD2 for opening netCDF files · Issue #406 · JuliaIO/JLD2.jl · GitHub. Though automatically parsing the netCDF-specific metadata is probably out of scope for JLD2 and might make more sense in a separate package.

1 Like

I have also a stalled attempt to build GMT due to cross compiling difficulties. I think all will be better when it is accepted that Windows-is-not-Unix.I have no idea on how difficult is to change the building tools. But if we could build the dll’s with Visual Studio (or clang if whatever the Cmake accepts to build native Windows libs) then all those problems would go away.

I love the evil idea of just switching windows to clang+msys+ucrt, but that might be a dream

You do understand that the problem is that you aren’t linking libraries required by Microsoft documentation, which has nothing to do with your Windows-is-not-Unix claim, right?

During years we suffered to learn how completely port GMT to windows. The biggest shit was the cross dll problem (one cannot allocate memory or open a file in one dll and free it/close file in another). There was also the problem of text files on Windows must be explicitly open as such, which Unix does not obliges to. And there several cases of POSIX functions not existing on the Windows API and we had to find the equivalent functionality.

But this has all been solved and GMT builds perfectly fine on Windows using MSVC. We (I’m talking as a GMT dev) never claimed that GMT could be build with MinGW and in fact we say use Cygwin and MinGW at your own risk. That is not officially supported

The problem you refer comes exactly from an apparently issue with building with MinGW. So, at the end I still think that the Windows-is-not-Unix applies here.

Likely because MSVC sneakily links the library for some other reasons. You don’t address the fact that you use a function which requires a specific library and you don’t link that library explicitly. That’s wrong whatever toolchain you use.

It also builds fine with the Intel compilers. But I guess it’s cmake who decide that type of things.