Adding HDF5: Tree Hash Mismatch! (A Docker-on-Windows problem; solved)

General Usage
, , ,
1

I’m running Julia v1.3.1 in Ubuntu and trying to ] add HDF5. I’m getting the following error:

(sim) pkg> add HDF5
 Resolving package versions...
┌ Error: Tree Hash Mismatch!
│   Expected git-tree-sha1:   7d43efbb6eaa15647f7532afa55301abc8e178b0
│   Calculated git-tree-sha1: a9575952a7fa7cea50fa17c5bfc6ff1292b4fd44
└ @ Pkg.Artifacts /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/Artifacts.jl:779
ERROR: Unable to automatically install 'HDF5' from '/root/.julia/packages/HDF5_jll/I6a3H/Artifacts.toml'

Here, I’m in a project environment (sim), but it’s the same if I do it for the default environment (v1.3). This Ubuntu machine is running in a Docker.

When I do this on my Windows machine (same version of Julia), it works fine.

Has anyone seen this before?

2

No. Could you please post the output of versioninfo?

3

Perhaps of interest: Cannot add Arpack - tree hash mismatch · Issue #89 · JuliaLinearAlgebra/Arpack.jl · GitHub

4 
julia> versioninfo()
Julia Version 1.3.1
Commit 2d5741174c (2019-12-30 21:36 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-3770S CPU @ 3.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, ivybridge)
Environment:
  JULIA_PKGDIR = /root/.julia

I’ll note that I’m running this in a Docker. The package directory is shared from the host and mounted at /root/.julia, as can be seen above.

Adding YAML and Interpolations packages worked like normal. Only HDF5 is having a problem.

5

Interesting. I’ll note that the host is Windows, so that might mess with symlinks.

6

If the package directory is shared from the host, you’re probably storing the files on a windows partition and Docker is doing some kind of mapping from host to container that messes with permissions; we are sensitive to the executable bit not being honored (Which is one of the things that could get messed up translating from posix permissions to windows filesystem permissions).

Could you try not mounting the package directory and see if that works for you? Unfortunately I’m not sure what we can do to work around this in the short term (other than try to detect that the filesystem lies to us) but we can at least attempt to get you a setup that works.

7

Thanks @staticfloat. When I use a directory that isn’t shared as the package directory (DEPOT_PATH), it works.

Here’s what I did:

> mkdir /root/.julia2
> JULIA_DEPOT_PATH=/root/.julia2 julia
julia>] add YAML
# Works, and the new YAML shows up in /root/julia2, so I did this part correctly.
julia>] add HDF5
# Now this also works.

So the problem is sharing a directory from a Windows host.

Hm, I’ll need to think about how to do this then. The Docker doesn’t “remember” installed packages, so that’s why I mounted a directory for it to store packages in in the first place. Then packages persist from session to session. I could add packages in the Docker setup, but then other uses of this system would need to re-run the Docker build every time something changes on the Julia side, and they won’t know that they need to do that, so that doesn’t really work. Maybe I can find a way to make the permissions all work out.

Just to check: there’s nothing in the package installer using symlinks, right? I know those don’t work out on mounted drives from Windows.

8

Just to check: there’s nothing in the package installer using symlinks, right? I know those don’t work out on mounted drives from Windows.

Unfortunately, we use symlinks extensively within Linux packages, since they save a lot of space when packages want to have libraries named (for instance) libfoo.so.1.2.3., libfoo.so.1 and libfoo.so all at once.

The Docker doesn’t “remember” installed packages, so that’s why I mounted a directory for it to store packages in in the first place.

You can try declaring a docker volume; instead of bridging something over from your host filesystem, you can just create a named volume. I don’t know for sure what Docker will do there, but I would hope that it would choose the simpler option and not store it in a windows filesystem.

1 Like
9

Thanks @staticfloat! Using a volume instead of a mounted local directory in fact works and is better than my original solution. This was also extremely easy. I owe you a beer.

My Docker command now includes:

docker run -rm -it .... -v julia-pkg-dir:/root/.julia ...

Docker automatically creates the julia-pkg-dir volume, so this is even easier than making a local directory for this purpose on host.

1 Like