Yet another precompilation-on-HPC issue

Internals & Design
1

I administer an HPC cluster and I’ve installed Julia (using spack as a package manager) globally. Spack compiled julia in /tmp/aaa/spack-stage/spack-stage-julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/spack-src/ and then installed it on another location. Somewhat Julia remembers that “compilation” location and at least in certain circumstances it fails to install new packages, with an error such as IOError: stat("/tmp/aaa/spack-stage/spack-stage-julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/spack-src/usr/share/julia/stdlib/v1.9/Zlib_jll/src/Zlib_jll.jl"): permission denied (EACCES). The “certain circumstances” are related to the fact that I may be compiling other software and the directory /tmp/aaa/spack-stage/ exists and is owned by a different user. I am not sure if there’s also a race condition between multiple users attempting to both install at the same time: if Julia does create the directory, it disappears so fast that I have not been able to see it.

Is there a way to tell Julia “use /tmp/$USER/” to workaround this problem?

I tried DEPOT_PATH and BUILD_STDLIB_PATH and even PKG_DEVDIR but I either used them incorrectly or they don’t do what I want. DEPOT_PATH especially appears to break everything else in Julia… Some old bugs mention Base.LOAD_CACHE_PATH but that does not exist anymore and the related Base.LOADING_CACHE and Base.LOAD_PATH do not seem to do what I want.

Suggestions?

2

Set TMPDIR:

% TMPDIR="/tmp/${USER}" julia -E 'tempdir()'
"/tmp/mose"

See tempdir

2 Likes
3

If you are running with a batch system then $TMPDIR should be set by the batch system and this area created and torn down at the end of the job. You can customise the location of $TMPDIR in the batch system config files

1 Like
4

Thanks both for the suggestion, but unfortunately that’s not it.

Setting TMPDIR as indicated makes it visible by julia as in the example @giordano showed. Despite that running things like using Revise still attempts to precompile it in that not appropriate location and fails with IOError: stat("/tmp/aaa/spack-stage/spack-stage-julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/spack-src/usr/share/julia/stdlib/v1.9/LibSSH2_jll/src/LibSSH2_jll.jl"): permission denied (EACCES)

Just in case I tried deleting my ~/.julia directory and of course in that way it says

ERROR: ArgumentError: Package Revise not found in current path.
- Run `import Pkg; Pkg.add("Revise")` to install the Revise package.

Redoing the import Pkg; Pkg.add("Revise") makes it fail again in the same way, so it must be using something of higher priority for setting this location.

Any idea about what that may be?

5

Just to show that I did not fat-finger this, or if there is any additional clue that can help debug the issue, here is a full capture

$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.2 (2023-07-05)
 _/ |\__'_|_|_|\__'_|  |
|__/                   |

julia> tempdir()
"/tmp/davide"

julia> import Pkg; Pkg.add("Revise")
  Installing known registries into `~/.julia`
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = RequestError: Recv failure: Connection reset by peer while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry /home/sw/spack-v0.20.1-b/opt/spack/linux-rhel8-icelake/gcc-13.1.0/julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/share/julia/stdlib/v1.9/Pkg/src/Registry/Registry.jl:69
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = RequestError: Recv failure: Connection reset by peer while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry /home/sw/spack-v0.20.1-b/opt/spack/linux-rhel8-icelake/gcc-13.1.0/julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/share/julia/stdlib/v1.9/Pkg/src/Registry/Registry.jl:69
     Cloning registry from "https://github.com/JuliaRegistries/General.git"
       Added registry `General` to `~/.julia/registries/General`
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = RequestError: Recv failure: Connection reset by peer while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry /home/sw/spack-v0.20.1-b/opt/spack/linux-rhel8-icelake/gcc-13.1.0/julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/share/julia/stdlib/v1.9/Pkg/src/Registry/Registry.jl:69
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = RequestError: Recv failure: Connection reset by peer while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry /home/sw/spack-v0.20.1-b/opt/spack/linux-rhel8-icelake/gcc-13.1.0/julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/share/julia/stdlib/v1.9/Pkg/src/Registry/Registry.jl:69
   Installed Requires ─────────── v1.3.0
   Installed CodeTracking ─────── v1.3.5
   Installed LoweredCodeUtils ─── v2.3.0
   Installed JuliaInterpreter ─── v0.9.26
   Installed OrderedCollections ─ v1.6.2
   Installed Revise ───────────── v3.5.7
    Updating `~/.julia/environments/v1.9/Project.toml`
  [295af30f] + Revise v3.5.7
    Updating `~/.julia/environments/v1.9/Manifest.toml`
  [da1fd8a2] + CodeTracking v1.3.5
  [aa1ae85d] + JuliaInterpreter v0.9.26
  [6f1432cf] + LoweredCodeUtils v2.3.0
  [bac558e1] + OrderedCollections v1.6.2
  [ae029012] + Requires v1.3.0
  [295af30f] + Revise v3.5.7
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8ba89e20] + Distributed
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [b27032c2] + LibCURL v0.6.3
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.9.2
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [deac9b47] + LibCURL_jll v7.84.0+0
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [c8ffd9c3] + MbedTLS_jll v2.28.2+0
  [14a3606d] + MozillaCACerts_jll v2022.10.11
  [83775a58] + Zlib_jll v1.2.13+0
  [8e850ede] + nghttp2_jll v1.48.0+0
  [3f19e933] + p7zip_jll v17.4.0+0

julia> ┌ Error: Pkg.precompile error
│   exception =
│    IOError: stat("/tmp/aaa/spack-stage/spack-stage-julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/spack-src/usr/share/julia/stdlib/v1.9/MbedTLS_jll/src/MbedTLS_jll.jl"): permission denied (EACCES)
│    Stacktrace:
│      [1] uv_error
│        @ ./libuv.jl:100 [inlined]
│      [2] stat(path::String)
│        @ Base.Filesystem ./stat.jl:152
│      [3] samefile
│        @ ./stat.jl:476 [inlined]
│      [4] stale_cachefile(modkey::Base.PkgId, build_id::UInt128, modpath::String, cachefile::String; ignore_loaded::Bool)
│        @ Base ./loading.jl:2818
│      [5] stale_cachefile
│        @ ./loading.jl:2717 [inlined]
│      [6] #stale_cachefile#954
│        @ ./loading.jl:2715 [inlined]
│      [7] stale_cachefile
│        @ ./loading.jl:2714 [inlined]
│      [8] _is_stale!(stale_cache::Dict{Tuple{Base.PkgId, UInt128, String, String}, Bool}, paths::Vector{String}, sourcepath::String)
│        @ Pkg.API /home/sw/spack-v0.20.1-b/opt/spack/linux-rhel8-icelake/gcc-13.1.0/julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/share/julia/stdlib/v1.9/Pkg/src/API.jl:1061
│      [9] macro expansion
│        @ /home/sw/spack-v0.20.1-b/opt/spack/linux-rhel8-icelake/gcc-13.1.0/julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/share/julia/stdlib/v1.9/Pkg/src/API.jl:1415 [inlined]
│     [10] (::Pkg.API.var"#245#276"{Bool, Bool, Pkg.Types.Context, Vector{Task}, Dict{Base.PkgId, String}, Dict{Base.PkgId, IOBuffer}, Pkg.API.var"#color_string#266", Base.Event, Base.Event, ReentrantLock,
Vector{Base.PkgId}, Vector{Base.PkgId}, Dict{Base.PkgId, String}, Vector{Base.PkgId}, Vector{Base.PkgId}, Dict{Base.PkgId, Bool}, Dict{Base.PkgId, Base.Event}, Dict{Base.PkgId, Bool}, Vector{Pkg.Types.Packa
geSpec}, Dict{Base.PkgId, String}, Dict{Tuple{Base.PkgId, UInt128, String, String}, Bool}, Vector{Base.PkgId}, Bool, Base.TTY, Base.Semaphore, String, Vector{String}, Vector{Base.PkgId}, Base.PkgId})()
│        @ Pkg.API ./task.jl:514
└ @ Pkg.API /home/sw/spack-v0.20.1-b/opt/spack/linux-rhel8-icelake/gcc-13.1.0/julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/share/julia/stdlib/v1.9/Pkg/src/API.jl:1302
┌ Error: Pkg.precompile error
│   exception =
│    IOError: stat("/tmp/aaa/spack-stage/spack-stage-julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/spack-src/usr/share/julia/stdlib/v1.9/Zlib_jll/src/Zlib_jll.jl"): permission denied (EACCES)
│    Stacktrace:
│      [1] uv_error
│        @ ./libuv.jl:100 [inlined]
│      [2] stat(path::String)
│        @ Base.Filesystem ./stat.jl:152
│      [3] samefile
│        @ ./stat.jl:476 [inlined]
│      [4] stale_cachefile(modkey::Base.PkgId, build_id::UInt128, modpath::String, cachefile::String; ignore_loaded::Bool)
│        @ Base ./loading.jl:2818
│      [5] stale_cachefile
│        @ ./loading.jl:2717 [inlined]
│      [6] #stale_cachefile#954
│        @ ./loading.jl:2715 [inlined]
│      [7] stale_cachefile
│        @ ./loading.jl:2714 [inlined]
│      [8] _is_stale!(stale_cache::Dict{Tuple{Base.PkgId, UInt128, String, String}, Bool}, paths::Vector{String}, sourcepath::String)
│        @ Pkg.API /home/sw/spack-v0.20.1-b/opt/spack/linux-rhel8-icelake/gcc-13.1.0/julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/share/julia/stdlib/v1.9/Pkg/src/API.jl:1061
│      [9] macro expansion
│        @ /home/sw/spack-v0.20.1-b/opt/spack/linux-rhel8-icelake/gcc-13.1.0/julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/share/julia/stdlib/v1.9/Pkg/src/API.jl:1415 [inlined]
│     [10] (::Pkg.API.var"#245#276"{Bool, Bool, Pkg.Types.Context, Vector{Task}, Dict{Base.PkgId, String}, Dict{Base.PkgId, IOBuffer}, Pkg.API.var"#color_string#266", Base.Event, Base.Event, ReentrantLock,
Vector{Base.PkgId}, Vector{Base.PkgId}, Dict{Base.PkgId, String}, Vector{Base.PkgId}, Vector{Base.PkgId}, Dict{Base.PkgId, Bool}, Dict{Base.PkgId, Base.Event}, Dict{Base.PkgId, Bool}, Vector{Pkg.Types.Packa
geSpec}, Dict{Base.PkgId, String}, Dict{Tuple{Base.PkgId, UInt128, String, String}, Bool}, Vector{Base.PkgId}, Bool, Base.TTY, Base.Semaphore, String, Vector{String}, Vector{Base.PkgId}, Base.PkgId})()
│        @ Pkg.API ./task.jl:514
└ @ Pkg.API /home/sw/spack-v0.20.1-b/opt/spack/linux-rhel8-icelake/gcc-13.1.0/julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/share/julia/stdlib/v1.9/Pkg/src/API.jl:1302
┌ Error: Pkg.precompile error
│   exception =
│    IOError: stat("/tmp/aaa/spack-stage/spack-stage-julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/spack-src/usr/share/julia/stdlib/v1.9/LibSSH2_jll/src/LibSSH2_jll.jl"): permission denied (EACCES)
┌ Error: Pkg.precompile error
│   exception =
│    IOError: stat("/tmp/aaa/spack-stage/spack-stage-julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/spack-src/usr/share/julia/stdlib/v1.9/LibSSH2_jll/src/LibSSH2_jll.jl"): permission denied (EACCES)
│    Stacktrace:
│      [1] uv_error
│        @ ./libuv.jl:100 [inlined]
│      [2] stat(path::String)
│        @ Base.Filesystem ./stat.jl:152
│      [3] samefile
│        @ ./stat.jl:476 [inlined]
│      [4] stale_cachefile(modkey::Base.PkgId, build_id::UInt128, modpath::String, cachefile::String; ignore_loaded::Bool)
│        @ Base ./loading.jl:2818
│      [5] stale_cachefile
│        @ ./loading.jl:2717 [inlined]
│      [6] #stale_cachefile#954
│        @ ./loading.jl:2715 [inlined]
│      [7] stale_cachefile
│        @ ./loading.jl:2714 [inlined]
│      [8] _is_stale!(stale_cache::Dict{Tuple{Base.PkgId, UInt128, String, String}, Bool}, paths::Vector{String}, sourcepath::String)
│        @ Pkg.API /home/sw/spack-v0.20.1-b/opt/spack/linux-rhel8-icelake/gcc-13.1.0/julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/share/julia/stdlib/v1.9/Pkg/src/API.jl:1061
│      [9] macro expansion
│        @ /home/sw/spack-v0.20.1-b/opt/spack/linux-rhel8-icelake/gcc-13.1.0/julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/share/julia/stdlib/v1.9/Pkg/src/API.jl:1415 [inlined]
│     [10] (::Pkg.API.var"#245#276"{Bool, Bool, Pkg.Types.Context, Vector{Task}, Dict{Base.PkgId, String}, Dict{Base.PkgId, IOBuffer}, Pkg.API.var"#color_string#266", Base.Event, Base.Event, ReentrantLock,
Vector{Base.PkgId}, Vector{Base.PkgId}, Dict{Base.PkgId, String}, Vector{Base.PkgId}, Vector{Base.PkgId}, Dict{Base.PkgId, Bool}, Dict{Base.PkgId, Base.Event}, Dict{Base.PkgId, Bool}, Vector{Pkg.Types.Packa
geSpec}, Dict{Base.PkgId, String}, Dict{Tuple{Base.PkgId, UInt128, String, String}, Bool}, Vector{Base.PkgId}, Bool, Base.TTY, Base.Semaphore, String, Vector{String}, Vector{Base.PkgId}, Base.PkgId})()
│        @ Pkg.API ./task.jl:514
└ @ Pkg.API /home/sw/spack-v0.20.1-b/opt/spack/linux-rhel8-icelake/gcc-13.1.0/julia-1.9.2-yux5s54oyok6xiifl5xwqeqjljazifjg/share/julia/stdlib/v1.9/Pkg/src/API.jl:1302
julia>

Note that sw is not an user of this machine and that /home/sw/ is only a location shared for everybody to read (as /opt is on other machines).

Note also (not shown) that the directory /tmp/davide exists and is writable by current user.

Thanks a lot to anybody who will provide additional insight!

6

For the benefit of future readers, we found the culprit and a workaround:

  • this happens only on Julia v1.10-, it appears to be fixed in v1.11+
  • apparently the issue arises when the julia build directory exists but it isn’t readable/executable by the user running the Julia process (which may happen if for example the package was built by a sysadmin and then a different user tries to run julia)
  • more details at Precompile errors with Spack compilation and deployment · Issue #55966 · JuliaLang/julia · GitHub
  • a workaround is to delete the root of the julia build directory (i.e. in the case of spack the stage directory, which isn’t needed anymore if julia was built correctly) and make everyone happy. Alternatively, make the stage directory executable by all users (chmod o+x /path/to/spack-stage)
2 Likes