Precompilation in a sandbox

General Usage
, , ,
1

I’m making sandboxed and reproducible Julia builds. One way I do this is by compiling each Julia package using Pkg.API.precompile(). I ensure that $JULIA_DEPOT is set to a list of paths which contains compiled/v1.11/Logging, along with other packages like ArgTools. Currently, $JULIA_DEPOT = :$A:$B, where

$ exa --follow-symlinks --tree -L 3 $A
$A
├── compiled
│   └── v1.11
│       └── TextWrap
$ exa --follow-symlinks --tree -L 3 $B
$B
└── compiled
    └── v1.11
        ├── ArgTools
        ├── Base64
        ├── Dates
        ├── Downloads
        ├── LibCURL
        ├── LibCURL_jll
        ├── LibGit2
        ├── LibGit2_jll
        ├── LibSSH2_jll
        ├── Logging
        ├── Markdown
        ├── MbedTLS_jll
        ├── MozillaCACerts_jll
        ├── NetworkOptions
        ├── nghttp2_jll
        ├── p7zip_jll
        ├── Pkg
        ├── Printf
        ├── Tar
        ├── TOML
        └── UUIDs

However, when I tried to run Pkg.precompile() the ArgParse package as a dependency of another package, using this particular set of depots, I got

ERROR: LoadError: KeyError: key Base.PkgId(UUID("56ddb016-857b-54e1-b83d-db4d58db5568"), "Logging") not found
Stacktrace:
  [1] getindex
    @ ./dict.jl:477 [inlined]
  [2] (::Base.Precompilation.var"#scan_deps!#51"{Base.Precompilation.var"#scan_pkg!#50"{Dict{Base.PkgId, Bool}}, Vector{Base.PkgId}, Dict{Base.PkgId, Bool}, Vector{Vector{Base.PkgId}}})(pkg::Base.PkgId, dmap::Dict{Base.PkgId, Vector{Base.PkgId}})
    @ Base.Precompilation ./precompilation.jl:613
  [3] (::Base.Precompilation.var"#scan_pkg!#50"{Dict{Base.PkgId, Bool}})(pkg::Base.PkgId, dmap::Dict{Base.PkgId, Vector{Base.PkgId}})
    @ Base.Precompilation ./precompilation.jl:607
  [4] (::Base.Precompilation.var"#scan_deps!#51"{Base.Precompilation.var"#scan_pkg!#50"{Dict{Base.PkgId, Bool}}, Vector{Base.PkgId}, Dict{Base.PkgId, Bool}, Vector{Vector{Base.PkgId}}})(pkg::Base.PkgId, dmap::Dict{Base.PkgId, Vector{Base.PkgId}})
    @ Base.Precompilation ./precompilation.jl:620
  [5] (::Base.Precompilation.var"#scan_pkg!#50"{Dict{Base.PkgId, Bool}})(pkg::Base.PkgId, dmap::Dict{Base.PkgId, Vector{Base.PkgId}})
    @ Base.Precompilation ./precompilation.jl:607
  [6] _precompilepkgs(pkgs::Vector{String}, internal_call::Bool, strict::Bool, warn_loaded::Bool, timing::Bool, _from_loading::Bool, configs::Vector{Pair{Cmd, Base.CacheFlags}}, io::IOContext{IO}, fancyprint::Bool, ignore_loaded::Bool)
    @ Base.Precompilation ./precompilation.jl:642
  [7] #precompilepkgs#10
    @ ./precompilation.jl:411 [inlined]
  [8] (::Pkg.API.var"#225#226"{Bool, Bool, Bool, Bool, Bool, Pair{Cmd, Base.CacheFlags}, Vector{Pkg.Types.PackageSpec}})()
    @ Pkg.API /nix/store/z1lkk0f61gnxlz3r8mmp7jlj3q04n048-julia-bin-1.11.6/share/julia/stdlib/v1.11/Pkg/src/API.jl:1158
  [9] activate(f::Pkg.API.var"#225#226"{Bool, Bool, Bool, Bool, Bool, Pair{Cmd, Base.CacheFlags}, Vector{Pkg.Types.PackageSpec}}, new_project::String)
    @ Pkg.API /nix/store/z1lkk0f61gnxlz3r8mmp7jlj3q04n048-julia-bin-1.11.6/share/julia/stdlib/v1.11/Pkg/src/API.jl:1379
 [10] precompile(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; internal_call::Bool, strict::Bool, warn_loaded::Bool, already_instantiated::Bool, timing::Bool, _from_loading::Bool, configs::Pair{Cmd, Base.CacheFlags}, kwargs::@Kwargs{io::IOContext{IO}})
    @ Pkg.API /nix/store/z1lkk0f61gnxlz3r8mmp7jlj3q04n048-julia-bin-1.11.6/share/julia/stdlib/v1.11/Pkg/src/API.jl:1156
 [11] precompile(pkgs::Vector{Pkg.Types.PackageSpec}; io::IOContext{IO}, kwargs::@Kwargs{already_instantiated::Bool})
    @ Pkg.API /nix/store/z1lkk0f61gnxlz3r8mmp7jlj3q04n048-julia-bin-1.11.6/share/julia/stdlib/v1.11/Pkg/src/API.jl:159
 [12] precompile(; name::Nothing, uuid::Nothing, version::Nothing, url::Nothing, rev::Nothing, path::Nothing, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::@Kwargs{already_instantiated::Bool})
    @ Pkg.API /nix/store/z1lkk0f61gnxlz3r8mmp7jlj3q04n048-julia-bin-1.11.6/share/julia/stdlib/v1.11/Pkg/src/API.jl:174

What could be causing this issue? Logging is a stdlib package so it is extra puzzling why Julia complains that it doesn’t exist. Notably ArgParse doesn’t have a manifest file, so I may need to create it.

I also tried setting JULIA_PKG_OFFLINE but it had no effect. The version I’m using is 1.11.6.

2

I solved this issue. It was because of a few reasons:

  1. ArgParse does not have a Manifest.toml file, so Julia could not load dependencies from the manifest.
  2. The dependencies of ArgParse need to be exposed via LOAD_PATH

With these issues resolved, the whole package can build along with its dependencies, with precompilation cached for efficiency.

2 Likes