Using `download` during dep build (Julia v1.11)

Hi!

The build script from package Oracle uses download function defined in the stdlib (Oracle.jl/deps/build.jl at cad6bc5bde0f1bd65cff493ca2113d7fe3f2d9e2 · felipenoris/Oracle.jl · GitHub). When migrating to Julia v1.11, I get this error during the package build.

If I use download directly from the REPL it works. Any clues on how to fix this?

(@v1.11) pkg> build Oracle
    Building Oracle → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/abc9a42efdcf2567980472d271bf941e4742514b/build.log`
ERROR: Error building `Oracle`: 
ERROR: LoadError: ArgumentError: Package Downloads does not have NetworkOptions in its dependencies:
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- Or, if you have Downloads checked out for development and have
  added NetworkOptions as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with Downloads
Stacktrace:
 [1] macro expansion
   @ ./loading.jl:2211 [inlined]
 [2] macro expansion
   @ ./lock.jl:273 [inlined]
 [3] __require(into::Module, mod::Symbol)
   @ Base ./loading.jl:2183
 [4] #invoke_in_world#3
   @ ./essentials.jl:1088 [inlined]
 [5] invoke_in_world
   @ ./essentials.jl:1085 [inlined]
 [6] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:2176
 [7] include
   @ ./Base.jl:557 [inlined]
 [8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
   @ Base ./loading.jl:2806
 [9] top-level scope
   @ stdin:4
in expression starting at /home/fnoro/work/local/julia-1.11.0/share/julia/stdlib/v1.11/Downloads/src/Downloads.jl:1
in expression starting at stdin:4
ERROR: LoadError: Failed to precompile Downloads [f43a241f-c20a-4ad4-852c-f6b1247861c6] to "/home/fnoro/.julia/compiled/v1.11/Downloads/jl_emAbKP".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool; flags::Cmd, cacheflags::Base.CacheFlags, reasons::Dict{String, Int64})
    @ Base ./loading.jl:3089
  [3] (::Base.var"#1081#1082"{Base.PkgId})()
    @ Base ./loading.jl:2477
  [4] mkpidlock(f::Base.var"#1081#1082"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile ~/work/local/julia-1.11.0/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:95
  [5] #mkpidlock#6
    @ ~/work/local/julia-1.11.0/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:90 [inlined]
  [6] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile ~/work/local/julia-1.11.0/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:116
  [7] #invokelatest#2
    @ ./essentials.jl:1056 [inlined]
  [8] invokelatest
    @ ./essentials.jl:1051 [inlined]
  [9] maybe_cachefile_lock(f::Base.var"#1081#1082"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:3613
 [10] maybe_cachefile_lock
    @ ./loading.jl:3610 [inlined]
 [11] _require(pkg::Base.PkgId, env::Nothing)
    @ Base ./loading.jl:2473
 [12] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
    @ Base ./loading.jl:2300
 [13] #invoke_in_world#3
    @ ./essentials.jl:1088 [inlined]
 [14] invoke_in_world
    @ ./essentials.jl:1085 [inlined]
 [15] _require_prelocked
    @ ./loading.jl:2287 [inlined]
 [16] _require_prelocked
    @ ./loading.jl:2286 [inlined]
 [17] macro expansion
    @ ./lock.jl:273 [inlined]
 [18] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:2283
 [19] Downloads
    @ ./download.jl:3 [inlined]
 [20] do_download(url::String, path::String)
    @ Base ./download.jl:24
 [21] download
    @ ./download.jl:19 [inlined]
 [22] download_source_files(; verbose::Bool)
    @ Main ~/.julia/packages/Oracle/iaH6i/deps/build.jl:56
 [23] download_source_files
    @ ~/.julia/packages/Oracle/iaH6i/deps/build.jl:40 [inlined]
 [24] main(; verbose::Bool)
    @ Main ~/.julia/packages/Oracle/iaH6i/deps/build.jl:169
 [25] main()
    @ Main ~/.julia/packages/Oracle/iaH6i/deps/build.jl:167
 [26] top-level scope
    @ ~/.julia/packages/Oracle/iaH6i/deps/build.jl:175
 [27] include(fname::String)
    @ Main ./sysimg.jl:38
 [28] top-level scope
    @ none:5
in expression starting at /home/fnoro/.julia/packages/Oracle/iaH6i/deps/build.jl:175

I could confirm this at first, with same error message, but then I managed on 1.11:

julia> @time using Oracle
Precompiling Oracle...
  1 dependency successfully precompiled in 3 seconds. 9 already precompiled.
  5.270485 seconds (811.50 k allocations: 49.501 MiB, 2.42% gc time, 40.96% compilation time)

I started with:

$ julia +1.11 --project=@.

So at least know it’s possible, and might not even be 1.11 related, I also had a problem with 1.10.5, and no longer.

So what did I do in the mean-time while it was failing? I at least deleted (actually moved out of the way) my ~/Manifest.toml but that alone didn’t work (but should have?).

Some of the issues I had:

julia> Pkg.build("Oracle")
    Building Oracle → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/abc9a42efdcf2567980472d271bf941e4742514b/build.log`
Precompiling project...
^C Interrupted: Exiting precompilation...            ]  4/18
  ◑ TidierCats Being precompiled by an async task in this process (pidfile: /home/pharaldsson/.julia/compiled/v1.10/TidierCats/cr3D3_mupLX.ji.pidfile)
  ◑ TidierData Being precompiled by an async task in this process (pidfile: /home/pharaldsson/.julia/compiled/v1.10/TidierData/ggVOJ_mupLX.ji.pidfile)

was seemingly hanging, and I do CTRL-C a lot while precompiling, here unreated packagtes. I thought it was safe, but no longer sure, I think I maybe be making trouble with it, that's still I think always solvable like I did.


julia> Pkg.build("Oracle")
    Building Oracle → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/abc9a42efdcf2567980472d271bf941e4742514b/build.log`
Precompiling project...
  Progress [====>                                    ]  1/13
  ◐ Stipple Being precompiled by an async task in this process (pidfile: /home/pharaldsson/.julia/compiled/v1.10/Stipple/9RuwB_mupLX.ji.pidfile)
  ◑ Makie Being precompiled by an async task in this process (pidfile: /home/pharaldsson/.julia/compiled/v1.10/Makie/iZ1Bl_mupLX.ji.pidfile)

I wonder if this is intentional due to 5. Creating Packages · Pkg.jl and GitHub - JuliaPackaging/Scratch.jl: Scratch spaces for all your persistent mutable data needs.

“A build step should generally not create or modify any files in the package directory. If you need to store some files from the build step, use the Scratch.jl package.”

I’m seeing the same error during a package test: Fix reading gzipped file in Julia 1.11 on Windows · JuliaData/CSV.jl@232d611 · GitHub

Oh… there’s a good clue here:

┌ Warning: Base.download is deprecated; use Downloads.download instead
│   caller = macro expansion at basics.jl:385 [inlined]

I think this will be fixed in v1.11.2 by Make loading work when stdlib deps are missing in the manifest by IanButterworth · Pull Request #56148 · JuliaLang/julia · GitHub

1 Like

So, the solution to my case was to simply to add Downloads package to deps, and replace all calls to Base.download to Downloads.download.

1 Like