Precompilation: Why does order matter when "using" packages for the first time?

New to Julia
1

Last night, I was trying to get a Lux tutorial running. This was the order in the tutorial:

using Lux, ComponentArrays, SciMLSensitivity, LuxAMDGPU, LuxCUDA,
Optimisers, OrdinaryDiffEq, Random, Statistics, Zygote, OneHotArrays, InteractiveUtils

Some I had. Others needed to be installed, which seemed to have worked fine.

However, when trying to use the packages, I kept getting a message to restart Julia because one of the packages was not precompiled correctly (ComponentArrays, or some ComponentArraysExt extension). Then it would repeat the precompilation attempt and give the same warning. Sorry I didn’t save the logs.

I found Zygote had to come first, then ComponentArrays, then Lux:

using Zygote, ComponentArrays, Lux, SciMLSensitivity, Optimisers, 
OrdinaryDiffEq, Random, Statistics, OneHotArrays, InteractiveUtils

I moved the AMDGPU/CUDA packages out for another reason, but they didn’t seem to play a role here. Eventually I got the idea to put ComponentArrays before Lux and this fixed one of the warnings. Then I put Zygote before that and it fixed the last one.

Is this the expected behavior?

2

It’s hard to tell without more details but certainly, when you import a new package, already (pre)compiled code can be invalidated and lead to additional (pre)compilations. So in fact, order does matter but most of them time and especially with recent version of Julia, invalidations are getting less and less.

Btw. I don’t know how a one-liner using A, B, C unfolds internally, but I don’t think there are any optimisations behind the scenes to check e.g. for common dependencies and try to guess how to minimise invalidations.

The warnings you mention are however another story. It depends which kind of warnings you got. Can you reproduce any of them? I don’t have such issues very often, most of the time I face precompilations are when I am heavily messing around with the REPL by changing lots of code, adding/removing packages and letting Revise.jl do it’s work. That can sometimes blow up, but very very rarely :wink:

1 Like
3

I was able to retrace my steps I believe.

Start here:

  [a8cc5b0e] Crayons v4.1.1
  [a93c6f00] DataFrames v1.6.1
  [864edb3b] DataStructures v0.18.15
  [31c24e10] Distributions v0.25.103
  [634d3b9d] DrWatson v2.13.0
  [3beb2ed1] PDFmerger v0.3.0
  [92933f4c] ProgressMeter v1.9.0
  [6f49c342] RCall v0.13.18
  [2913bbd2] StatsBase v0.34.2
  [f3b207a7] StatsPlots v0.15.6
  [dbf13d8f] ThreadTools v0.2.0
  [b8865327] UnicodePlots v3.6.0
  [de0858da] Printf
  [9a3f8284] Random

Add AMDGPU (installs the newest version):

  [21141c5a] AMDGPU v0.8.0
  [a8cc5b0e] Crayons v4.1.1
  [a93c6f00] DataFrames v1.6.1
  [864edb3b] DataStructures v0.18.15
  [31c24e10] Distributions v0.25.103
  [634d3b9d] DrWatson v2.13.0
  [3beb2ed1] PDFmerger v0.3.0
  [92933f4c] ProgressMeter v1.9.0
  [6f49c342] RCall v0.13.18
  [2913bbd2] StatsBase v0.34.2
  [dbf13d8f] ThreadTools v0.2.0
  [b8865327] UnicodePlots v3.6.0
  [de0858da] Printf
  [9a3f8284] Random

And yes, I had forgotten. There were some issues during the install of the Lux tutorial packages:

So then I restarted the REPL and got more failures to precompile. It kept telling me stuff like:

┌ Warning: Module LuxLibTrackerExt with build ID fafbfcfd-7025-f8e4-0000-58a066eeb70f is missing from the cache.

│ This may mean LuxLibTrackerExt [26da0b32-022f-5c34-b83a-bb48123e9290] does not support precompilation but is imported by a module that does.

â”” @ Base loading.jl:1942

┌ Error: Error during loading of extension LuxLibTrackerExt of LuxLib, use `Base.retry_load_extensions()` to retry.

[...]

11 dependencies precompiled but different versions are currently loaded. Restart julia to access the new versions

Now I tried changing the order, but get variations on the same theme… I may have also installed individually in the package manager at some point in who knows what order. Perhaps that mattered? Anyway, now I am back to where I was last night.

4

So, I removed all those new packages and wanted to get back to the AMDGPU v0.8.0 state. But now that package has updated to v0.8.1 (since I posted the OP):

  [21141c5a] AMDGPU v0.8.1
  [a8cc5b0e] Crayons v4.1.1
  [a93c6f00] DataFrames v1.6.1
  [864edb3b] DataStructures v0.18.15
  [31c24e10] Distributions v0.25.103
  [634d3b9d] DrWatson v2.13.0
  [3beb2ed1] PDFmerger v0.3.0
  [6f49c342] RCall v0.13.18
  [2913bbd2] StatsBase v0.34.2
  [f3b207a7] StatsPlots v0.15.6
  [dbf13d8f] ThreadTools v0.2.0
  [b8865327] UnicodePlots v3.6.0
  [de0858da] Printf
  [9a3f8284] Random

So perhaps other package versions from last night were different from today…

5

Ok, so doing this cleared up a lot of the errors. But now I am stuck in the “3 dependencies precompiled, restart Julia” loop:

] add ComponentArrays
] add Zygote
] add Lux
] add LuxAMDGPU

using Zygote, ComponentArrays, Lux, SciMLSensitivity, Optimisers,
    OrdinaryDiffEq, Random, Statistics, OneHotArrays, InteractiveUtils

julia> using Zygote, ComponentArrays, Lux, SciMLSensitivity, Optimisers,
           OrdinaryDiffEq, Random, Statistics, OneHotArrays, InteractiveUtils
 │ Packages [SciMLSensitivity, Optimisers, OrdinaryDiffEq, OneHotArrays] not found, but packages named [SciMLSensitivity, Optimisers, OrdinaryDiffEq,
 │ OneHotArrays] are available from a registry. 
 │ Install packages?
 │   ( ) pkg> add SciMLSensitivity Optimisers OrdinaryDiffEq OneHotArrays 
 â”” (y/n/o) [y]: y
   Resolving package versions...
    Updating `/media/user1/2TB_ssd/JuliaProjects/ /Project.toml`
  [0b1bfda6] + OneHotArrays v0.2.4
  [3bd65402] + Optimisers v0.3.1
  [1dea7af3] + OrdinaryDiffEq v6.59.3
  [1ed8b502] + SciMLSensitivity v7.47.0
    Updating `/media/user1/2TB_ssd/JuliaProjects/ /Manifest.toml`
  [7d9f7c33] + Accessors v0.1.33
  [ec485272] + ArnoldiMethod v0.2.0
  [62783981] + BitTwiddlingConvenienceFunctions v0.1.5
  [2a0fbf3d] + CPUSummary v0.2.4
  [7057c7e9] + Cassette v0.3.12
  [fb6a15b2] + CloseOpenIntervals v0.1.12
  [38540f10] + CommonSolve v0.2.4
  [a33af91c] + CompositionsBase v0.1.2
  [adafc99b] + CpuId v0.3.1
  [2b5f629d] + DiffEqBase v6.141.0
  [459566f4] + DiffEqCallbacks v2.34.0
  [77a26b50] + DiffEqNoiseProcess v5.19.0
  [da5c29d0] + EllipsisNotation v1.8.0
  [4e289a0a] + EnumX v1.0.4
  [7da242da] + Enzyme v0.11.10
  [f151be2c] + EnzymeCore v0.6.4
  [d4d017d3] + ExponentialUtilities v1.25.0
  [7034ab61] + FastBroadcast v0.2.8
  [9aa1b823] + FastClosures v0.3.2
  [29a986be] + FastLapackInterface v2.0.0
  [6a86dc24] + FiniteDiff v2.21.1
  [f62d2435] + FunctionProperties v0.1.0
  [069b7b12] + FunctionWrappers v1.1.3
  [77dc65aa] + FunctionWrappersWrappers v0.1.3
  [c145ed77] + GenericSchur v0.5.3
  [86223c79] + Graphs v1.9.0
  [3e5b6fbb] + HostCPUFeatures v0.1.16
  [d25df0c9] + Inflate v0.1.4
  [18e54dd8] + IntegerMathUtils v0.1.2
  [3587e190] + InverseFunctions v0.1.12
  [ccbc3e58] + JumpProcesses v9.8.0
  [ef3ab10e] + KLU v0.4.1
  [ba0b0d4f] + Krylov v0.9.4
  [73f95e8e] + LatticeRules v0.0.1
  [10f19ff3] + LayoutPointers v0.1.15
  [50d2b5c4] + Lazy v0.15.1
  [2d8b4e74] + LevyArea v1.0.0
  [d3d80556] + LineSearches v7.2.0
  [7ed4a6bd] + LinearSolve v2.20.0
  [bdcacae8] + LoopVectorization v0.12.166
  [d125e4d3] + ManualMemory v0.1.8
  [46d2c3a1] + MuladdMacro v0.2.4
  [d41bc354] + NLSolversBase v7.8.3
  [2774e3e8] + NLsolve v4.5.1
  [8913a72c] + NonlinearSolve v2.8.2
  [d8793406] + ObjectFile v0.4.1
  [0b1bfda6] + OneHotArrays v0.2.4
  [429524aa] + Optim v1.7.8
  [1dea7af3] + OrdinaryDiffEq v6.59.3
  [d96e819e] + Parameters v0.12.3
  [e409e4f3] + PoissonRandom v0.4.4
  [f517fe37] + Polyester v0.7.9
  [1d0040c9] + PolyesterWeave v0.2.1
  [85a6dd25] + PositiveFactorizations v0.2.4
  [d236fae5] + PreallocationTools v0.4.12
  [27ebfcd6] + Primes v0.5.5
  [8a4e6c94] + QuasiMonteCarlo v0.3.3
  [74087812] + Random123 v1.6.1
  [e6cf234a] + RandomNumbers v1.5.3
  [731186ca] + RecursiveArrayTools v2.38.10
  [f2c3362d] + RecursiveFactorization v0.2.21
  [ae5879a3] + ResettableStacks v1.1.1
  [37e2e3b7] + ReverseDiff v1.15.1
  [7e49a35a] + RuntimeGeneratedFunctions v0.5.12
  [94e857df] + SIMDTypes v0.1.0
  [476501e8] + SLEEFPirates v0.6.42
  [0bca4576] + SciMLBase v2.9.1
  [e9a6253c] + SciMLNLSolve v0.1.9
  [c0aeaf25] + SciMLOperators v0.3.7
  [1ed8b502] + SciMLSensitivity v7.47.0
  [727e6d20] + SimpleNonlinearSolve v0.1.25
  [699a6c99] + SimpleTraits v0.9.4
  [ce78b400] + SimpleUnPack v1.1.0
  [ed01d8cd] + Sobol v1.5.0
  [47a9eef4] + SparseDiffTools v2.13.0
  [e56a9233] + Sparspak v0.3.9
  [789caeaf] + StochasticDiffEq v6.63.2
  [7792a7ef] + StrideArraysCore v0.5.2
  [53d494c1] + StructIO v0.3.0
  [2efcf032] + SymbolicIndexingInterface v0.2.2
  [8290d209] + ThreadingUtilities v0.5.2
  [9f7883ad] + Tracker v0.2.30
  [a2a6695c] + TreeViews v0.3.0
  [d5829a12] + TriangularSolve v0.1.20
  [410a4b4d] + Tricks v0.1.8
  [3d5dd08c] + VectorizationBase v0.21.65
  [19fa3120] + VertexSafeGraphs v0.2.0
⌅ [7cc45869] + Enzyme_jll v0.0.89+0
        Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m`
Precompiling project...
  89 dependencies successfully precompiled in 192 seconds. 361 already precompiled.
Precompiling ComponentArraysZygoteExt
  6 dependencies successfully precompiled in 6 seconds. 183 already precompiled.
[ Info: Precompiling ComponentArraysZygoteExt [3acbf79e-e1c1-58d9-9a19-e1c5cc2c64d6]
Precompiling LuxZygoteExt
  1 dependency successfully precompiled in 2 seconds. 141 already precompiled.
Precompiling LuxComponentArraysExt
  ✓ Lux → LuxChainRulesExt
  4 dependencies successfully precompiled in 9 seconds. 154 already precompiled.
  1 dependency precompiled but a different version is currently loaded. Restart julia to access the new version
Precompiling ComponentArraysSciMLBaseExt
  2 dependencies successfully precompiled in 2 seconds. 183 already precompiled.
Precompiling LuxComponentArraysReverseDiffExt
  ✓ Lux → LuxChainRulesExt
  ✓ Lux → LuxComponentArraysExt
  ✓ Lux → LuxTrackerExt
  4 dependencies successfully precompiled in 8 seconds. 154 already precompiled.
  3 dependencies precompiled but different versions are currently loaded. Restart julia to access the new versions



[restart REPL]


  julia> using Zygote, ComponentArrays, Lux, SciMLSensitivity, Optimisers,
           OrdinaryDiffEq, Random, Statistics, OneHotArrays, InteractiveUtils
Precompiling LuxZygoteExt
  1 dependency successfully precompiled in 3 seconds. 141 already precompiled.
Precompiling LuxComponentArraysReverseDiffExt
  ✓ Lux → LuxChainRulesExt
  ✓ Lux → LuxComponentArraysExt
  ✓ Lux → LuxTrackerExt
  4 dependencies successfully precompiled in 7 seconds. 154 already precompiled.
  3 dependencies precompiled but different versions are currently loaded. Restart julia to access the new versions

The order in “using” seems to have no effect. But once again progress!

Maybe I need to reinstall lux?

Update:
Individually reinstalling Lux, ComponentArrays, and Zygote didn’t change anything. The tutorial runs the same as it did before, it just does that attempt to precompile every time now.

So I dunno, but there is definitely some dependency hell going on here that depends on the order of installation.

6

For some reason, this combination of packages seems to put quite a good amount of pressure on the dependency resolver, I think :wink:

Btw. do you have anything in your ~/.julia/config/startup.jl? Or in your global environment?

7

I don’t see any config directory, but I did use this one:
~/.julia/juliaup/julia-1.10.0-rc1+0.x64.linux.gnu/etc/julia/startup.jl

# This file should contain site-specific commands to be executed on Julia startup;
# Users may store their own personal commands in `~/.julia/config/startup.jl`.

cd("/JuliaProjects")

# Project Management
using DrWatson

# Editing/Debugging
using Revise, CodeTracking

# Terminal Output
using OhMyREPL, Term
install_term_repr()

Then this is the global environment (I think), which is just what gets loaded in that startup.jl:

~/.julia/environments/v1.10$ cat Project.toml
[deps]
CodeTracking = "da1fd8a2-8d9e-5ec2-8556-3022fb5608a2"
DrWatson = "634d3b9d-ee7a-5ddf-bec9-22491ea816e1"
OhMyREPL = "5fb14364-9ced-5910-84b2-373655c76a03"
Revise = "295af30f-e4ad-537b-8983-00126c2a3abe"
Term = "22787eb5-b846-44ae-b979-8e399b8463ab"

I literally just did a clean install of Mint 21 so I could get the latest ROCm for this purpose. So everything should be clean. And that startup.jl was the same last night when I somehow got it working correctly.

8

I left out ComponentArrays:

using Lux,  SciMLSensitivity, LuxAMDGPU,
       Optimisers, OrdinaryDiffEq, Random, Statistics, Zygote, OneHotArrays, InteractiveUtils
Precompiling LuxZygoteExt
  1 dependency successfully precompiled in 3 seconds. 143 already precompiled.

Restarted:


julia> using Lux,  SciMLSensitivity, LuxAMDGPU,
       Optimisers, OrdinaryDiffEq, Random, Statistics, Zygote, OneHotArrays, InteractiveUtils

julia> using ComponentArrays
Precompiling LuxComponentArraysReverseDiffExt
  ✓ Lux → LuxChainRulesExt
  ✓ Lux → LuxTrackerExt
  4 dependencies successfully precompiled in 10 seconds. 156 already precompiled.
  2 dependencies precompiled but different versions are currently loaded. Restart julia to access the new versions
[ Info: Precompiling LuxComponentArraysReverseDiffExt [b94e9d44-33ac-594b-a774-8f94d137cabd]

Now we are down to two bad dependencies. Restart:


using Lux,  SciMLSensitivity, LuxAMDGPU,
       Optimisers, OrdinaryDiffEq, Random, Statistics, Zygote, OneHotArrays, InteractiveUtils

Precompiling LuxZygoteExt
  1 dependency successfully precompiled in 3 seconds. 143 already precompiled.

Precompiling LuxZygoteExt came back… Restart again:


Julia> using Lux,  SciMLSensitivity, LuxAMDGPU,
       Optimisers, OrdinaryDiffEq, Random, Statistics, Zygote, OneHotArrays, InteractiveUtils

julia> using ComponentArrays

And now it is back to normal. So I managed to mcmc myself to a solution, but obviously it is a problem to have this depend on the order.

4 Likes
9

That might be worth opening a Github issue. Never used Lux, but I’d expect this precompilation to be settled when adding packages to the environment, and you don’t seem to be manually doing anything that would cause using to precompile, let alone for different versions than those loaded. It might be possible that some other packages in your environment is causing dependency hell, so if you make a fresh environment with only the packages in the Lux tutorial and can still reproduce these problems, you are fully justified in making an issue about the tutorial and package.

10

Please do the following and report the output. Invoking Pkg.precompile() will report more errors than ] precompile.

julia> using Pkg

julia> Pkg.precompile()
11

You mean remove all the packages and start over? Because right now it is fine:

julia> Pkg.precompile()
Precompiling project...
  13 dependencies successfully precompiled in 21 seconds. 563 already precompiled.

At what stage of the installation do you want this output?

12

It might be possible that some other packages in your environment is causing dependency hell, so if you make a fresh environment with only the packages in the Lux tutorial and can still reproduce these problems, you are fully justified in making an issue about the tutorial and package.

I made a new project and ignored the startup.jl. Then installed the packages without any errors. This is unlike in my other project. Since I was able to get it working, it seems there was some other order-dependent conflict as well.

  [b0b7db55] ComponentArrays v0.15.5
  [b2108857] Lux v0.5.10
  [83120cb1] LuxAMDGPU v0.2.0
  [d0bbae9a] LuxCUDA v0.3.1
  [0b1bfda6] OneHotArrays v0.2.4
  [3bd65402] Optimisers v0.3.1
  [1dea7af3] OrdinaryDiffEq v6.59.3
  [1ed8b502] SciMLSensitivity v7.47.0
  [e88e6eb3] Zygote v0.6.67

It still puts me in the “two dependency, restart julia” loop:

julia> using Lux, ComponentArrays, SciMLSensitivity, LuxAMDGPU, LuxCUDA,
       Optimisers, OrdinaryDiffEq, Random, Statistics, Zygote, OneHotArrays, InteractiveUtils
Precompiling SciMLSensitivity
  7 dependencies successfully precompiled in 132 seconds. 298 already precompiled.
Precompiling LinearSolveCUDAExt
  ✓ LinearSolve → LinearSolveKernelAbstractionsExt
  ✓ LinearSolve → LinearSolveRecursiveArrayToolsExt
  3 dependencies successfully precompiled in 10 seconds. 245 already precompiled.
  2 dependencies precompiled but different versions are currently loaded. Restart julia to access the new versions

Btw, if ]status shows empty project is that sufficient to show a fresh environment? Elsewhere it was mentioned maybe I need to modify LOAD_PATH:

julia> LOAD_PATH
3-element Vector{String}:
 "@"
 "@v#.#"
 "@stdlib"

This experience makes me think whatever I did to resolve the issue could be automated. Ie, if there is an error installing/precompiling, the package manager could try it in different orders.

13

100% file an issue with the tutorial’s package and link the tutorial the using statement is from.

No, the installation and import order shouldn’t matter, and making it matter would introduce a whole new hard-to-reproduce dimension to dependency hell. Imagine if updating a package or adding another one to the environment requires a change to the order, you’d have to re-precompile things with no guarantee the new order even works, and you’d have to manually edit all your import statements. You’re also assuming that the reordering fixed it because you’re not getting warnings; what if an unintended problem remains but it simply got masked or isn’t being caught? While there’s never a guarantee that an arbitrary combination of independently developed packages is compatible, a combination in a tutorial should be. Chances are the actual dozen packages updating to the v1.9 era didn’t quite get it right, and user feedback is valuable for keeping these up to standards.

1 Like
14

I thought you were still trying having trouble getting them to precompile. Ok, let’s see if this works.

There’s no need to remove any packages. We can just create a new temporary environment. Start a new Julia REPL session and then run this in the REPL.

using Pkg
pkg"activate --temp"
pkg"add Lux, ComponentArrays, SciMLSensitivity, LuxAMDGPU, LuxCUDA, Optimisers, OrdinaryDiffEq, Random, Statistics, Zygote, OneHotArrays, InteractiveUtils"
Pkg.precompile()
pkg"status"
pkg"status -m"
@time_imports using Lux, ComponentArrays, SciMLSensitivity, LuxAMDGPU, LuxCUDA,
Optimisers, OrdinaryDiffEq, Random, Statistics, Zygote, OneHotArrays, InteractiveUtils

Please report all output. Thank you.

15

Sure, I didn’t notice any issues doing it that way:

16

Are you sure I don’t need to modify that LOAD_PATH variable? It seems like if the packages were already precompiled by another project it could just load those.

17

Yes, I’m sure you do not need to change the LOAD_PATH. That affects where Julia looks for packages.

Julia packages are compiled in the ~/.julia/v1.9/ directory and they are assigned a hash depending on their version and the version of their dependencies.

If you wanted a fully independent test, you could try to set the JULIA_DEPOT_PATH environment variable. That will change your ~/.julia directory to another one. Julia would have to repopulate thay directory from scratch.

I’m still trying to decipher your workflow. It seems like you were relying on the using to both add packages to your environment and load them. This is rather inefficient. It it is better to add them all at once like I suggested above.

18

Thanks for the help. I am actually learning a lot about Julia from this issue.

My workflow was just to paste in the tutorial and see if it ran. I am just trying to get AMDGPU working at this point and needed an example.

But it appears using a temp directory was insufficient to isolate the problem. What I did is remove all those packages from my project then run ]gc --all. I saw this did delete them in ~/.julia/packages.

Then I ran your code using julia --startup-file=no and got all the original errors/etc:

19

After seeing the results in my last post do you think I should still open the github issue with Lux?

It seems to me this specific problem could be fixed but ultimately it is with how the package manager handles dependencies with extensions (correct me if I’m wrong, I just started trying Julia last month). As you say, depending on order of installation is descending into a whole new lower level of dependency hell.

20

I just realized there is another factor here that may matter. I am using juliaup, so at least the relevant directories may not be the same. Perhaps if using that there should be a note logged in versioninfo() or somewhere to help future debugging.