Precompilation takes ages

General Usage
1

Hey,

I am having some kind of an issue that does not match what I see here and what others reports.

I have an environment with the following Project.toml:

[deps]
BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
Colors = "5ae59095-9a9b-59fe-a467-6f913c188581"
Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
FiniteDifferences = "26cc04aa-876d-5657-8c51-4c34ba976000"
Format = "1fa38f19-a742-5d3f-a2b9-30dd87b9d5f8"
ForwardDiff = "f6369f11-7733-5829-9624-2563aa707210"
HypothesisTests = "09f84164-cd44-5f33-b23f-e6b0d136a0d5"
LaTeXStrings = "b964fa9f-0449-5b57-a5c2-d3ea65f4040f"
LaTeXTabulars = "266f59ce-6e72-579c-98bb-27b39b5c037e"
LineSearches = "d3d80556-e9d4-5f37-9878-2ab0fcc64255"
LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
Measurements = "eff96d63-e80a-5855-80a2-b1b0885c5ab7"
NLSolversBase = "d41bc354-129a-5804-8e4c-c37616107c6c"
Optim = "429524aa-4258-5aef-a3af-852621145aeb"
Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
PreallocationTools = "d236fae5-4411-538c-8e31-a6e3d9e00b46"
ProgressMeter = "92933f4c-e287-5a05-a399-4b506db050ca"
RDatasets = "ce6b1742-4840-55fa-b093-852dadbb1d8b"
Revise = "295af30f-e4ad-537b-8983-00126c2a3abe"
SpecialFunctions = "276daf66-3868-5448-9aa4-cd146d93841b"
StatsBase = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
StatsPlots = "f3b207a7-027a-5e70-b257-86293d7955fd"
UnicodePlots = "b8865327-cd53-5732-bb35-84acbb429228"

Nothing really fancyā€¦ However, precompilation takes ages :

Precompiling project...
  55 dependencies successfully precompiled in 362 seconds. 219 already precompiled.

Each time I change something (add a package and/or delete one, simply update packages) I have to pay this ? Or am I missing something here ?

Btw, Plots.jl is clearly the biggest issue by eye-bowling the precompilation output it accounts for at least 70%.

This is an issue since I end up paying this almost once or twice a day ā€“ good time for coffee of course, but still.

Am I doing something wrong, or is it just the way it is ?

1 Like
2

Can you provide more details? Maybe run versioninfo() in the Julia REPL and coypaste the results:

julia> versioninfo()
Julia Version 1.10.0
Commit 3120989f39b (2023-12-25 18:01 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 8 Ɨ Apple M2
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, apple-m1)
  Threads: 1 on 4 virtual cores
3

This shouldnā€™t be the case. As you see in the output 219 already precompiled, most changes should only incur an incremental precompilation cost. In my experience Plots is a bit of an issue here in that it has loads of dependencies so if you change something in the environment which affects one of its dependencies that will recompile Plots which as you say is probably the biggest driver - and having both Plots and StatsPlots doesnā€™t help (you probably donā€™t need that as StatsPlots reexports Plots).

EDIT scratch that last bit, StatsPlots would have Plots as a dependency anyway, so I guess whether they are both in the Project.toml or not doesnā€™t make a difference.

4

I guess you nailed the problem. It does not recompile everything each time, but it looks like every change i made (add a pakcgae, remove one, update another one, or anything related) instantly triggers a few other packages precompilation, but almost every time including Plots.jl, the most problematic oneā€¦

Is there a way i can tell Julia to NOT recompile Plots.jl ? Maybe by ā€œpinningā€ its version and its dependencies versions somehow ?

Better yet: is there a way I can tell it to re-do as little as possible ? Say I install a given new package, the default behavior i want is to chose the version of this new package to disrupt at least as possible the current environement state and thus trigger less precompilation. Say I update a given package, well same idea: keep compat bounds tight to avoid as much as possible work. Say I remove a package, well NOTHING should be updated. I would love such a ā€œconservativeā€ mode.

@Amval I am on 1.9.3, Iā€™ll give a versionInfo once my current run is finished. I thought of updating to 1.10, but for the very reason I am posting this today i did not ^^

Edit: done abit of reading around, and found out this thread

and then this PR :

Looks like others were having this kind of issues before me :slight_smile: Now i just have to understand how to use these new functionalities to solve my issueā€¦

5

The docs on pinning a package version. Of course you have to keep in mind that pinning that package also restricts other packages indirectly because of dependency compatibility. Pkg.add also has settings for version resolution but Iā€™ve nevered changed the default because the rest sound intimidating.

6 minutes does seem like a lot, over 4 of it is parts of Plots. It is a hefty package so maybe this isnā€™t actually unusual.

6

I also get Plots precompiled sometimes up to 10 times per day, but it usually takes less than 30 seconds for me, so 70% of 6 minutes still sounds like a lot

2 Likes
7

I am not sure pinning Plots.jl is enough: if one of its dependency is changed (within compat bounds) it will still precompile Plots.jl again right ?

I think what I want is PRESERVE_TIERED_INSTALLED. The docs says

To change the default strategy to PRESERVE_TIERED_INSTALLED set the env var JULIA_PKG_PRESERVE_TIERED_INSTALLED to true.

Do someone know how i can setup this environement variable globally on my machine (windows machine) ?

Indeed, maybe there is a reason mine takes so longā€¦ but i do not know which one. Might be related to the frequency of re-precompilation ? or to the number of stuff in the env ? dunno.

1 Like
8

Just freeze your Project.toml. Add PkgHelpers.jl to your global environment, and then launch Julia in your project environment, and then execute:

using Pkg, PkgHelpers
freeze(Pkg)

The next time you add a package only that package will get compiled.

2 Likes
9

I add

import Pkg
Pkg.UPDATED_REGISTRY_THIS_SESSION[] = true

to my ~/.julia/config/startup.jl so that it doesnā€™t update the whole registry every time I add a package (only if I explicitly run update).

1 Like
10

I think the only way to do this is building a sysimage with Plots in it.

EDIT that was supposed to be an answer to ā€œpin Plots so it doesnā€™t get updatedā€ but I failed at Discourse.

11

So I have several options thereā€¦

  1. @ufechner7 suggests adding to startup.jl:
using Pkg, PkgHelpers
freeze(Pkg)
  1. @stevengj suggests insterad adding
import Pkg
Pkg.UPDATED_REGISTRY_THIS_SESSION[] = true
  1. Pkgā€™s documentation suggests adding
ENV["JULIA_PKG_PRESERVE_TIERED_INSTALLED"] = true
  1. @nilshg suggests building a sysimage with Plots.jl in it.

I am not clear on the differents ups and downs of the four approachesā€¦

  • I dont thing option 4 is the way to go as there is not only Plots.jl but i want a more general behavior, and I do not want to freeze stuff, I want to be able to update packages if i ask for it (so option 3 is probably the way to go)
  • After reading PkgHelpers.jl docs it looks like freeze() will add stuff to compat entries of the project, so option 1 looks a bit invasive to my tasteā€¦
  • I am not understanding correctly what option 2 really does. Maybe 2+3 is the right call for me ?
1 Like
12

As I understand it 2 basically tells Julia not to update the general registry. This is often the source of ā€œunnecessaryā€ precompilation in the sense that some dependency of a dependency of a dependency of Plots releases some patch release, causing recompilation of the entire Plots stack.

13

Well, I never suggested to add this to startup.jl. Just execute this once.

14

yes i extrapolated sorry

15

Iā€™d recommend just doing 3 in your .julia/config/startup.jl

ENV["JULIA_PKG_PRESERVE_TIERED_INSTALLED"] = true

Iā€™ve been running with that since 1.9 came out and I havenā€™t seen any issues.

The reason itā€™s not the default is that some were concerned that it would mean users would generally get stuck on older package versions, but that hasnā€™t been a noticeable issue for me.

6 Likes