Julia 1.9.0-beta2 precompilation taking almost twice as long

I’ve been testing out the new Docker image for Julia 1.9-beta2. I do indeed notice the FFTX improvements, however I’m also noticing that precompilation of every package now takes on average between 1.5x-2x as long. versioninfo() yields:

Julia Version 1.9.0-beta2
Commit 7daffeecb8c (2022-12-29 07:45 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 12 × AMD Ryzen 5 5600G with Radeon Graphics
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, znver3)
  Threads: 1 on 12 virtual cores
Environment:
  JULIA_GPG = 3673DF529D9049477F76B37566E3C7DC03D6E495
  JULIA_PATH = /usr/local/julia
  JULIA_VERSION = 1.9.0-beta2

Tested against the latest version:

Julia Version 1.8.4
Commit 00177ebc4fc (2022-12-23 21:32 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 12 × AMD Ryzen 5 5600G with Radeon Graphics
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, znver3)
  Threads: 1 on 12 virtual cores
Environment:
  JULIA_GPG = 3673DF529D9049477F76B37566E3C7DC03D6E495
  JULIA_PATH = /usr/local/julia
  JULIA_VERSION = 1.8.4

I’m using Podman 4.3.1 to run the containers (podman run -it --rm julia:<latest or rc>).
I’ve compiled a list of results with some larger packages (I create a new container each time to avoid using any dependencies already precompiled):

Package Julia 1.8.4 (seconds) Julia 1.9.0-beta2 (seconds)
UnicodePlots 3.3.1 41 84
Enzyme 0.10.13 14 46
Unitful 1.12.2 15 32
Geostats 0.36.5 53 67
CSV 0.10.8 19 48
VegaLite 2.6.0 17 33
Turing 0.23.3 24 46

I haven’t seen any discussion about any precompilation regressions, is this expected for the beta?

1 Like

Yes: https://github.com/JuliaLang/julia/pull/47184#issuecomment-1283162301

Can anybody explain the reason? Are there any plans to reduce this time in the future?

To put it simple, to make this possible: Julia v1.9.0-beta2 is fast. Julia now compiles and stores more code during precompilation (a one-off) to save time at runtime for each single session.

7 Likes

For OrdinaryDiffEq, this precompilation time ratio is even worse: > 3.2, making it difficult to accept.

In a temporary env with an initial empty ~/.julia/compiled :

# on 1.8.4
julia> @time using OrdinaryDiffEq  # full deps stack precompilation + loading time
462.330531 seconds (19.18 M allocations: 1.423 GiB, 0.29% gc time, 0.18% compilation time: 17% of which was recompilation)
julia> @time Base.compilecache(Base.module_keys[OrdinaryDiffEq])  # ODE precompilation sequence only
179.162770 seconds (5.59 k allocations: 528.938 KiB, 0.00% compilation time)

# on 1.9.0.beta2
julia> @time using OrdinaryDiffEq
890.138045 seconds (10.74 M allocations: 818.022 MiB, 0.09% gc time, 0.04% compilation time)
julia> @time Base.compilecache(Base.module_keys[OrdinaryDiffEq])
579.490303 seconds (6.45 k allocations: 616.953 KiB)

EDIT: single core precompilation, on a quite old desktop Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz CPU.

1 Like

In an environment with only OrdinaryDiffEq and clean compiled/ directory each time, v1.8.3:

(tmp) pkg> precompile
Precompiling project...
  126 dependencies successfully precompiled in 125 seconds

master (commit de73c26fbf):

(tmp) pkg> precompile
Precompiling environment...
  112 dependencies successfully precompiled in 296 seconds

PR #47797 (commit 9647a58):

(tmp) pkg> precompile
Precompiling environment...
  112 dependencies successfully precompiled in 194 seconds

(In case anyone was wondering why there are different number of packages in the environments of v1.8 and v1.10, it’s because I had to re-resolve the environment to be able to fully precompile the environment, I was getting an error without re-resolving the environment, depsite the fact DelimitedFiles isn’t anywhere to be found in the Manifest.)

2 Likes

@giordano How many cpu cores do you have? I am afraid this doesn’t help much if you have only two of them…

I have 8. As far as I understand, PR #47797 uses half of the total cores for parallel image generation, so on a two-core machines that probably doesn’t help at all.

Going from Julia 1.8 to Julia 1.9.0-beta3, precompiling Pluto.jl (during Pkg.precompile()) went from 15 seconds to 60 seconds, pretty significant :worried:. We ran these tests many times, on different systems.

After precompilation, TTFX did improve a little, it went from 30 seconds to 20 seconds. (:tada:) This is fantastic, but it certainly comes with a trade-off.

My main concern is the UX of Pkg.precompile for new Julia/Pluto users. On an older laptop, folks might be waiting 2-3 minutes on precompilation of a single package, and decide that it must be stuck or broken.

8 Likes

Yes, UX can do a lot to to alleviate waiting.
(Though this has improved a lot over versions of Pkg).

Related, I suspect it might make Julia friendlier if it could tell when it was compiling sth during REPL usage (instead of silently hanging for a while)

Power users, used to JIT, can then turn off this feedback if they like

1 Like
4 Likes

That’s a great suggestion.

I was thinking here of JIT compilation instead of precompilation (admittedly a bit off-topic here)
e.g:

julia> plot([1, 3, 2]);
Compiling … 

(which, after a few seconds, becomes:

julia> plot([1, 3, 2]);
Compiling … ✔

Repeating the call prints no such thing, as there’s no more JIT compilation

i.e.
julia> plot([1, 3, 2]);
Compiling … ✔

julia> plot([1, 3, 2]);

julia> 
)
2 Likes

Strange idea ? … or maybe a good one. I think that may make JIT more palatable to a segment of users at least initially. Actually even for experienced users you could maybe toggle it on to get some information quickly.

5 Likes

I suspect it might do a lot for user retention.
Even on Julia 1.9, using Plots; display(scatter(rand(1_000))) still hangs for 7 seconds, much longer than it does in Python/R/Matlab.

(Source for timing: @nilshg’s great measurements here)

Having a spinner during those seven seconds, as stupid as it sounds, will make the wait more palatable indeed. And someone just trying out Julia will be less likely to immediately drop it again. (I have no hard data of course)

2 Likes

Yes thats my intuition as well. Of course it’s one thing to say that’s great, and another to code it. I don’t have time at the moment. But it’s good to air the idea in any case.

3 Likes

Is there any way to estimate [in a manner with palatably low error, maybe ±50%] the progress of compilation time left? Could be nice to have some idea whether I have to wait 7 seconds or 70—should I twiddle my thumbs or get a coffee.

That’s a pretty large increase in TTFNotebook if measured from the installation/update of Pluto!
I haven’t tried 1.9 yet and not sure how is the typical situation with other packages - how TTFX changes 1.8 → 1.9 if precompilation is included?

Seems like a major degradation for notebooks experience in general. In Pluto, each notebook you create is a separate environment, and package versions are likely to be slightly different between them. So, running a notebook from scratch involves (at least partial) precompilation.

1 Like

After reading @fons’ post above, I was terrified. All my teaching is done using Pluto notebooks, and the entire enterprise would be killed if notebooks took more than one or two minutes to precompile. Students usually are a bit impatient when waiting for the notebook to run.

I tested the latest version of Pluto v.0.19.20 (which came out yesterday) with Julia v.1.8.5 and v.1.9.0-beta3.

My system is as follows:

OS: Windows (x86_64-w64-mingw32)
CPU: 8 × Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-13.0.1 (ORCJIT, skylake)
Threads: 4 on 8 virtual cores

and the notebook uses the following packages

PlotlyBase, HypertextLiteral, PlutoUI, ColorSchemes
CSV, DataFrames, Dates, PeriodicalDates, NLsolve, StatsBase

The results of various trials are as follows:

Precompile Pluto:

  • Version 1.9.0-beta3 is 3x faster than v.1.8.5 (≈11s vs ≈33s).

Running the notebook:

  • Version 1.9.0-beta3 is ≈23% faster than v.1.8.5 (≈54s vs ≈70s).

This increase in FTTX is terrific. So, either I am doing something wrong in my trails, or @fons have played a fantastic trick in the latest version of Pluto.

Did you delete your .julia folder before measuring the time needed for pre-compilation?

Please refrain from doing such dangerous suggestions again.

7 Likes