Why does precompilation take so long for this no-op module?

General Usage
1

Suppose I have a basically empty package which depends on a “heavy” (i.e. slow to precompile/load) package. E.g.

module Foo
using Zygote
end

Why does precompiling my empty package (Julia 1.4.2) take significantly longer than precompiling the single heavy package?

# time to precompile and load Zygote
julia> @time using Zygote
[ Info: Precompiling Zygote [e88e6eb3-aa80-5325-afca-941959d7151f]
 24.688621 seconds (28.59 M allocations: 1.660 GiB, 1.91% gc time)

# time to just load Zygote
julia> @time using Zygote
 10.273923 seconds (27.57 M allocations: 1.613 GiB, 4.69% gc time)

# time to precompile and load Foo when Zygote also needs to be precompiled
julia> @time using Foo
[ Info: Precompiling Foo [9717d8fb-fffd-47ec-8476-db8e9679d15e]
 36.702254 seconds (37.17 M allocations: 2.075 GiB, 1.65% gc time)

In all cases here I have ensured all of Zygote’s dependencies are precompiled, so the timing should reflect just Zygote and Foo.

So shouldn’t precompiling Foo when Zygote also needs to be precompiled (the last case) not take any longer than the original 24 seconds?

This kind of thing comes up in my workflow alot since I often structure my analyses with modules that depend on each other like GenericAnalysisPackage ← MoreSpecificAnalysisPackage ← EvenMoreSpecificAnalysisPackage. When everything is precompiled this is great, but changing GenericAnalysisPackage triggers what feels like a needlessly slow recompilation.

Is this something simple which can be fixed, or is this secretly a more complex problem?

5 Likes
2

Try the 1.5 branch, these things have improved a lot.

3

1.5 seems a bit faster overall which is great, but still has the same problem that precompiling Foo takes longer than precompiling its dependency. Here’s with some slightly more reproducible (bash) code:

zygote=$(julia -e "using Zygote,Foo; println(pathof(Zygote))")

# time to precompile and load Zygote
touch $zygote
julia -e "@time using Zygote"
 22.005036 seconds (24.60 M allocations: 1.395 GiB, 1.08% gc time)

# time to just load Zygote
julia -e "@time using Zygote"
  7.943963 seconds (23.90 M allocations: 1.361 GiB, 0.09% gc time)

# time to precompile and load Foo when Zygote also needs to be precompiled
touch $zygote
julia -e "@time using Foo"
 27.318374 seconds (24.60 M allocations: 1.395 GiB, 0.90% gc time)
6 Likes