Unfortunately, I think you’re ignoring the implicit dependencies of Plots.jl here. Recompilation isn’t just triggered based on whether the version of Plots itself is different but also if the version of one of its dependencies is different. And the latter can easily happen in different environments due to other (different) packages in these environments. That’s also why the OP is exploring options to already tell Julia on ] add that it should (try to) resolve in such a way, that the entire dependency tree is preserved.
6 Likes
Do we have a way to obtain more diagnostic information here?
Could we use --trace-compile=logfile to figure out if there a particular system configuration where compilation would become particularly slow?
Somewhat related: I seem to remember a blog post from one of the Julia devs that showed a breakdown of time spent (type inferencing, code generation, etc). But I cannot seem to find it again anywhere. Did I mis-remember this one?
I’ll admit that I’ve never really thought too much about precompilation - I frequently find it surprising when some unrelated package (mostly Plots!) gets precompiled again after some Pkg operation, but I have a beefier laptop so don’t mind the occasional 1-2 minute precompilation breaks.
Therefore I’m not sure whether the following suggestion actually works, but short of compiling a system image, could you just install Plots into the main environment and not change that? As environments are stacked, this allows you to use Plots when working in other project environments, and it shouldn’t be affected by changes to those environments. If you actively use your default environment and frequently change it, maybe you could have a separate Plots environment which you add to the LOAD_PATH with only Plots in which you never (or as you say very infrequently) update?
4 Likes
This sounds like a much better version of the hack I implemented here.
It still has the issue that known incompatible versions of dependents of Plots and dependents of whatever other environment I’m working on might be used, but for now I’m willing to accept that issue until I start running into actual bugs from it.
That sounds rather dangerous. It’ll silently use the wrong versions of packages right? So the reproducibility (not even speaking of the correctness) of the plots might go out of the window without you even realizing?
If tiered is the default (TIL), you’re right, it would just cause add to error. I think that perhaps your best bet is to increase MAX_NUM_PRECOMPILE_FILES. That way you can re-use the precompiled results more often and decrease your precompile times.
I would agree with others that the precompile time is suspiciously long. There are a few packages like OrdinaryDiffEq that take a very long time, but most of them do not.
@t-bltg just contributed a very nice change to SnoopPrecompile (not yet released, but you can check out #master) that will let you turn off the extra SnoopPrecompile workload for select packages.
2 Likes
That comes down to LoopVectorization.jl. I would suspect that might be a common thread. Whether it’s just because that’s the amount of code it needs, or if it’s generating too much code it doesn’t need, is still up in the air.
There is a practically infinite set of possible package combinations and I suspect that most of my reprecompilation time is spent on novel (to my machine) combinations so even with infinite storage this would not address most of the problem.
I explained most of the difference between my results and others’ here and @Oscar_Smith explained what is likely the rest of the difference here. “Get a faster computer and close more background processes” is not a satisfactory solution for me, especially in this venue. I came here—to the Julia community—in hopes of having Julia take fewer cycles to reprecompile, not in hopes of allocating more cycles per second to Julia.
1 Like
Another for your list is anti-malware scanners. The ones my employer’s IT department imposes are eating at least one core all the time.
Probably not related, but scanning the GMT dll (~62 MB and ~95 sec compile time) with a binary editor I see lots of blocks with zero bytes. One of them has a ~2.5 MB block filled only with zeros.
1 Like
I wasn’t suggesting that. Anyway, you and others are doing an excellent job of figuring out what’s causing it. I’ll shut up, carry on.
1 Like
I know, but thanks for saying it anyway!
My current (unsafe) solution is to add this to my startup.jl
# https://discourse.julialang.org/t/10-15-minute-ttfp-with-plots-jl-please-help/92636/53?u=lilith
if isinteractive()
import Pkg;
function _add(pkg::AbstractString)
Pkg.activate(pkg, shared=true); Pkg.add(pkg)
open(joinpath(first(DEPOT_PATH), "config", "startup.jl"), "a") do io
println(io, "\nisinteractive() && push!(LOAD_PATH, \"@$pkg\")")
end
end
function _up(pkg::AbstractString)
Pkg.activate(pkg, shared=true); Pkg.up()
end
end
and run
_add("Plots")
once from any REPL. The isinteractive() branch is optional and only works in Julia 1.8+.
This is a combination of my idea to use a designated Plots environment and ignore select compat entries and @nilshg’s idea to use load paths
One idea is to use start Jula with julia --pkgimages=no. This turns off object-caching which made precompilation more costly. The trade-off is that you don’t get to benefit from the object caching work, but if you often do single use pkg environments it might save time.
2 Likes
I was thinking in keeping Julia 1.7 for that purpose (1.8 has too many issues) but that is definitely a need for package developers (debugging for instances forces permanently Julia restarts).
Instead of being on an old Julia version, it seems easier to me to add an alias with the command line flag instead.