Can we have non-blocking precompilation? I think this would be great.
So, currently, when I type using MyPackage at the REPL and the cache is stale, I have to sit through precompilation before I can touch anything in MyPackage.*. But, precompilation only really pays off the next time I load the package. Julia can already load packages without caches at all (--compiled-modules=no), so in principle the REPL could give me the package immediately, with full JIT compilation as I go, while precompilation runs in a background process for next time.
Consider two workflows:
REPL. I want the fastest possible time-to-first-input for a newly installed package. I donāt care about eating some extra JIT compilation this session, and I would still benefit from whatever caches already exist. I just do not want to wait up front for caches that only help future me, when current me is just trying to run something.
Scripts. For a re-runnable script, using the caches matters more. But even then: if the cache is stale when the script starts, why must the whole precompile finish before line one runs? The script could run uncached now while precompilation happens in the background, and the next invocation gets the warm start.
I know Pkg already precompiles eagerly after Pkg.add/instantiate, which hides the wait for freshly installed packages (though I wish this was also non-blocking). The case I keep hitting is everything else: a package I just edited, a one-off install to do something, a changed Project.toml, a single changed dependency version. Then I have to wait on using.
Now, the obvious tradeoff is paying twice: the first session JIT-compiles some stuff that the background precompile is also compiling. I think itās worth it though. Maybe the desire is for background jobs to not compete for cores, but I feel like the userās focus time is more valuable.
So: is there a fundamental reason blocking precompilation is the default, or maybe just that nobody has built it yet? I think it would be pretty nice!
The immediate problem is thereās no firm line between the module you want immediately and the part youāre calling precompilation. Precompilation caches things during the evaluation of the module in a worker process unbiased by the current sessionās state, then the cache might be loaded into the current session (next paragraph for more thoughts). There isnāt a module evaluation phase followed by a module precompile phase; we can obviously put function calls and precompile calls between methods (ideally the further changes to world age wouldnāt invalidate). PrecompileTools.jl isolates a precompile phase by only running its setup and workload blocks when precompilation occurs, but even that is strictly within the module block and can occur before other expressions. Note that doesnāt apply to the plain function calls and precompile calls that run regardless of precompilation. That could be addressed by offering PrecompileToolsā tricks as a feature, but people would have to manually opt into them and could have good reason not to.
Another problem is module inconsistency. We already observe this when a package and its dependencies are precompiled, but we get a warning that different versions of some dependencies are already loaded from another environment or a prior environment state. We either restart the session or put up with more precompilation weād probably only use for that sessionās weird state. Youāre at least skipping the precompilation for the current session, but youāre embracing the same risk of the active module being different from the precompiled module loaded by the next session. Since youāre not waiting for precompilation, you donāt get that warning until after you do some work in the probably wrong environment, and itās not clear where that warning could interject. This could be solved by getting the version warnings after the module is evaluated and before the worker processes make caches, then we could choose to call exit() and wait for the workers to finish before the session really ends.
Thanks, thatās helpful. Looking at this now. My thought for the first issue was to load the module normally from source in the foreground, while a separate standard precompile worker independently evaluates it and writes a cache only for a future session; the current session would simply never adopt that cache.
For the version-consistency warnings, would it make sense to capture the workerās diagnostics and surface warnings asynchronously (or at exit), while remaining quiet on success? Or do you think waiting for workers at exit is necessary?
Thatās more of the premise than addressing the first issue. To put it another way, the first issue is that evaluating the module itself can take a long time and PrecompileTools only isolates what it currently does. Weād need new features to push the rest to precompile-time (and ironically this would disproportionately involve the devs who intentionally shifted to lists of precompile statements to save the call execution time in PrecompileTools workloads), and that may not be feasible for some things e.g. conditional method definition that needs a very time-consuming check.
Iād want the warning right when I get the module so I can decide what to do with it. Figuring out the version clashes before making caches would be the time-saver. Waiting for workers at exit() is just my assumption that the session has to stay alive to not kill the workers; we do want those precompile caches next session.
Oh I see, youāre talking about making module evaluation non-blocking? This is different from what I meant: I only mean backgrounding precompile-cache generation. Evaluating the module would still happen normally in the foreground.
Not at all, the point is to get the module for the session, not for the session to skip it. Iām talking about skipping more than just the PrecompileTools workloads so we actually get the bare minimum module. If I provided a package with very time-consuming precompile calls and no PrecompileTools workloads to your proposed system, the current sessionās module would take just as long to evaluate as the next sessionās precompiled module. Longer to us, because the worker processes can at least work on dependencies in parallel.
Does --compiled-modules=no already skip the precompilation-only work you have in mind? If so, I guess I might expect --compiled-modules=background to use the same foreground-loading behavior while generating the cache separately in the background. Not sure though.
But itās presumably highly experimental for now, hence include non-blocking precompilation in a future Julia release, but disabled by default in the REPL.
The parallel question is, should it be the default for agentic sessions? The broader discussion of this is for a separate thread, but the work described here is very timely and responsive to some of the challenges noted in this cautionary story from the Haskell world, in particular the compile feedback cycle time. We should have this additional context in mind as we add improvements such as this.
No, because thereās nothing that specifies that work to only occur at precompile-time. --compiled-modules=background canāt do anything about that either, it just sounds a lot more useful so itās worth bringing up the unskippable āprecompilationā (maybe this should just be called module evaluation-time compilation).
This would require another feature that developers of the package and all its dependencies need to opt into. Now I think about it, throwing precompile calls into @compile_workload blocks might actually work, but a different option is needed for direct top-level calls to only cache package-owned signatures.
I hope not, Iād typically rather wait upfront and take a break than to spread TTFX across separate manual calls. I might want a uncached package if I was developing it across sessions, but I could also just omit its precompile files and may want to precompile its mostly unchanged dependencies (--compiled-modules=existing has a similar purpose, but it still runs unskippable precompilation and wonāt precompile dependencies). Could be nice to have a manual @futurecache using Package.
At the risk of wading into a subjective defaults war (fun!), even though the PR makes it non-blocking precompilation opt-in (please note!), I also personally think it would be more ergonomic to have the behavior be opt-out at some point in the future.
Then, if someone did in fact want to wait, they could run Pkg.precompile() to get it immediately, or Base.Precompilation.monitor_background_precompile() to wait on the background one (exists from the Pkg feature).
(And I know the canonical local fix will be something like ājust set these three 20-character environment variables to 0, sacrifice a goat, and Julia will maybe sometimes skip the century-long precompilation of the complete transitive dependency graph of human civilization because of a changed docstring, all so you can make a log-log plot for a lecture.āā¦ but I am not a fan).
