Can we have non-blocking precompilation? I think this would be great.
So, currently, when I type using MyPackage at the REPL and the cache is stale, I have to sit through precompilation before I can touch anything in MyPackage.*. But, precompilation only really pays off the next time I load the package. Julia can already load packages without caches at all (--compiled-modules=no), so in principle the REPL could give me the package immediately, with full JIT compilation as I go, while precompilation runs in a background process for next time.
Consider two workflows:
REPL. I want the fastest possible time-to-first-input for a newly installed package. I don’t care about eating some extra JIT compilation this session, and I would still benefit from whatever caches already exist. I just do not want to wait up front for caches that only help future me, when current me is just trying to run something.
Scripts. For a re-runnable script, using the caches matters more. But even then: if the cache is stale when the script starts, why must the whole precompile finish before line one runs? The script could run uncached now while precompilation happens in the background, and the next invocation gets the warm start.
I know Pkg already precompiles eagerly after Pkg.add/instantiate, which hides the wait for freshly installed packages (though I wish this was also non-blocking). The case I keep hitting is everything else: a package I just edited, a one-off install to do something, a changed Project.toml, a single changed dependency version. Then I have to wait on using.
Now, the obvious tradeoff is paying twice: the first session JIT-compiles some stuff that the background precompile is also compiling. I think it’s worth it though. Maybe the desire is for background jobs to not compete for cores, but I feel like the user’s focus time is more valuable.
So: is there a fundamental reason blocking precompilation is the default, or maybe just that nobody has built it yet? I think it would be pretty nice!
The immediate problem is there’s no firm line between the module you want immediately and the part you’re calling precompilation. Precompilation caches things during the evaluation of the module in a worker process unbiased by the current session’s state, then the cache might be loaded into the current session (next paragraph for more thoughts). There isn’t a module evaluation phase followed by a module precompile phase; we can obviously put function calls and precompile calls between methods (ideally the further changes to world age wouldn’t invalidate). PrecompileTools.jl isolates a precompile phase by only running its setup and workload blocks when precompilation occurs, but even that is strictly within the module block and can occur before other expressions. Note that doesn’t apply to the plain function calls and precompile calls that run regardless of precompilation. That could be addressed by offering PrecompileTools’ tricks as a feature, but people would have to manually opt into them and could have good reason not to.
Another problem is module inconsistency. We already observe this when a package and its dependencies are precompiled, but we get a warning that different versions of some dependencies are already loaded from another environment or a prior environment state. We either restart the session or put up with more precompilation we’d probably only use for that session’s weird state. You’re at least skipping the precompilation for the current session, but you’re embracing the same risk of the active module being different from the precompiled module loaded by the next session. Since you’re not waiting for precompilation, you don’t get that warning until after you do some work in the probably wrong environment, and it’s not clear where that warning could interject. This could be solved by getting the version warnings after the module is evaluated and before the worker processes make caches, then we could choose to call exit() and wait for the workers to finish before the session really ends.
Thanks, that’s helpful. Looking at this now. My thought for the first issue was to load the module normally from source in the foreground, while a separate standard precompile worker independently evaluates it and writes a cache only for a future session; the current session would simply never adopt that cache.
For the version-consistency warnings, would it make sense to capture the worker’s diagnostics and surface warnings asynchronously (or at exit), while remaining quiet on success? Or do you think waiting for workers at exit is necessary?
That’s more of the premise than addressing the first issue. To put it another way, the first issue is that evaluating the module itself can take a long time and PrecompileTools only isolates what it currently does. We’d need new features to push the rest to precompile-time (and ironically this would disproportionately involve the devs who intentionally shifted to lists of precompile statements to save the call execution time in PrecompileTools workloads), and that may not be feasible for some things e.g. conditional method definition that needs a very time-consuming check.
I’d want the warning right when I get the module so I can decide what to do with it. Figuring out the version clashes before making caches would be the time-saver. Waiting for workers at exit() is just my assumption that the session has to stay alive to not kill the workers; we do want those precompile caches next session.
Oh I see, you’re talking about making module evaluation non-blocking? This is different from what I meant: I only mean backgrounding precompile-cache generation. Evaluating the module would still happen normally in the foreground.
Not at all, the point is to get the module for the session, not for the session to skip it. I’m talking about skipping more than just the PrecompileTools workloads so we actually get the bare minimum module. If I provided a package with very time-consuming precompile calls and no PrecompileTools workloads to your proposed system, the current session’s module would take just as long to evaluate as the next session’s precompiled module. Longer to us, because the worker processes can at least work on dependencies in parallel.
Does --compiled-modules=no already skip the precompilation-only work you have in mind? If so, I guess I might expect --compiled-modules=background to use the same foreground-loading behavior while generating the cache separately in the background. Not sure though.