PSA for SnoopPrecompile: turning off extra workload for specific packages

General Usage
1

EDIT: SnoopPrecompile has been replaced by PrecompileTools. It also has a different mechanism for disabling precompile workloads, see its documentation.

The latest version of SnoopPrecompile has added the ability to reduce your compilation costs for specific packages:

using SnoopPrecompile, Preferences
set_preferences!(SnoopPrecompile, "skip_precompile" => ["PackageA", "PackageB"])

If you’re a package developer, you might find it helpful to include the main package(s) you’re working on in the list of skip_precompiles.

I know that a number of packages have added support for various ad-hoc ENV variables to accomplish the same thing. I’d recommend removing that support and using the Preferences-based mechanism now supported by SnoopPrecompile. Why? Because with ENV variables it’s possible for your package cache to get into an inconsistent state (Unable to precompile Plots, Precompilation segfault after SnoopPrecompile was updated from v1.0.1 to v1.0.3 · Issue #2585 · MakieOrg/Makie.jl · GitHub), and the Preferences system is designed to prevent that from happening.

The biggest disadvantage of the Preferences system is that if you change the packages you list in skip_precompile, you will likely have to re-precompile every package that depends (directly or indirectly) on SnoopPrecompile, and we are rapidly approaching the state where that’s pretty much the entire Julia ecosystem. As painful as that sounds, that is the main point of using Preferences: if you change how a low-level package is precompiled, every package that uses it needs to be recompiled and linked properly against the new version. It’s a bit unfortunate that even unrelated packages that happen to use SnoopPrecompile will also be re-precompiled, but better safe than sorry. And, of course, we welcome ideas (and especially pull requests :wink:) for how to make this better.

17 Likes
2

Many thanks to @t-bltg for implementing this important feature!

6 Likes
3

This is awesome! How does the Preferences system interact with custom sysimages? Is something expected to break if I mix the two?

4

Compile-time preferences will be “locked” in a custom sysimage.

5

Many of you have probably noticed this already, but

does not really seem to be a problem in practice. The reason? The best time to set this preference is when you’re in the package’s environment (because you’re developing it), and in that case LocalPreferences.toml will be just that: local to that environment. This gives you the best of both worlds:

  • when you’re developing the project (julia --project or Pkg.activate the package’s environment, or select it in VSCode), you’re not running the extra precompile workloads
  • when you’re using the package externally, you do run the precompile workload. This is likely what you want: once you stop developing the project, it’s more likely to be worth investing in extensive precompilation. Of course you can also turn it off in external environments, too, and in cases where you frequently “co-develop” certain package combinations it might be wise to set up a project environment that disables precompilation for all of them. But again, it will take effect only when you’re in that environment.

This is a great example of where the locality of the Preferences system and Julia’s environments really pays handsomely. I’ve found this split to be a huge win for my enjoyment of package development.

Another tip: consider putting LocalPreferences.toml in your global .gitignore. That will prevent you from accidentally checking it in.

2 Likes