What does precompilation do on a package that does not use SnoopCompile at all?

General Usage
,
1

I’m dev’ing package does not use SnoopCompile or attempt any precompilation. I make frequent changes to the package locally, and everytime I do I suffer a decent precompilation delay (30 seconds) when using the package, e.g. whenever I run the formatter.

I’d previously asked about disabling precompilation on a package that I’m dev’ing: Recommended way of disabling precompilation for a package I'm dev'ing. The __precompile__(false) approach does work here: it brings the cost down to just the load time of the package, which is around 7 seconds. (I also tried the SnoopCompile Preferences approach mentioned here and excluded my package, but that didn’t seem to help, which probably makes sense given that my package does not use SnoopCompile.)

But here I just want to ask: what is this precompilation even doing, if I have no snoop compile calls? The only thing I can think of is that the dependencies are being precompiled, but that doesn’t make too much sense given that the precompilation time I’m suffering is when I edit my package, and only my package (which no other package depends on).

1 Like
2

Have you considered using Revise ?

This is the typical use case when developing to avoid restarting julia and hit precompilation whenever source file content change.

3

Right now I’m running into the load time issue when I’m trying to get a formatting script on my package to auto-run. Perhaps there are ways to fix that too, would love to hear about them, but this question is also just asked in curiousity about what actually contributes to the increased time when the package does not attempt to do precompilation.

4

Exactly.

The only thing I can think of is that the dependencies are being precompiled

Your package dependencies won’t be precompiled if they didn’t change (or if compile-time Preferences weren’t modified).

5

Right, agreed. So I’m still not sure what it means to be “precompiling” a package that does not use SnoopCompile, and why this takes a substantial amount of time.

2 Likes
6

All packages have their methods, types, global variables, etc. stored in a cache file. The source code for your package doesn’t get parsed when you say using MyPkg, instead we load the snapshot of the module stored in the cache file.

But you don’t have to be explicitly using SnoopPrecompile for these cache files to contain more than just methods, types, and global variables: if you run anything from top level, you’re also performing type inference and compilation, and depending on your Julia version those may also get saved to the cache file. SnoopPrecompile is just more thorough about “marking” things that should be stored in the cache.

2 Likes
7

That makes sense. My package does run code from the top level, which contributes to a load time of around 7 seconds. So precompilation of this code bumps it to 30 seconds, which makes sense.

Given that the SnoopPrecompile + preferences approach didn’t work, is there any planned way to disable this precompilation for specific packages without editing the source code of the package? (I also wonder if it could be any smarter about this; the whole precompilation process gets triggered with just a mere edit of a docstring)

8

Just to make sure we’re on the same page: running code at top level in your package code contributes to precompilation time, not load time. Because you’re just loading a snapshot. Here’s a very simple demo:

module Pkg1

foo(x) = 2x

sleep(30)

end

with results

julia> @time using Pkg1
[ Info: Precompiling Pkg1 [da3b0148-1b4b-4ae0-b07b-5f650a6bd540]
 30.673267 seconds (28.77 k allocations: 1.911 MiB, 0.09% compilation time)

julia>
tim@diva:/tmp/pkgsA/dev$ JULIA_DEPOT_PATH=/tmp/pkgsA julia -q
julia> @time using Pkg1
  0.007512 seconds (13.29 k allocations: 929.343 KiB)

So the precompilation time was 30s because we ran some code at top level, and that code required 30s to run. But loading the package was practically instantaneous, because that sleep doesn’t run when you load the package.

It’s quite easy to run code at top level and not realize it: heck, const x = [1, 2] runs Base.vect to build the vector from the literals.

For more complicated packages, there is some time simply to cache methods etc. But the most likely culprit probably remains the stuff that runs at toplevel. On 1.9 and higher, you can use PkgCacheInspector and MethodAnalysis to learn some of the things that got compiled while you were building the package. (Julia will discard code that it can’t prove will be useful for the package itself, and that’s what the “marking” in SnoopPrecompile is about. So even PkgCacheInspector will not necessarily be comprehensive.)

7 Likes
9

Related! Top level code in JuliaLang doesn't do what you think it does

1 Like