# ANN: New package SnoopPrecompile

**URL:** <https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778>\
**Category:** Community\
**Tags:** package, ttfx, latency\
**Created:** [July 25, 2022, 9:40pm UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778 "2022-07-25T21:40:56Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [July 25, 2022, 9:40pm UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/1 "2022-07-25T21:40:56Z")

</div>

EDIT: **SnoopPrecompile has been replaced by [PrecompileTools](https://github.com/JuliaLang/PrecompileTools.jl).**

I’m pleased to announce the availability of [SnoopPrecompile](https://github.com/timholy/SnoopCompile.jl/tree/master/SnoopPrecompile), a new package that can help reduce latency particularly on Julia 1.8 and higher. The concept of the package is to make precompilation easy. As described in greater detail in the [documentation](https://timholy.github.io/SnoopCompile.jl/dev/snoop_pc/), you can use it like this:

```julia
module MyPackage

using SnoopPrecompile

# define methods, types, etc

@precompile_all_calls begin
    # In here put "toy workloads" that exercise the code you want to precompile
    some_workload_you_want_to_make_faster(args...)
end

end # MyPackage

```

`@precompile_all_calls` does a small amount of work for you to help ensure that (particularly on Julia 1.8 and higher) all the needed support calls get precompiled. This style of precompilation is appropriate if `some_workload_you_want_to_make_faster` can be run safely on its arguments without bad side effects; `delete_my_whole_harddrive()` would not be a recommended candidate for this style of precompilation 🙂 .

Improving Julia’s precompilation is an ongoing adventure; 1.8rc3 has some [known issues](https://github.com/JuliaLang/julia/pull/46010) but still should be better, for most users, than any previous Julia version. We expect additional progress in the area of precompilation in future versions of Julia. For those who want more detail, there is a JuliaCon talk on the topic on Thursday morning.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [July 26, 2022, 12:53am UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/2 "2022-07-26T00:53:04Z")

</div>

What’s the difference between this and just putting a `let` block of example code into a package that is run on `using` time?

---

<div class="post-metadata">

**Author:** ![Alec\_Loudenback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alec_loudenback/32/278_2.png) [@Alec\_Loudenback](https://discourse.julialang.org/u/Alec_Loudenback)\
**Post date:** [July 26, 2022, 1:16am UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/3 "2022-07-26T01:16:11Z")

</div>

What happens on 1.7 if you include the code snippets above? It just doesn’t cache(?) the precompilation items whereas 1.8 has that capability?

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [July 29, 2022, 12:06pm UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/4 "2022-07-29T12:06:57Z")

</div>

It does 3 things:

1. run the block only when precompilling (it incorporates `if ccall(:jl_generating_output, Cint, ()) == 1`) so that if you’re running with `--compiled-modules=no` you don’t waste time
2. disables the interpreter when running the block (to ensure everything gets compiled)
3. intercept runtime dispatch to force precompilation of calls to methods defined in other packages

Item 3 is the really new thing. To explain in detail, any call made by runtime dispatch will break the chain of backedges. If those backedges don’t link back to the currently-precompiling package, the type-inferred code won’t be cached in the precompile file. Thus if the callee is a method in your package, no sweat (it’s already in your package so will be cached), but if the (runtime-dispatched) callee is in `Base` or elsewhere it won’t get precompiled by default on 1.8. `@precompile_all_calls` fixes this by snooping on inference and recording all new entrances to inference; once the `@precompile_all_calls` block exits it just iterates through the list of all newly-inferred MethodInstances and generates a manual `precompile(f, (argtypes...))` command (which on 1.8 does force caching even if the method is external).

None of this is a concern for inferrable dispatch, because on Julia 1.8 all inferrable dispatch gets cached regardless of module ownership. This is really just to scoop up those runtime-dispatched dependencies.

This really eliminates the need to ever statically generate those `precompile` directives, unless your workload has undesirable side-effects.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [July 29, 2022, 12:07pm UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/5 "2022-07-29T12:07:29Z")

</div>

It shouldn’t do anything bad on earlier versions, but the main benefits will arrive only on 1.8 and higher.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [July 29, 2022, 12:36pm UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/6 "2022-07-29T12:36:19Z")

</div>

> [@tim.holy](#):
>
> Item 3 is the really new thing. To explain in detail, any call made by runtime dispatch will break the chain of backedges. If those backedges don’t link back to the currently-precompiling package, the type-inferred code won’t be cached in the precompile file. Thus if the callee is a method in your package, no sweat (it’s already in your package so will be cached), but if the (runtime-dispatched) callee is in `Base` or elsewhere it won’t get precompiled by default on 1.8. `@precompile_all_calls` fixes this by snooping on inference and recording all new entrances to inference; once the `@precompile_all_calls` block exits it just iterates through the list of all newly-inferred MethodInstances and generates a manual `precompile(f, (argtypes...))` command (which on 1.8 does force caching even if the method is external).

That fixes the issues with LoopVectorization then!

Thanks, will use it right away 😅

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [July 29, 2022, 1:12pm UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/7 "2022-07-29T13:12:46Z")

</div>

LoopVectorization has some invalidation challenges, but I’m about to start looking into that in detail.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [July 29, 2022, 1:17pm UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/8 "2022-07-29T13:17:24Z")

</div>

Its biggest issue was that any instability before LV would cause all precompilation of LV to be removed, which is about 17 seconds of extra run time. So making that a lot more predictably precompiled is a very huge change, even if LV doesn’t precompile fully yet.

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [July 29, 2022, 2:31pm UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/9 "2022-07-29T14:31:02Z")

</div>

I’m gonna leave in vacation tomorrow and won’t be able to follow this closely. But if I uncomment

> <https://github.com/GenericMappingTools/GMT.jl/blob/master/src/GMT.jl#L316>

(plus line#6 and the `Project.toml`)

then I get a segmentation fault (Windows)

```julia
julia> cd("C:/v"); @time using GMT
[Info: Precompiling GMT [5752ebe1-31b9-557e-87aa-f909b540aa54]

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ffb0229d596 -- splitpath at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)
in expression starting at C:\Users\joaqu\.julia\dev\GMT\src\GMT.jl:316
splitpath at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)
Allocations: 83598528 (Pool: 83552476; Big: 46052); GC: 77
ERROR: Failed to precompile GMT [5752ebe1-31b9-557e-87aa-f909b540aa54] to C:\Users\joaqu\.julia\compiled\v1.7\GMT\jl_D31D.tmp.
Stacktrace:
 [1] error(s::String)
   @ Base .\error.jl:33
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
   @ Base .\loading.jl:1466
 [3] compilecache(pkg::Base.PkgId, path::String)
   @ Base .\loading.jl:1410
 [4] _require(pkg::Base.PkgId)
   @ Base .\loading.jl:1120
 [5] require(uuidkey::Base.PkgId)
   @ Base .\loading.jl:1013
 [6] require(into::Module, mod::Symbol)
   @ Base .\loading.jl:997
 [7] top-level scope
   @ timing.jl:220

```

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [July 29, 2022, 9:06pm UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/10 "2022-07-29T21:06:07Z")

</div>

> [@tim.holy](#):
>
> LoopVectorization has some invalidation challenges

```julia
julia> using TriangularSolve, LinearAlgebra, MKL;

julia> BLAS.set_num_threads(1)

julia> BLAS.get_config().loaded_libs
1-element Vector{LinearAlgebra.BLAS.LBTLibraryInfo}:
 LBTLibraryInfo(libmkl_rt.so, ilp64)

julia> N = 100;

julia> A = rand(N,N); B = rand(N,N); C = similar(A);

julia> @time TriangularSolve.rdiv!(C, A, UpperTriangular(B));
  2.799829 seconds (1.21 M allocations: 56.681 MiB, 98.19% compilation time: 97% of which was recompilation)

```

98.19% compile time, of which 97% was recompilation.

Checking `@snoopr using TriangularSolve`, there are a lot of invalidations.  
Many of them still involve `Static.jl`.

My own focus will be on continuing to replace LoopVectorization in a way that allows us to avoid needing to add a ton of types that trigger further invalidations.

Although, many of these are easily fixable and thus could be worth spending time on, e.g. :

```julia
 inserting ifelse(u::Bool, v1::VectorizationBase.Double, v2::VectorizationBase.Double)
     @ VectorizationBase ~/.julia/packages/VectorizationBase/nsLCg/src/special/double.jl:108 invalidated:
   backedges: 1: superseding ifelse(condition::Bool, x, y)
     @ Base essentials.jl:565 with MethodInstance for ifelse(::Bool, ::Any, ::Any) (4 children)

```

I should definitely stop using `CPUSummary.num_threads()` (and use `Threads.nthreads()` instead).

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [July 29, 2022, 10:16pm UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/11 "2022-07-29T22:16:11Z")

</div>

> [@Elrod](#):
>
> `@ Base essentials.jl:565 with MethodInstance for ifelse(::Bool, ::Any, ::Any) (4 children)`

Ouch, could we please not specialize on that one in Base?

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [July 29, 2022, 11:24pm UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/12 "2022-07-29T23:24:15Z")

</div>

that’s the main method of ifelse. what else would it be?

---

<div class="post-metadata">

**Author:** ![Zach\_Christensen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zach_christensen/32/7220_2.png) [@Zach\_Christensen](https://discourse.julialang.org/u/Zach_Christensen)\
**Post date:** [July 29, 2022, 11:38pm UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/13 "2022-07-29T23:38:39Z")

</div>

There are some cases where inference fails to determine that a Bool is passed to ifelse.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [July 30, 2022, 1:11pm UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/14 "2022-07-30T13:11:22Z")

</div>

Dunno, is that reproducible? The segfault is in a windows system library, seems very unlikely to be due to this package.

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [July 30, 2022, 1:48pm UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/15 "2022-07-30T13:48:42Z")

</div>

> [@joa-quim](#):
>
> ```julia
> #@precompile_all_calls begin
> #plot(rand(5,2))
> #makecpt(T=(0,10))
> #grdimage(rand(Float32,32,32))
> #end
> 
> ```

This code looks like it needs to go in ` __init__ `. Right now it’s at the top level and is probably being executed before ` __init__ `.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [August 1, 2022, 3:14pm UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/16 "2022-08-01T15:14:41Z")

</div>

That’s a perceptive comment. I hadn’t thought about putting the precompiles in the ` __init__ ` but it seems like it should work. In such cases `@precompile_setup` will have no impact but `@precompile_all_calls` should work as intended. (It’s a bit subtle because ` __init__ ` will be compiled before it runs, but since it’s a method in your package all the backedges should be fine for anything that’s not runtime-dispatched, and once the snooping is on it should pick up the runtime-dispatched.)

---

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [August 3, 2022, 7:38pm UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/17 "2022-08-03T19:38:52Z")

</div>

I’ve been messing around with this and getting some pretty awesome results 🙏 However, one place it doesn’t seem to do much is for precompiling Zygote gradients. It doesn’t seem to improve TTFG (time to first gradient) by much at all. For some random thing I was just trying e.g. it goes from 13s → 11s, whereas for some of my other non-gradient code I’m getting like 15s → 2s. Is this a known limitation or is there anything to do about this? (This is with 1.8 btw)

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [August 3, 2022, 10:44pm UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/18 "2022-08-03T22:44:23Z")

</div>

> [@marius311](#):
>
> Is this a known limitation or is there anything to do about this? (This is with 1.8 btw)

This is most likely because Zygote gradients use so many closures, and closures are a different type per session so you cannot precompile them. If those closures were turned into callable structs everywhere in ChainRules.jl, then we’d probably be able to precompile a lot more (@chriselrod we should add this as another reason in SciMLStyle to not use closures 😅). I don’t know how the ChainRules.jl devs would feel about that kind of style change though: it would be drastic, but it could also improve error messages.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [August 3, 2022, 11:28pm UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/19 "2022-08-03T23:28:44Z")

</div>

Brought up here: [Remove closures for callable types · Issue #657 · JuliaDiff/ChainRules.jl · GitHub](https://github.com/JuliaDiff/ChainRules.jl/issues/657) We’ll see if we go through with it.

---

<div class="post-metadata">

**Author:** ![Zach\_Christensen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zach_christensen/32/7220_2.png) [@Zach\_Christensen](https://discourse.julialang.org/u/Zach_Christensen)\
**Post date:** [August 3, 2022, 11:35pm UTC](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778/20 "2022-08-03T23:35:50Z")

</div>

I’ve found a few cases where inference failed on v1.6 and I had to `ntuple(i-> foo(x, i), Val{N}())` instead of `ntuple(Base.Fix1(foo, x), Val{N}())` to overcome this.

[Next page](https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778.md?page=2)