# Can I add multiple package's SnoopCompile precompile scripts to one?

**URL:** <https://discourse.julialang.org/t/can-i-add-multiple-packages-snoopcompile-precompile-scripts-to-one/40832>\
**Category:** General Usage\
**Created:** [June 5, 2020, 7:44pm UTC](https://discourse.julialang.org/t/can-i-add-multiple-packages-snoopcompile-precompile-scripts-to-one/40832 "2020-06-05T19:44:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [June 5, 2020, 7:44pm UTC](https://discourse.julialang.org/t/can-i-add-multiple-packages-snoopcompile-precompile-scripts-to-one/40832/1 "2020-06-05T19:44:19Z")

</div>

I’m trying to better understand [SnoopCompile](https://github.com/timholy/SnoopCompile.jl). It seems `@snoopi`ng on my package creates alot of precompile statements in other modules, and my gut feeling is that for my case _these_ are the really important ones in terms of startup time. I think they are appearing because I’m snooping on some “high level” functions in my package which leave lots of things uninferred, which I _think_ is causing something analogous to:

```julia
julia> foo() = sum(Base.inferencebarrier(rand(Float32,2)))
foo (generic function with 1 method)

julia> SnoopCompile.parcel(@snoopi(foo()))
Dict{Symbol,Array{String,1}} with 2 entries:
  :Main => ["precompile(Tuple{typeof(foo)})"]
  :Base => ["precompile(Tuple{typeof(sum),Array{Float32,1}})"]

```

In this case, I’d guess the `precompile(Tuple{typeof(foo)})` will do essentially nothing, the real work is `precompile(Tuple{typeof(sum),Array{Float32,1}})`, but that will be written to a file `precompile_Base.jl` and I’m not sure how / if I’m supposed so include this in my package’s precompile scripts.

I tried in my real case to include some of these other package’s precompile scripts in my main package and some did not work because some symbols were not defined (seems the precompile scripts assume they will be in the package in question?) and when it did it to work, I didn’t see any improvement. Am I doing anything wrong here or misunderstanding something about SnoopCompile? Thanks.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [June 5, 2020, 8:22pm UTC](https://discourse.julialang.org/t/can-i-add-multiple-packages-snoopcompile-precompile-scripts-to-one/40832/2 "2020-06-05T20:22:17Z")

</div>

> how / if I’m supposed so include this in my package’s precompile scripts

My presumption is that if you included them (and there is no actual harm in doing so), they wouldn’t be written to the `*.ji` file. You can essentially think of it as “precompile type piracy”: if you own their `foo` nor any of the types that `foo` acts on, then I think you can’t direct the precompilation to your `*.ji` file. See [RFC: allow precompile to associate a MethodInstance with a module by timholy · Pull Request #31466 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/pull/31466#issuecomment-533299331). However, you might be able to force them in that other package? As should be evident, my understanding of this stage of precompilation is pretty limited.

> some symbols were not defined (seems the precompile scripts assume they will be in the package in question?)

If it generates a precompile script that errors, that’s a SnoopCompile bug. Please report a MWE, if possible.

> and when it did it to work, I didn’t see any improvement

Invalidations can make hash of precompilation. Precompilation has worked for quite a few of my packages but that’s partly because I’ve gone in and made changes elsewhere in the ecosystem to eliminate the invalidations. I am hoping to distribute more information soon about how you can do this yourself (and I expect Julia 1.6 to be far more invalidation-friendly overall), but currently it’s pretty bleeding-edge stuff.

---

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [June 5, 2020, 10:38pm UTC](https://discourse.julialang.org/t/can-i-add-multiple-packages-snoopcompile-precompile-scripts-to-one/40832/3 "2020-06-05T22:38:25Z")

</div>

> [@tim.holy](#):
>
> You can essentially think of it as “precompile type piracy”: if you [don’t?] own their `foo` nor any of the types that `foo` acts on, then I think you can’t direct the precompilation to your `*.ji` file. See [RFC: allow precompile to associate a MethodInstance with a module by timholy · Pull Request #31466 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/pull/31466#issuecomment-533299331).

Thanks, thats helpful. Yea, the example of precompiling `setindex!` from your issue I think is very similar to what I’m trying to do here. One difference is that in your example, `MyPkg` _does_ own `T` so maybe its not full “precompile type piracy”? But maybe that’s not the important piece, in any case in my MWE in the original post and my real case my package indeed does not own either function or argument types.

What I don’t understand though is, to take your example, if the call to `setindex!` is inside one of my fully inferred functions, does mean it still doesn’t get precompiled? Like, will this end up precompiling `setindex!`? :

```julia
module MyPkg

struct T end

foo() = setindex!(Dict{T,Int}(), 1, T())

if ccall(:jl_generating_output, Cint, ()) == 1
    precompile(foo, ())
end

end

```

If yes, then to me it seems like I should be able to make my MWE work too, and its just the `inferencebarrier` thats the problem? If not, then that’s definitely a current limitation of SnoopCompile I hadn’t appreciated.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [June 5, 2020, 11:27pm UTC](https://discourse.julialang.org/t/can-i-add-multiple-packages-snoopcompile-precompile-scripts-to-one/40832/4 "2020-06-05T23:27:42Z")

</div>

In my experience, Dicts are a special case; for example, check Revise on Julia nightly:

```julia
julia> using SnoopCompile

julia> tinf = @snoopi tmin=0.01 using Revise
4-element Array{Tuple{Float64,Core.MethodInstance},1}:
 (0.011423826217651367, MethodInstance for setindex!(::Dict{String,Revise.WatchList}, ::Revise.WatchList, ::String))
 (0.011632919311523438, MethodInstance for setindex!(::Dict{Base.PkgId,Revise.PkgData}, ::Revise.PkgData, ::Base.PkgId))
 (0.01972484588623047, MethodInstance for setindex!(::OrderedCollections.OrderedDict{Module,OrderedCollections.OrderedDict{Revise.RelocatableExpr,Union{Nothing, Array{Any,1}}}}, ::OrderedCollections.OrderedDict{Revise.RelocatableExpr,Union{Nothing, Array{Any,1}}}, ::Module))
 (0.03306698799133301, MethodInstance for setindex!(::Dict{Base.PkgId,CodeTracking.PkgFiles}, ::CodeTracking.PkgFiles, ::Base.PkgId))

```

`setindex!` methods for various Dict are types are the only ones that _don’t_ successfully precompile. (The list would be far longer if there weren’t precompile directives for CodeTracking, JuliaInterpreter, LoweredCodeUtils, and Revise.) The issue is summarized in [Change precompilation format to be more amenable to copying by timholy · Pull Request #32705 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/pull/32705) (see the explanation about `roots`), though I suspect that proposed fix will not be merged. It turns out, though, that this doesn’t seem to be an issue for the vast majority of methods that Revise needs at startup.

Julia 1.4 has several other items that fail to precompile, due to invalidations (totally separate issue and much more common).