Speculator.jl: Reduce latency through speculative compilation

[jakobjpeters]

Introduction

Hi everyone! I’m excited to announce Speculator.jl, which reduces latency by automatically searching for compilable method specializations.

Features

The primary feature is the function speculate. It will search the methods of a callable value and recursively search the values of a Module. Some specializations are known to already be compiled, and are skipped. The verbosity can be used to specify various logging behavior,
which only shows warnings for failed calls to precompile by default.

julia> speculate(Int; verbosity = debug)
[ Info: Skipped `Int64(::Float64)`
[ Info: Compiled `Int64(::Float32)`
[ Info: Compiled `Int64(::Float16)`

Values obtained from a module can also be skipped by providing a predicate as the first parameter.

julia> module Example
           export f

           f() = nothing
           g(::Int) = nothing
       end;

julia> speculate(Base.ispublic, Example; verbosity = debug)
[ Info: Compiled `Main.Example.f()`

julia> speculate(Example; verbosity = debug)
[ Info: Skipped `Main.Example.f()`
[ Info: Compiled `Main.Example.g(::Int64)`

By default, speculate will attempt to compile each specialization. Instead, they can be skipped by setting dry = true, which is useful for testing which specializations are generated.

julia> speculate(Char; dry = true, verbosity = debug)
[ Info: Skipped `Char(::LinearAlgebra.WrapperChar)`
[ Info: Skipped `Char(::UInt32)`

Methods with abstractly typed parameters are also searched. The search generates specializations using the Cartesian product of each parameter’s subtypes. Since the number of specializations is exponentially large, the search skips methods whose number of specializations exceeds the limit. The default limit is 1, which will generate every specialization where each of the method’s parameters are either a concrete type or have been annotated with @nospecialize.

julia> i(::Union{String, Symbol}, ::AbstractChar) = nothing;

julia> speculate(i; limit = 4, verbosity = debug)
[ Info: Compiled `Main.i(::Symbol, ::LinearAlgebra.WrapperChar)`
[ Info: Compiled `Main.i(::String, ::LinearAlgebra.WrapperChar)`
[ Info: Compiled `Main.i(::Symbol, ::Char)`
[ Info: Compiled `Main.i(::String, ::Char)`

Precompilation directives may also be saved by specifying a path.

julia> path = tempname();

julia> speculate(Bool; path)

julia> print(read(path, String))
precompile(Tuple{Type{Bool}, BigFloat})
precompile(Tuple{Type{Bool}, Float16})

The value all_modules can be used to search every loaded module. Since the limit is 1, this example finds 6354 methods with only a single possible specialization out of 26786 total methods.

julia> speculate(all_modules; dry = true, verbosity = review)
[ Info: Generated `6354` methods from `26786` generic methods in `2.4881` seconds

Package Precompilation

Calls to speculate are designed for use in a package to automatically handle precompilation.
The function is suitable to call from the top-level, because it only runs when called during package precompilation, to save precompilation directives to a path, and during an interactive session. In fact, the precompilation for Speculator.jl itself is simply speculate(Speculator; limit = 4). A call to speculate(Base.ispublic, ::Module) at the end of a package ensures that every concretely typed method of public values is precompiled.

Interactive Use

By default, speculate will run in the foreground. It can instead be ran in another thread by setting background = true. This is useful in an interactive session, because compilation can occur while code is not being ran.

julia> using Plots

julia> speculate(Plots; background = true, verbosity = review)

julia> plot(1)

┌ Info: Generated `554` methods from `2114` generic methods in `11.1532` seconds
│ Compiled `313`
│ Skipped  `241`
└ Warned   `0`

Use install_speculator to automatically call speculate, with background = true by default, on the input value. This can also be placed in a startup.jl file and be removed using uninstall_speculator.

julia> install_speculator(; verbosity = review)
[ Info: The input speculator has been installed into the REPL

julia> f() = nothing;

┌ Info: Generated `1` methods from `1` generic methods in `0.0148` seconds
│ Compiled `1`
│ Skipped  `0`
└ Warned   `0`
julia> Iterators
Base.Iterators

┌ Info: Generated `25` methods from `180` generic methods in `0.6180` seconds
│ Compiled `22`
│ Skipped  `3`
└ Warned   `0`

Comparisons

This is a brief overview of precompilation in similar packages. More detailed comparisons to other techniques will be documented in a future release. Speculator.jl is unique among these techniques in that it supports automatic speculative compilation in the REPL.

PrecompileTools.jl

This package automates precompilation by running a workload. Additionally, it supports healing invalidations and precompiling methods from dynamic dispatch. However, it requires maintaining and running a workload. The workload retains the benefit of ensuring that it is precompiled, whereas Speculator.jl may be able to precompile all, some, or none of the workload.

PrecompileSignatures.jl

This package implements similar functionality as Speculator.jl; notably that PrecompileSignatures.@precompile_signatures ::Module is roughly equivalent to Speculator.speculate(::Module). However, it only handles Union parameter types and has less configuration.

Future Work

• More documenation
• Check the predicate for parameter types
• Disable during development using Preferences.jl?
• Support for UnionAll types?
• Specialization hints?

Conclusion

Please reach out with comments, questions, suggestions, issues, and contributions.
Thank you for taking the time to checkout Speculator.jl!

[curtd]

This looks wonderful! Can’t wait to try it out

[linwaytin]

This looks an important package for julia. I always want to reduce the latency for the first plot. Currently, I think with either Plots or PyPlot it will take more than 2 seconds, while in python it takes less than 1s with matplotlib on my machine.

This package may be useful for using julia in scripting, like doing simple compuation and then plotting. Do you have any numbers for the reduction in latency?

[jakobjpeters]

Thanks! Here’s an estimate of the duration of compilation time in Plots.jl. Note that restricting the search to Base.ispublic compiles fewer methods, while increasing the limit would compile more methods.

(@v1.11) pkg> activate --temp
  Activating new project at `/tmp/jl_lUEr8N`

(jl_lUEr8N) pkg> add Speculator, Plots

julia> using Speculator, Plots

# precompilation for basic functionality looks good
julia> @elapsed plot(0)
0.100117311

julia> @elapsed plot(0)
0.000245418

# ensure that methods in Speculator.jl are compiled
julia> speculate(Plots; dry = true, verbosity = review)
[ Info: ...

# measure runtime within Speculator.jl and calls to `precompile`
julia> speculate(Plots; verbosity = review)
┌ Info: Generated `554` methods from `2114` generic methods in `10.8543` seconds
│ Compiled `313`
│ Skipped  `241`
└ Warned   `0`

# measure runtime within Speculator.jl
julia> speculate(Plots; verbosity = review)
┌ Info: Generated `554` methods from `2114` generic methods in `0.6412` seconds
│ Compiled `0`
│ Skipped  `554`
└ Warned   `0`

By this estimate, speculate is able to compile concretely typed methods within Plots.jl for 10 seconds. For comparison, repeating the test using the predicate Base.ispublic yields about 4 seconds of compilation. I’m uncertain how much of this time translates to a realistic workflow.

If you know that you’re going to use Plots.jl in a given environment, it could be worth checking for that environment and placing speculate(Plots; background = true) in a startup.jl. Alternatively, you could use install_speculator() and then enter Plots in the REPL for the same effect. If you give this a test, I’m curious to hear if it’s effective in your workflow!

[linwaytin]

Thank you for the reply. I tried it but did not see difference. Maybe I misunderstood how to use it?

julia> using Speculator, Plots

julia> speculate(Plots; verbosity = review)
┌ Info: Generated `625` methods from `2563` generic methods in `12.6170` seconds
│ Compiled `381`
│ Skipped  `244`
└ Warned   `0`

julia> @elapsed plot(0)
0.071012581

[jakobjpeters]

The call to plot(0) has already been precompiled by Plots.jl, so speculate won’t have much of an impact for that call. It was used as an example, but perhaps added unnecessary confusion. Since you mentioned it takes a couple of seconds for your first plot, I presume that you’re doing something more complicated. It’s possible that speculate is able to help in that case, but I haven’t tried it personally other than the test shown above. Can you try it again with something that would typically cause more latency for you?

[linwaytin]

I am confused now. Could you please give an example of how to use it to reduce the latency? Thanks.

[Benny]

@linwaytin

Could you please give an example of how to use it to reduce the latency?

You’ll have to specify the circumstances and what latency you’re trying to avoid, and depending on where it is, you’ll have very different options or possibly nothing. Let’s saying I have a package A that precompiles a foo(::Bar) call signature; that precompilation adds latency to using A because there’s more code to load. If I need foo(::Bar), then I still save time because I don’t need to repeat the compilation each session, just the loading. If I needed a different call signature like foo(::Baz), then I still have to compile it each session after using A. Calling foo(Baz(...)) would automatically do that for me; manually precompiling it after an import wouldn’t save time overall because the compiled code is not saved for future sessions, and it may only appear it does if we neglect to time the precompilation step e.g. @elapsed plot(0) ignores the preceding speculate(Plots). If you know the exact runtime calls, you could just use precompile, you don’t need to take a chance on the method being suitable for Speculator. Runtime precompilation of a wide set of useful call signatures can still be beneficial in the time before we decide on the exact calls for the job; while the overall time compiling and executing is the same and we still wait until the first plot, overall idle time could be decreased and we wait less on each precompiled call.

[jakobjpeters]

@Benny

depending on where it is, you’ll have very different options or possibly nothing

This package is substantially less useful than PrecompileTools.jl in cases of dynamic dispatch and any codebase with many untyped parameters. I’m still trying to get a sense of how helpful it is, and when.

[jakobjpeters]

@Benny

Runtime precompilation of a wide set of useful call signatures can still be beneficial in the time before we decide on the exact calls for the job; while the total time compiling and executing is the same and we still wait until the first plot, idle time could be decreased and we wait less on each precompiled call.

This very well articulates the goal of install_speculator, as there are many “idle” minutes in the REPL thinking or writing code

[Benny]

@jakobjpeters

I’m still trying to get a sense of how helpful it is, and when.

I think it’s as limited and convenient as the announcement said. It can’t touch methods with infinite specializations, and it only compiles methods with a number of specializations <= limit. However, the alternatives are also limited:

• manually listing every precompile directive you want for a package or startup. This is the most precise approach, but it’s insanely time-consuming to write and maintain, so we go for more automated approaches.
• PrecompileTools workload is a main-like routine that generates precompile directives for calls we expect most users would use. That often saves a lot of writing, but it may not; in the worst case we’re manually listing calls again or we’re deliberately ignoring some calls just because it’s hard to maintain.

So it’s convenient to have an even more automatic option, even if it’s only possible for methods with a finite number of specializations. At an extreme, say we want to precompile a module of methods fully for the <= limit specializations we designed each to have; one call for the module sure beats a large @eval loop combining the various input types before calling every function (note the limit is per method, not per function, so you can do > limit calls of a function if you split them among the internal methods evenly enough). Most code isn’t like this, and precompiling all methods with 1 specialization can easily lead to undesired bloat, but Speculator lets us go more granular.

[MilesCranmer]

To use it inside my package (for use by external users), would I just do something like this?

module MyPackage

#= package code =#

using Speculator: speculate, silent
speculate(@__MODULE__; limit=128, verbosity=silent)

end

Also, how does it work for submodules in my package? Would I need to run it on each one independently?

[jakobjpeters]

Yes, that’s correct. It will automatically run on submodules unless otherwise specified by the predicate.

julia> module A
           module B
               struct X end

               # skipped because `Int` is not in the `names` of `A` or `B`
               Base.Int(::X) = 1
           end

           module C
               # skipped because `C` is not allowed by the predicate
               f() = 2
           end
       end;

julia> speculate((m, _) -> m != A.C, A; verbosity = debug)
[ Info: Compiled `Main.A.B.X()`

[jakobjpeters]

@Benny

I think it’s as limited and convenient as the announcement said.

I’m glad that you feel that the announcement was appropriate in that regard!

@Benny

in the worst case we’re manually listing calls again or we’re deliberately ignoring some calls just because it’s hard to maintain.

It would be nice if PrecompileTools.jl was also able to give more of a sense of breadth (number of functions in your package precompiled) and depth (number of specializations per method). It does provide logging, which can help but may be difficult to reason about

@Benny

one call for the module sure beats a large @eval loop combining the various input types before calling every function.

[StevenSiew]

why can’t we have a feature like this?

bash> julia --detect-methods-for-future-precompilations myprog.jl

So that every time I run my julia program, it starts up faster and faster and faster

[jakobjpeters]

We do have a feature that enables the detection of compilation, but it doesn’t save the compiled code itself and therefore doesn’t reduce latency in future sessions.

> julia --trace-compile=precompile.jl

You can use PackageCompiler.jl to create a system image, which saves and loads the compiled code to reduce latency in future sessions. However, this process has to be repeated each time you want to update the precompilation directives.

I don’t know enough to provide a good answer as to why we don’t or can’t have a feature that automatically updates a system image during each session. Perhaps someone else here knows

[Benny]

@StevenSiew

bash> julia --detect-methods-for-future-precompilations myprog.jl

So that every time I run my julia program, it starts up faster and faster and faster

@jakobjpeters

We do have a feature that enables the detection of compilation, but it doesn’t save the compiled code itself and therefore doesn’t reduce latency in future sessions.

For an isolated script execution that doesn’t need to run inside a larger scope? JuliaScript.jl appropriated packaging to save code compiled in the first run; further runs won’t cache newer runtime dispatches, that’s how you run out of storage. In any case, you’ll still need to steer clear of a lot of global or external state, things go wrong when those get saved instead of freshly executed; main function all the way.

[jakobjpeters]

Speculator.jl version 0.2.0 is now released! See the news for a detailed list of updates.

The primary feature in this release is that the predicate now checks parameter types. Each component of a Union type is checked individually. Skipped parameter types do not count towards the method specialization limit.

julia> module M
           abstract type A end
           struct B <: A end
           struct C <: A end
           struct D end

           f(::Union{A, D}) = nothing
       end;

julia> speculate((_, n) -> n != :A, M.f; verbosity = debug)
[ Info: Compiled `Main.M.f(::Main.M.D)`

julia> speculate((_, n) -> n != :B, M.f; limit = 2, verbosity = debug)
[ Info: Compiled `Main.M.f(::Main.M.C)`
[ Info: Skipped `Main.M.f(::Main.M.D)`

julia> speculate(M.f; limit = 3, verbosity = debug)
[ Info: Skipped `Main.M.f(::Main.M.C)`
[ Info: Compiled `Main.M.f(::Main.M.B)`
[ Info: Skipped `Main.M.f(::Main.M.D)`