# Speculator.jl: Reduce latency through speculative compilation

**URL:** <https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344>\
**Category:** Package Announcements\
**Tags:** compilation, precompilation, latency, precompile\
**Created:** [January 1, 2025, 9:38pm UTC](https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344 "2025-01-01T21:38:22Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![jakobjpeters](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakobjpeters/32/207797_2.png) [@jakobjpeters](https://discourse.julialang.org/u/jakobjpeters)\
**Post date:** [January 1, 2025, 9:38pm UTC](https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344/1 "2025-01-01T21:38:22Z")

</div>

![](https://global.discourse-cdn.com/julialang/original/3X/7/8/78644c4df2c1fd7e5405fa01171fa9177d2fe287.png)

## Introduction

Hi everyone! I’m excited to announce [Speculator.jl](https://github.com/jakobjpeters/Speculator.jl), which reduces latency by automatically searching for compilable method specializations.

## Features

The primary feature is the function `speculate`. It will search the `methods` of a callable value and recursively search the values of a `Module`. Some specializations are known to already be compiled, and are skipped. The `verbosity` can be used to specify various logging behavior,  
which only shows warnings for failed calls to `precompile` by default.

```julia-repl
julia> speculate(Int; verbosity = debug)
[ Info: Skipped `Int64(::Float64)`
[ Info: Compiled `Int64(::Float32)`
[ Info: Compiled `Int64(::Float16)`

```

Values obtained from a module can also be skipped by providing a `predicate` as the first parameter.

```julia-repl
julia> module Example
           export f

           f() = nothing
           g(::Int) = nothing
       end;

julia> speculate(Base.ispublic, Example; verbosity = debug)
[ Info: Compiled `Main.Example.f()`

julia> speculate(Example; verbosity = debug)
[ Info: Skipped `Main.Example.f()`
[ Info: Compiled `Main.Example.g(::Int64)`

```

By default, `speculate` will attempt to compile each specialization. Instead, they can be skipped by setting `dry = true`, which is useful for testing which specializations are generated.

```julia-repl
julia> speculate(Char; dry = true, verbosity = debug)
[ Info: Skipped `Char(::LinearAlgebra.WrapperChar)`
[ Info: Skipped `Char(::UInt32)`

```

Methods with abstractly typed parameters are also searched. The search generates specializations using the Cartesian product of each parameter’s subtypes. Since the number of specializations is exponentially large, the search skips methods whose number of specializations exceeds the `limit`. The default `limit` is `1`, which will generate every specialization where each of the method’s parameters are either a concrete type or have been annotated with `@nospecialize`.

```julia-repl
julia> i(::Union{String, Symbol}, ::AbstractChar) = nothing;

julia> speculate(i; limit = 4, verbosity = debug)
[ Info: Compiled `Main.i(::Symbol, ::LinearAlgebra.WrapperChar)`
[ Info: Compiled `Main.i(::String, ::LinearAlgebra.WrapperChar)`
[ Info: Compiled `Main.i(::Symbol, ::Char)`
[ Info: Compiled `Main.i(::String, ::Char)`

```

Precompilation directives may also be saved by specifying a `path`.

```julia-repl
julia> path = tempname();

julia> speculate(Bool; path)

julia> print(read(path, String))
precompile(Tuple{Type{Bool}, BigFloat})
precompile(Tuple{Type{Bool}, Float16})

```

The value `all_modules` can be used to search every loaded module. Since the `limit` is `1`, this example finds `6354` methods with only a single possible specialization out of `26786` total methods.

```julia-repl
julia> speculate(all_modules; dry = true, verbosity = review)
[ Info: Generated `6354` methods from `26786` generic methods in `2.4881` seconds

```

## Package Precompilation

Calls to `speculate` are designed for use in a package to automatically handle precompilation.  
The function is suitable to call from the top-level, because it only runs when called during package precompilation, to save precompilation directives to a `path`, and during an interactive session. In fact, the precompilation for Speculator.jl itself is simply `speculate(Speculator; limit = 4)`. A call to `speculate(Base.ispublic, ::Module)` at the end of a package ensures that every concretely typed method of public values is precompiled.

## Interactive Use

By default, `speculate` will run in the foreground. It can instead be ran in another thread by setting `background = true`. This is useful in an interactive session, because compilation can occur while code is not being ran.

```julia-repl
julia> using Plots

julia> speculate(Plots; background = true, verbosity = review)

julia> plot(1)

┌ Info: Generated `554` methods from `2114` generic methods in `11.1532` seconds
│ Compiled `313`
│ Skipped `241`
└ Warned `0`

```

Use `install_speculator` to automatically call `speculate`, with `background = true` by default, on the input value. This can also be placed in a `startup.jl` file and be removed using `uninstall_speculator`.

```julia-repl
julia> install_speculator(; verbosity = review)
[ Info: The input speculator has been installed into the REPL

julia> f() = nothing;

┌ Info: Generated `1` methods from `1` generic methods in `0.0148` seconds
│ Compiled `1`
│ Skipped `0`
└ Warned `0`
julia> Iterators
Base.Iterators

┌ Info: Generated `25` methods from `180` generic methods in `0.6180` seconds
│ Compiled `22`
│ Skipped `3`
└ Warned `0`

```

## Comparisons

This is a brief overview of precompilation in similar packages. More detailed comparisons to other techniques will be documented in a future release. Speculator.jl is unique among these techniques in that it supports automatic speculative compilation in the REPL.

### PrecompileTools.jl

This package automates precompilation by running a workload. Additionally, it supports healing invalidations and precompiling methods from dynamic dispatch. However, it requires maintaining and running a workload. The workload retains the benefit of ensuring that it is precompiled, whereas Speculator.jl may be able to precompile all, some, or none of the workload.

### PrecompileSignatures.jl

This package implements similar functionality as Speculator.jl; notably that `PrecompileSignatures.@precompile_signatures ::Module` is roughly equivalent to `Speculator.speculate(::Module)`. However, it only handles `Union` parameter types and has less configuration.

## Future Work

- More documenation
- Check the `predicate` for parameter types
- Disable during development using Preferences.jl?
- Support for `UnionAll` types?
- Specialization hints?

## Conclusion

Please reach out with comments, questions, suggestions, issues, and contributions.  
Thank you for taking the time to checkout Speculator.jl!

---

<div class="post-metadata">

**Author:** ![curtd](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/curtd/32/15868_2.png) [@curtd](https://discourse.julialang.org/u/curtd)\
**Post date:** [January 2, 2025, 4:49pm UTC](https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344/2 "2025-01-02T16:49:20Z")

</div>

This looks wonderful! Can’t wait to try it out

---

<div class="post-metadata">

**Author:** ![linwaytin](https://avatars.discourse-cdn.com/v4/letter/l/898d66/32.png) [@linwaytin](https://discourse.julialang.org/u/linwaytin)\
**Post date:** [January 2, 2025, 7:26pm UTC](https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344/3 "2025-01-02T19:26:03Z")

</div>

This looks an important package for julia. I always want to reduce the latency for the first plot. Currently, I think with either `Plots` or `PyPlot` it will take more than 2 seconds, while in python it takes less than 1s with `matplotlib` on my machine.

This package may be useful for using julia in scripting, like doing simple compuation and then plotting. Do you have any numbers for the reduction in latency?

---

<div class="post-metadata">

**Author:** ![jakobjpeters](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakobjpeters/32/207797_2.png) [@jakobjpeters](https://discourse.julialang.org/u/jakobjpeters)\
**Post date:** [January 2, 2025, 8:12pm UTC](https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344/4 "2025-01-02T20:12:55Z")

</div>

Thanks! Here’s an estimate of the duration of compilation time in Plots.jl. Note that restricting the search to `Base.ispublic` compiles fewer methods, while increasing the `limit` would compile more methods.

```julia-repl
(@v1.11) pkg> activate --temp
  Activating new project at `/tmp/jl_lUEr8N`

(jl_lUEr8N) pkg> add Speculator, Plots

julia> using Speculator, Plots

# precompilation for basic functionality looks good
julia> @elapsed plot(0)
0.100117311

julia> @elapsed plot(0)
0.000245418

# ensure that methods in Speculator.jl are compiled
julia> speculate(Plots; dry = true, verbosity = review)
[ Info: ...

# measure runtime within Speculator.jl and calls to `precompile`
julia> speculate(Plots; verbosity = review)
┌ Info: Generated `554` methods from `2114` generic methods in `10.8543` seconds
│ Compiled `313`
│ Skipped `241`
└ Warned `0`

# measure runtime within Speculator.jl
julia> speculate(Plots; verbosity = review)
┌ Info: Generated `554` methods from `2114` generic methods in `0.6412` seconds
│ Compiled `0`
│ Skipped `554`
└ Warned `0`

```

By this estimate, `speculate` is able to compile concretely typed methods within Plots.jl for 10 seconds. For comparison, repeating the test using the predicate `Base.ispublic` yields about 4 seconds of compilation. I’m uncertain how much of this time translates to a realistic workflow.

If you know that you’re going to use Plots.jl in a given environment, it could be worth checking for that environment and placing `speculate(Plots; background = true)` in a `startup.jl`. Alternatively, you could use `install_speculator()` and then enter `Plots` in the REPL for the same effect. If you give this a test, I’m curious to hear if it’s effective in your workflow!

---

<div class="post-metadata">

**Author:** ![linwaytin](https://avatars.discourse-cdn.com/v4/letter/l/898d66/32.png) [@linwaytin](https://discourse.julialang.org/u/linwaytin)\
**Post date:** [January 3, 2025, 2:16am UTC](https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344/5 "2025-01-03T02:16:28Z")

</div>

Thank you for the reply. I tried it but did not see difference. Maybe I misunderstood how to use it?

```julia
julia> using Speculator, Plots

julia> speculate(Plots; verbosity = review)
┌ Info: Generated `625` methods from `2563` generic methods in `12.6170` seconds
│ Compiled `381`
│ Skipped `244`
└ Warned `0`

julia> @elapsed plot(0)
0.071012581

```

---

<div class="post-metadata">

**Author:** ![jakobjpeters](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakobjpeters/32/207797_2.png) [@jakobjpeters](https://discourse.julialang.org/u/jakobjpeters)\
**Post date:** [January 3, 2025, 2:30am UTC](https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344/6 "2025-01-03T02:30:24Z")

</div>

The call to `plot(0)` has already been precompiled by Plots.jl, so `speculate` won’t have much of an impact for that call. It was used as an example, but perhaps added unnecessary confusion. Since you mentioned it takes a couple of seconds for your first plot, I presume that you’re doing something more complicated. It’s possible that `speculate` is able to help in that case, but I haven’t tried it personally other than the test shown above. Can you try it again with something that would typically cause more latency for you?

---

<div class="post-metadata">

**Author:** ![linwaytin](https://avatars.discourse-cdn.com/v4/letter/l/898d66/32.png) [@linwaytin](https://discourse.julialang.org/u/linwaytin)\
**Post date:** [January 3, 2025, 2:54am UTC](https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344/7 "2025-01-03T02:54:02Z")

</div>

I am confused now. Could you please give an example of how to use it to reduce the latency? Thanks.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [January 3, 2025, 4:34am UTC](https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344/8 "2025-01-03T04:34:22Z")

</div>

> [@linwaytin](#):
>
> Could you please give an example of how to use it to reduce the latency?

You’ll have to specify the circumstances and what latency you’re trying to avoid, and depending on where it is, you’ll have very different options or possibly nothing. Let’s saying I have a package `A` that precompiles a `foo(::Bar)` call signature; that precompilation adds latency to `using A` because there’s more code to load. If I need `foo(::Bar)`, then I still save time because I don’t need to repeat the compilation each session, just the loading. If I needed a different call signature like `foo(::Baz)`, then I still have to compile it each session after `using A`. Calling `foo(Baz(...))` would automatically do that for me; manually precompiling it after an import wouldn’t save time overall because the compiled code is not saved for future sessions, and it may only appear it does if we neglect to time the precompilation step e.g. `@elapsed plot(0)` ignores the preceding `speculate(Plots)`. If you know the exact runtime calls, you could just use `precompile`, you don’t need to take a chance on the method being suitable for Speculator. Runtime precompilation of a wide set of useful call signatures can still be beneficial in the time before we decide on the exact calls for the job; while the overall time compiling and executing is the same and we still wait until the first plot, overall idle time could be decreased and we wait less on each precompiled call.

---

<div class="post-metadata">

**Author:** ![jakobjpeters](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakobjpeters/32/207797_2.png) [@jakobjpeters](https://discourse.julialang.org/u/jakobjpeters)\
**Post date:** [January 3, 2025, 4:48am UTC](https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344/9 "2025-01-03T04:48:43Z")

</div>

> [@Benny](#):
>
> depending on where it is, you’ll have very different options or possibly nothing

This package is substantially less useful than PrecompileTools.jl in cases of dynamic dispatch and any codebase with many untyped parameters. I’m still trying to get a sense of how helpful it is, and when.

---

<div class="post-metadata">

**Author:** ![jakobjpeters](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakobjpeters/32/207797_2.png) [@jakobjpeters](https://discourse.julialang.org/u/jakobjpeters)\
**Post date:** [January 3, 2025, 4:55am UTC](https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344/10 "2025-01-03T04:55:57Z")

</div>

> [@Benny](#):
>
> Runtime precompilation of a wide set of useful call signatures can still be beneficial in the time before we decide on the exact calls for the job; while the total time compiling and executing is the same and we still wait until the first plot, idle time could be decreased and we wait less on each precompiled call.

This very well articulates the goal of `install_speculator`, as there are many “idle” minutes in the REPL thinking or writing code 🙂

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [January 3, 2025, 11:53am UTC](https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344/11 "2025-01-03T11:53:00Z")

</div>

> [@jakobjpeters](#):
>
> I’m still trying to get a sense of how helpful it is, and when.

I think it’s as limited and convenient as the announcement said. It can’t touch methods with infinite specializations, and it only compiles methods with a number of specializations `<= limit`. However, the alternatives are _also_ limited:

- manually listing every `precompile` directive you want for a package or startup. This is the most precise approach, but it’s insanely time-consuming to write and maintain, so we go for more automated approaches.
- PrecompileTools workload is a main-like routine that generates precompile directives for calls we expect most users would use. That often saves a lot of writing, but it may not; in the worst case we’re manually listing calls again or we’re deliberately ignoring some calls just because it’s hard to maintain.

So it’s convenient to have an even more automatic option, even if it’s only possible for methods with a finite number of specializations. At an extreme, say we want to precompile a module of methods fully for the `<= limit` specializations we designed each to have; one call for the module sure beats a large `@eval` loop combining the various input types before calling every function (note the `limit` is per _method_, not per function, so you can do `> limit` calls of a function if you split them among the internal methods evenly enough). Most code isn’t like this, and precompiling all methods with 1 specialization can easily lead to undesired bloat, but Speculator lets us go more granular.

---

<div class="post-metadata">

**Author:** ![MilesCranmer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/milescranmer/32/21070_2.png) [@MilesCranmer](https://discourse.julialang.org/u/MilesCranmer)\
**Post date:** [January 3, 2025, 12:56pm UTC](https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344/12 "2025-01-03T12:56:31Z")

</div>

To use it _inside_ my package (for use by external users), would I just do something like this?

```julia
module MyPackage

#= package code =#

using Speculator: speculate, silent
speculate(@ __MODULE__ ; limit=128, verbosity=silent)

end

```

Also, how does it work for submodules in my package? Would I need to run it on each one independently?

---

<div class="post-metadata">

**Author:** ![jakobjpeters](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakobjpeters/32/207797_2.png) [@jakobjpeters](https://discourse.julialang.org/u/jakobjpeters)\
**Post date:** [January 3, 2025, 7:20pm UTC](https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344/13 "2025-01-03T19:20:43Z")

</div>

Yes, that’s correct. It will automatically run on submodules unless otherwise specified by the predicate.

```julia-repl
julia> module A
           module B
               struct X end

               # skipped because `Int` is not in the `names` of `A` or `B`
               Base.Int(::X) = 1
           end

           module C
               # skipped because `C` is not allowed by the predicate
               f() = 2
           end
       end;

julia> speculate((m, _) -> m != A.C, A; verbosity = debug)
[ Info: Compiled `Main.A.B.X()`

```

---

<div class="post-metadata">

**Author:** ![jakobjpeters](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakobjpeters/32/207797_2.png) [@jakobjpeters](https://discourse.julialang.org/u/jakobjpeters)\
**Post date:** [January 3, 2025, 7:28pm UTC](https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344/14 "2025-01-03T19:28:59Z")

</div>

> [@Benny](#):
>
> I think it’s as limited and convenient as the announcement said.

I’m glad that you feel that the announcement was appropriate in that regard!

> [@Benny](#):
>
> in the worst case we’re manually listing calls again or we’re deliberately ignoring some calls just because it’s hard to maintain.

It would be nice if PrecompileTools.jl was also able to give more of a sense of breadth (number of functions in your package precompiled) and depth (number of specializations per method). It does provide logging, which can help but may be difficult to reason about

> [@Benny](#):
>
> one call for the module sure beats a large `@eval` loop combining the various input types before calling every function.

👏

---

<div class="post-metadata">

**Author:** ![StevenSiew](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevensiew/32/218393_2.png) [@StevenSiew](https://discourse.julialang.org/u/StevenSiew)\
**Post date:** [January 4, 2025, 12:34am UTC](https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344/15 "2025-01-04T00:34:29Z")

</div>

why can’t we have a feature like this?

bash\> julia --detect-methods-for-future-precompilations myprog.jl

So that every time I run my julia program, it starts up faster and faster and faster

---

<div class="post-metadata">

**Author:** ![jakobjpeters](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakobjpeters/32/207797_2.png) [@jakobjpeters](https://discourse.julialang.org/u/jakobjpeters)\
**Post date:** [January 4, 2025, 1:18am UTC](https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344/16 "2025-01-04T01:18:03Z")

</div>

We do have a feature that enables the detection of compilation, but it doesn’t save the compiled code itself and therefore doesn’t reduce latency in future sessions.

```julia
> julia --trace-compile=precompile.jl

```

You can use PackageCompiler.jl to create a system image, which saves and loads the compiled code to reduce latency in future sessions. However, this process has to be repeated each time you want to update the precompilation directives.

I don’t know enough to provide a good answer as to why we don’t or can’t have a feature that automatically updates a system image during each session. Perhaps someone else here knows 😃

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [January 4, 2025, 3:22am UTC](https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344/17 "2025-01-04T03:22:25Z")

</div>

> [@StevenSiew](#):
>
> bash\> julia --detect-methods-for-future-precompilations myprog.jl
> 
> So that every time I run my julia program, it starts up faster and faster and faster

> [@jakobjpeters](#):
>
> We do have a feature that enables the detection of compilation, but it doesn’t save the compiled code itself and therefore doesn’t reduce latency in future sessions.

For an isolated script execution that doesn’t need to run inside a larger scope? JuliaScript.jl appropriated packaging to save code compiled in the first run; further runs won’t cache newer runtime dispatches, that’s how you run out of storage. In any case, you’ll still need to steer clear of a lot of global or external state, [things go wrong](https://docs.julialang.org/en/v1/manual/modules/#Module-initialization-and-precompilation) when those get saved instead of freshly executed; main function all the way.

---

<div class="post-metadata">

**Author:** ![jakobjpeters](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakobjpeters/32/207797_2.png) [@jakobjpeters](https://discourse.julialang.org/u/jakobjpeters)\
**Post date:** [January 6, 2025, 1:51am UTC](https://discourse.julialang.org/t/speculator-jl-reduce-latency-through-speculative-compilation/124344/18 "2025-01-06T01:51:40Z")

</div>

Speculator.jl version 0.2.0 is now released! See the [news](https://github.com/jakobjpeters/Speculator.jl/blob/v0.2.0/NEWS.md) for a detailed list of updates.

The primary feature in this release is that the `predicate` now checks parameter types. Each component of a `Union` type is checked individually. Skipped parameter types do not count towards the method specialization `limit`.

```julia-repl
julia> module M
           abstract type A end
           struct B <: A end
           struct C <: A end
           struct D end

           f(::Union{A, D}) = nothing
       end;

julia> speculate((_, n) -> n != :A, M.f; verbosity = debug)
[ Info: Compiled `Main.M.f(::Main.M.D)`

julia> speculate((_, n) -> n != :B, M.f; limit = 2, verbosity = debug)
[ Info: Compiled `Main.M.f(::Main.M.C)`
[ Info: Skipped `Main.M.f(::Main.M.D)`

julia> speculate(M.f; limit = 3, verbosity = debug)
[ Info: Skipped `Main.M.f(::Main.M.C)`
[ Info: Compiled `Main.M.f(::Main.M.B)`
[ Info: Skipped `Main.M.f(::Main.M.D)`

```