# Benchmarking function compile time

**URL:** <https://discourse.julialang.org/t/benchmarking-function-compile-time/125013>\
**Category:** Performance\
**Tags:** question, compilation, benchmark\
**Created:** [January 21, 2025, 11:01am UTC](https://discourse.julialang.org/t/benchmarking-function-compile-time/125013 "2025-01-21T11:01:59Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![AntonReinhard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antonreinhard/32/211306_2.png) [@AntonReinhard](https://discourse.julialang.org/u/AntonReinhard)\
**Post date:** [January 21, 2025, 11:01am UTC](https://discourse.julialang.org/t/benchmarking-function-compile-time/125013/1 "2025-01-21T11:01:59Z")

</div>

I have some code that generates functions at runtime that vary greatly in size (10 to 100k lines of code). I’d like to benchmark the _compilation_ of these functions. Is there any intended way to do this? Something like a specific call to force recompilation or deleting the compiled function from Julia’s cache? Does anyone have experience with this?

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [January 21, 2025, 1:22pm UTC](https://discourse.julialang.org/t/benchmarking-function-compile-time/125013/2 "2025-01-21T13:22:03Z")

</div>

> [@AntonReinhard](#):
>
> deleting the compiled function from Julia’s cache

FWIW the internal function `Base.delete_method` should allow you to delete a method. Like `Base.delete_method(@which f(a, b))`.

---

<div class="post-metadata">

**Author:** ![AntonReinhard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antonreinhard/32/211306_2.png) [@AntonReinhard](https://discourse.julialang.org/u/AntonReinhard)\
**Post date:** [January 21, 2025, 1:38pm UTC](https://discourse.julialang.org/t/benchmarking-function-compile-time/125013/3 "2025-01-21T13:38:12Z")

</div>

Interesting, though that seems to delete the entire method, not just the compiled version. But it could maybe be used to `eval` the code again and force a recompilation that way.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [January 21, 2025, 7:16pm UTC](https://discourse.julialang.org/t/benchmarking-function-compile-time/125013/4 "2025-01-21T19:16:46Z")

</div>

It does seem hypothetically possible to delete cached compiled code without deleting the method, but that has little utility besides profiling targetted recompilation like this, and the simplest approach ignores code inlined into other methods, which can turn out fairly differently from isolated compiled code. The mechanisms for recompilation are also tied to the methods e.g. backedges tracking caller methods, so if you hack further into Julia’s internals for profiling compilation, it might be simpler to avoid deleting anything and just compile code to be thrown away. After all, the `@code_` reflection methods do type inference fresh.

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [January 21, 2025, 9:31pm UTC](https://discourse.julialang.org/t/benchmarking-function-compile-time/125013/5 "2025-01-21T21:31:18Z")

</div>

> [@AntonReinhard](#):
>
> But it could maybe be used to `eval` the code again and force a recompilation that way.

Note that this won’t force the recompilation of any methods downstream called by the deleted method though.

* * *

My preferred approach is with Distributed.jl

For example:

```julia
using Distributed

function time_compilation(expr; setup=nothing)
    ps = addprocs(1)
    (;compile_time, recompile_time) = remotecall_fetch(only(ps)) do
        @eval begin
            $setup
            @timed $expr
        end
    end
    rmprocs(ps)
    (;compile_time, recompile_time)
end

```

and then

```julia-repl
julia> time_compilation(:(sin(x)); setup=:(x=[1 2; 3 4]))
(compile_time = 3.439371696, recompile_time = 0.0)

```

says that this takes 3.43 seconds to compile.

This is repeatable because every time it’s run in a new process:

```julia-repl
julia> time_compilation(:(sin(x)); setup=:(x=[1 2; 3 4]))
(compile_time = 3.617281099, recompile_time = 0.0)

julia> time_compilation(:(sin(x)); setup=:(x=[1 2; 3 4]))
(compile_time = 3.38245935, recompile_time = 0.0)

```

---

<div class="post-metadata">

**Author:** ![AntonReinhard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antonreinhard/32/211306_2.png) [@AntonReinhard](https://discourse.julialang.org/u/AntonReinhard)\
**Post date:** [January 22, 2025, 2:09am UTC](https://discourse.julialang.org/t/benchmarking-function-compile-time/125013/6 "2025-01-22T02:09:48Z")

</div>

That looks really good, I’ll try this!  
I’m guessing this still captures the function’s execution time, too. But there’s probably no way around that. I’m measuring the execution time separately anyway, so I can just subtract it, even though it’s probably negligible.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [January 22, 2025, 2:20am UTC](https://discourse.julialang.org/t/benchmarking-function-compile-time/125013/7 "2025-01-22T02:20:45Z")

</div>

> [@AntonReinhard](#):
>
> this still captures the function’s execution time, too. But there’s probably no way around that.

Spitballing here, what about `precompile`-ing a call signature?

---

<div class="post-metadata">

**Author:** ![PatrickHaecker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/patrickhaecker/32/222891_2.png) [@PatrickHaecker](https://discourse.julialang.org/u/PatrickHaecker)\
**Post date:** [January 22, 2025, 6:54am UTC](https://discourse.julialang.org/t/benchmarking-function-compile-time/125013/8 "2025-01-22T06:54:55Z")

</div>

If you are able to use Julia nightly, you can call it with the new `--trace-compile-timing` command line argument as in (using `juliaup`)

```julia
julia +nightly --trace-compile=stderr --trace-compile-timing --eval "mysin(x) = sin(x); mysin(42.0)"

```

which outputs here

```julia
#= 5.2 ms =# precompile(Tuple{typeof(Main.mysin), Float64})

```

If you need the timing value during runtime, you can provide a file name instead of `stderr` and read the timing from there.

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [January 22, 2025, 7:59am UTC](https://discourse.julialang.org/t/benchmarking-function-compile-time/125013/9 "2025-01-22T07:59:33Z")

</div>

No, that was just the compilation time.

---

<div class="post-metadata">

**Author:** ![AntonReinhard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antonreinhard/32/211306_2.png) [@AntonReinhard](https://discourse.julialang.org/u/AntonReinhard)\
**Post date:** [January 22, 2025, 9:38am UTC](https://discourse.julialang.org/t/benchmarking-function-compile-time/125013/10 "2025-01-22T09:38:06Z")

</div>

Oh I see, I didn’t know `@timed` was that powerful, that’s good to know, thank you.

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [January 22, 2025, 10:33am UTC](https://discourse.julialang.org/t/benchmarking-function-compile-time/125013/11 "2025-01-22T10:33:11Z")

</div>

No problem! `@timed` is using the same mechanism that `@time` uses when it tells you that e.g. a computation was `60% compilation time` or whatever. Julia has some internal mechanisms for estimating how long it spent inside the compiler, and `@time` / `@timed` plug into those.