Benchmarking function compile time

General Usage Performance
, ,
1

I have some code that generates functions at runtime that vary greatly in size (10 to 100k lines of code). I’d like to benchmark the compilation of these functions. Is there any intended way to do this? Something like a specific call to force recompilation or deleting the compiled function from Julia’s cache? Does anyone have experience with this?

2

FWIW the internal function Base.delete_method should allow you to delete a method. Like Base.delete_method(@which f(a, b)).

3

Interesting, though that seems to delete the entire method, not just the compiled version. But it could maybe be used to eval the code again and force a recompilation that way.

4

It does seem hypothetically possible to delete cached compiled code without deleting the method, but that has little utility besides profiling targetted recompilation like this, and the simplest approach ignores code inlined into other methods, which can turn out fairly differently from isolated compiled code. The mechanisms for recompilation are also tied to the methods e.g. backedges tracking caller methods, so if you hack further into Julia’s internals for profiling compilation, it might be simpler to avoid deleting anything and just compile code to be thrown away. After all, the @code_ reflection methods do type inference fresh.

5

Note that this won’t force the recompilation of any methods downstream called by the deleted method though.

My preferred approach is with Distributed.jl

For example:

using Distributed

function time_compilation(expr; setup=nothing)
    ps = addprocs(1)
    (;compile_time, recompile_time) = remotecall_fetch(only(ps)) do
        @eval begin
            $setup
            @timed $expr
        end
    end
    rmprocs(ps)
    (;compile_time, recompile_time)
end

and then

julia> time_compilation(:(sin(x)); setup=:(x=[1 2; 3 4]))
(compile_time = 3.439371696, recompile_time = 0.0)

says that this takes 3.43 seconds to compile.

This is repeatable because every time it’s run in a new process:

julia> time_compilation(:(sin(x)); setup=:(x=[1 2; 3 4]))
(compile_time = 3.617281099, recompile_time = 0.0)

julia> time_compilation(:(sin(x)); setup=:(x=[1 2; 3 4]))
(compile_time = 3.38245935, recompile_time = 0.0)
3 Likes
6

That looks really good, I’ll try this!
I’m guessing this still captures the function’s execution time, too. But there’s probably no way around that. I’m measuring the execution time separately anyway, so I can just subtract it, even though it’s probably negligible.

7

Spitballing here, what about precompile-ing a call signature?

8

If you are able to use Julia nightly, you can call it with the new --trace-compile-timing command line argument as in (using juliaup)

julia +nightly --trace-compile=stderr --trace-compile-timing --eval "mysin(x) = sin(x); mysin(42.0)"

which outputs here

#=    5.2 ms =# precompile(Tuple{typeof(Main.mysin), Float64})

If you need the timing value during runtime, you can provide a file name instead of stderr and read the timing from there.