How to asychronously build a benchmark group

Tooling Profiling
,
1

Can I build several inter-depending benchmarks from a serial procedure in “one go” using BenchmarkTools.jl ?

Say I have the following functionality laid out in a function

function mymutatingfunc!(x)
   mutfunc1!(x)
   mutfunc2!(x)
   mutfunc3!(x)
end

and x needs to be initialized x = init()

All mutfunc!s need to be called in that order, otherwise they don’t make sense.
E.g., mutfunc3! can operate normally only if mutfunc1!,mutfunc2! are called in beforehand.

Current solution

I would like to benchmark all mutfunc!s individually.
This is complex because it would be something like

x = init()
SUITE["muttfunc1"] = @benchmarkable muttfunc1!(y) setup=(y = x |> deepcopy)
SUITE["muttfunc2"] = @benchmarkable muttfunc2!(y) setup=(y = x |> deepcopy |> mutfunc1!)
SUITE["muttfunc3"] = @benchmarkable muttfunc3!(y) setup=(y = x |> deepcopy |> mutfunc1!  |> mutfunc2!)

This looks like a waste of resources.

Repeatedly calling mtfunc! is also not possible as the content changes significantly.
E.g. I cannot do something like

x = init()
SUITE["muttfunc1"] = @benchmarkable muttfunc1!(y) setup=(y = x |> deepcopy)
muttfunc1!(x) 
SUITE["muttfunc2"] = @benchmarkable muttfunc2!(y) setup=(y = x |> deepcopy)
muttfunc2!(x) 
SUITE["muttfunc3"] = @benchmarkable muttfunc3!(y) setup=(y = x |> deepcopy)

Desirable

Could I “asychronously” build the benchmarks by sampling in a round-robin fashion.
E.g., it would be awesome if I could do something like

x = init()
@startbenchmark SUITE setup=(y=deepcopy(x))
  @yieldbenchmeasure "muttfunc1" muttfunc1!(y)
  @yieldbenchmeasure "muttfunc2" muttfunc2!(y)
  @yieldbenchmeasure "muttfunc3" muttfunc3!(y)
end

Ofc it’s understandable they will all need to have the same benchmark config (e.g. samples, evals).
I hope my imaginery code is understandable of what it is meant to accomplish.

2

I keep on daydreaming here.
It might be interesting to think about it in terms of defining a BenchmarkFunctionModel, sort of like Turing.jl prepends @model in a function it samples.

E.g, something like

@benchmarkablegroup function myfunc(arg1, arg2, arg3)
  operations1(arg1, arg2, arg3)
  @yieldbenchmeasure "op2" operations2(arg1, arg2, arg3)
  operations3(arg1, arg2, arg3)
  @yieldbenchmeasure "op4" operations4(arg1, arg2, arg3)
end

So the previous desirable code could be stated as

@benchmarkablegroup function mybenchfunc(x)
  @yieldbenchmeasure "muttfunc1" muttfunc1!(x)
  @yieldbenchmeasure "muttfunc2" muttfunc2!(x)
  @yieldbenchmeasure "muttfunc3" muttfunc3!(x)
end

and theoretically you would call it with something like

julia> mybenchfunc(init())
3-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "muttfunc1" => Benchmark(evals=1, seconds=5.0, samples=10000)
  "muttfunc2" => Benchmark(evals=1, seconds=5.0, samples=10000)
  "muttfunc3" => Benchmark(evals=1, seconds=5.0, samples=10000)

So, as I believe that such functionality is not implemented I speculated a bit on a possible interface about it.
I’ll try to stop my delirium here. Let me know what you think

3

This sounds more like GitHub - KristofferC/TimerOutputs.jl: Formatted output of timed sections in Julia

1 Like
4

That’s a cool package and fairly close to what I am asking. But I think it doesn’t work with BenchmarkTools.jl (?)

1 Like