Why am I getting this error? And how to setup a benchmark inside a function? In the global scope this works:
julia> using BenchmarkTools
julia> let
a = rand(10)
setup = begin
x = deepcopy(a)
end
t1 = @benchmark sum(x) setup=setup evals=1
end
ERROR: UndefVarError: setup not defined
...
(the most annoying is not being able to setup within a for loop, actually, for the same scoping reason)
Iβm guessing that @benchmark expects a setup expression which can be parsed when the macro is expanded (that is, before runtime).
In your example, setup is the result of the x = deepcopy(a) expression, which is just the array x itself. The @benchmark macro has no way to know what to do with this array (which by the way is not known at macro expansion time). In the global scope, things βworkβ simply because the begin block defines the x array globally, and the setup expression is just not used.
Maybe there are better ways to do this, but I guess one option is to define a setup function:
using BenchmarkTools
function bench(a)
setup() = deepcopy(a)
@benchmark sum(x) setup=(x = $setup()) evals=1
end
a = rand(10)
bench(a)
But here you are interpolating setup(), will that be evaluated again for every new benchmark evaluation and sample? The point is that I need the array to be reset to the initial value before each run.
(in that example I think I would need to add evals=1)
Yeah, that is the solution (the interpolation of a inside the setup). My problem was more specifically this one (since I was initially trying to use the setup as you are suggesting here):
This works:
julia> let
a = rand(10000)
b = rand(10000)
@benchmark sort!(x + y) setup=(x = deepcopy($a); y = deepcopy($b)) evals=1
end
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
Range (min β¦ max): 436.213 ΞΌs β¦ 848.002 ΞΌs β GC (min β¦ max): 0.00% β¦ 43.88%
Time (median): 445.743 ΞΌs β GC (median): 0.00%
Time (mean Β± Ο): 449.586 ΞΌs Β± 25.089 ΞΌs β GC (mean Β± Ο): 0.13% Β± 1.74%
βββ ββββ β β
βββββββββββββ ββββββββββββββββββββββββ β ββ β β ββ β ββββββββββββ β βββ β
436 ΞΌs Histogram: log(frequency) by time 552 ΞΌs <
Memory estimate: 78.17 KiB, allocs estimate: 2.
But without interpolating it doesnβt (with only one variable it runs, though, that is what was confusing):
julia> let
a = rand(10000)
b = rand(10000)
@benchmark sort!(x + y) setup=(x = deepcopy(a); y = deepcopy(b)) evals=1
end
ERROR: UndefVarError: b not defined
I suppose interpolating the the wright thing to do there. I donβt really understand when that is necessary, when notβ¦ until now I thought that the setup variables didnβt require that. Probably I need to learn some basics of metaprograming to understand that for goodβ¦