Setup `@benchmark` inside function: setup not defined

General Usage
1

Why am I getting this error? And how to setup a benchmark inside a function? In the global scope this works:

julia> using BenchmarkTools

julia> let 
           a = rand(10)
           setup = begin
               x = deepcopy(a)
           end
           t1 = @benchmark sum(x) setup=setup evals=1
       end
ERROR: UndefVarError: setup not defined
...

(the most annoying is not being able to setup within a for loop, actually, for the same scoping reason)

2

I’m guessing that @benchmark expects a setup expression which can be parsed when the macro is expanded (that is, before runtime).

In your example, setup is the result of the x = deepcopy(a) expression, which is just the array x itself. The @benchmark macro has no way to know what to do with this array (which by the way is not known at macro expansion time). In the global scope, things β€œwork” simply because the begin block defines the x array globally, and the setup expression is just not used.

Maybe there are better ways to do this, but I guess one option is to define a setup function:

using BenchmarkTools

function bench(a)
    setup() = deepcopy(a)
    @benchmark sum(x) setup=(x = $setup()) evals=1
end

a = rand(10)
bench(a)
1 Like
3

But here you are interpolating setup(), will that be evaluated again for every new benchmark evaluation and sample? The point is that I need the array to be reset to the initial value before each run.

(in that example I think I would need to add evals=1)

4 
julia> let 
           a = rand(10)
           setup = quote
               x = deepcopy($a)
           end
           t1 = @eval @benchmark sum(x) setup=$setup evals=1
       end
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max):  17.000 ns … 132.000 ns  β”Š GC (min … max): 0.00% … 0.00%
 Time  (median):     26.000 ns               β”Š GC (median):    0.00%
 Time  (mean Β± Οƒ):   24.744 ns Β±   3.659 ns  β”Š GC (mean Β± Οƒ):  0.00% Β± 0.00%

                             β–‡    β–‚        β–ˆ        β–„           
  β–„β–β–β–β–ˆβ–β–β–β–β–ƒβ–β–β–β–‚β–β–β–β–β–‚β–β–β–β–β–‚β–β–β–β–ˆβ–β–β–β–β–ˆβ–β–β–β–ƒβ–β–β–β–β–ˆβ–β–β–β–β–ˆβ–β–β–β–ˆβ–β–β–β–β–†β–β–β–β–ƒ β–ƒ
  17 ns           Histogram: frequency by time           30 ns <

 Memory estimate: 0 bytes, allocs estimate: 0.

julia> let 
           a = rand(10)
           @benchmark sum(x) setup=(x = deepcopy($a)) evals=1
       end
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max):  17.000 ns …  17.739 ΞΌs  β”Š GC (min … max): 0.00% … 0.00%
 Time  (median):     24.000 ns               β”Š GC (median):    0.00%
 Time  (mean Β± Οƒ):   26.664 ns Β± 177.178 ns  β”Š GC (mean Β± Οƒ):  0.00% Β± 0.00%

                             β–ˆ             β–ˆ                    
  β–‚β–β–β–β–ƒβ–β–β–β–β–‚β–β–β–β–‚β–β–β–β–β–‚β–β–β–β–β–‚β–β–β–β–ˆβ–β–β–β–β–ˆβ–β–β–β–ƒβ–β–β–β–β–ˆβ–β–β–β–β–„β–β–β–β–ƒβ–β–β–β–β–ƒβ–β–β–β–ƒ β–‚
  17 ns           Histogram: frequency by time           30 ns <

 Memory estimate: 0 bytes, allocs estimate: 0.
1 Like
5

Oops, I removed it for some reason… Right, you would need the evals=1.

1 Like
6

Yeah, that is the solution (the interpolation of a inside the setup). My problem was more specifically this one (since I was initially trying to use the setup as you are suggesting here):

This works:

julia> let 
           a = rand(10000)
           b = rand(10000)
           @benchmark sort!(x + y) setup=(x = deepcopy($a); y = deepcopy($b)) evals=1
       end
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max):  436.213 ΞΌs … 848.002 ΞΌs  β”Š GC (min … max): 0.00% … 43.88%
 Time  (median):     445.743 ΞΌs               β”Š GC (median):    0.00%
 Time  (mean Β± Οƒ):   449.586 ΞΌs Β±  25.089 ΞΌs  β”Š GC (mean Β± Οƒ):  0.13% Β±  1.74%

  β–‚β–†β–…β–ƒβ–‡β–ˆβ–…β–‚                                                      β–‚
  β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‡β–‡β–ˆβ–…β–‡β–ˆβ–‡β–†β–†β–†β–†β–‡β–†β–‡β–ˆβ–ˆβ–ˆβ–‡β–ˆβ–‡β–‡β–†β–†β–†β–†β–‡β–†β–…β–…β–„β–…β–…β–…β–„β–…β–…β–†β–ˆβ–ˆβ–ˆβ–‡β–†β–„β–„β–ƒβ–„β–ƒβ–…β–…β–ƒβ–β–„ β–ˆ
  436 ΞΌs        Histogram: log(frequency) by time        552 ΞΌs <

 Memory estimate: 78.17 KiB, allocs estimate: 2.

But without interpolating it doesn’t (with only one variable it runs, though, that is what was confusing):

julia> let 
           a = rand(10000)
           b = rand(10000)
           @benchmark sort!(x + y) setup=(x = deepcopy(a); y = deepcopy(b)) evals=1
       end
ERROR: UndefVarError: b not defined

I suppose interpolating the the wright thing to do there. I don’t really understand when that is necessary, when not… until now I thought that the setup variables didn’t require that. Probably I need to learn some basics of metaprograming to understand that for good…