# Setup \`@benchmark\` inside function: setup not defined

**URL:** <https://discourse.julialang.org/t/setup-benchmark-inside-function-setup-not-defined/73251>\
**Category:** General Usage\
**Tags:** benchmarktools\
**Created:** [December 17, 2021, 11:58am UTC](https://discourse.julialang.org/t/setup-benchmark-inside-function-setup-not-defined/73251 "2021-12-17T11:58:20Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [December 17, 2021, 11:58am UTC](https://discourse.julialang.org/t/setup-benchmark-inside-function-setup-not-defined/73251/1 "2021-12-17T11:58:20Z")

</div>

Why am I getting this error? And how to `setup` a benchmark inside a function? In the global scope this works:

```julia
julia> using BenchmarkTools

julia> let 
           a = rand(10)
           setup = begin
               x = deepcopy(a)
           end
           t1 = @benchmark sum(x) setup=setup evals=1
       end
ERROR: UndefVarError: setup not defined
...

```

(the most annoying is not being able to setup within a `for` loop, actually, for the same scoping reason)

---

<div class="post-metadata">

**Author:** ![jipolanco](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jipolanco/32/12129_2.png) [@jipolanco](https://discourse.julialang.org/u/jipolanco)\
**Post date:** [December 17, 2021, 1:01pm UTC](https://discourse.julialang.org/t/setup-benchmark-inside-function-setup-not-defined/73251/2 "2021-12-17T13:01:52Z")

</div>

I’m guessing that `@benchmark` expects a `setup` expression which can be parsed when the macro is expanded (that is, before runtime).

In your example, `setup` is the result of the `x = deepcopy(a)` expression, which is just the array `x` itself. The `@benchmark` macro has no way to know what to do with this array (which by the way is not known at macro expansion time). In the global scope, things “work” simply because the `begin` block defines the `x` array globally, and the `setup` expression is just not used.

Maybe there are better ways to do this, but I guess one option is to define a setup function:

```julia
using BenchmarkTools

function bench(a)
    setup() = deepcopy(a)
    @benchmark sum(x) setup=(x = $setup()) evals=1
end

a = rand(10)
bench(a)

```

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [December 17, 2021, 1:09pm UTC](https://discourse.julialang.org/t/setup-benchmark-inside-function-setup-not-defined/73251/3 "2021-12-17T13:09:04Z")

</div>

> [@jipolanco](#):
>
> ` @benchmark sum(x) setup=(x = $setup())`

But here you are interpolating `setup()`, will that be evaluated again for every new benchmark evaluation and sample? The point is that I need the array to be reset to the initial value before each run.

(in that example I think I would need to add `evals=1`)

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [December 17, 2021, 1:18pm UTC](https://discourse.julialang.org/t/setup-benchmark-inside-function-setup-not-defined/73251/4 "2021-12-17T13:18:45Z")

</div>

```julia
julia> let 
           a = rand(10)
           setup = quote
               x = deepcopy($a)
           end
           t1 = @eval @benchmark sum(x) setup=$setup evals=1
       end
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max): 17.000 ns … 132.000 ns ┊ GC (min … max): 0.00% … 0.00%
 Time (median): 26.000 ns ┊ GC (median): 0.00%
 Time (mean ± σ): 24.744 ns ± 3.659 ns ┊ GC (mean ± σ): 0.00% ± 0.00%

                             ▇ ▂ █ ▄           
  ▄▁▁▁█▁▁▁▁▃▁▁▁▂▁▁▁▁▂▁▁▁▁▂▁▁▁█▁▁▁▁█▁▁▁▃▁▁▁▁█▁▁▁▁█▁▁▁█▁▁▁▁▆▁▁▁▃ ▃
  17 ns Histogram: frequency by time 30 ns <

 Memory estimate: 0 bytes, allocs estimate: 0.

julia> let 
           a = rand(10)
           @benchmark sum(x) setup=(x = deepcopy($a)) evals=1
       end
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max): 17.000 ns … 17.739 μs ┊ GC (min … max): 0.00% … 0.00%
 Time (median): 24.000 ns ┊ GC (median): 0.00%
 Time (mean ± σ): 26.664 ns ± 177.178 ns ┊ GC (mean ± σ): 0.00% ± 0.00%

                             █ █                    
  ▂▁▁▁▃▁▁▁▁▂▁▁▁▂▁▁▁▁▂▁▁▁▁▂▁▁▁█▁▁▁▁█▁▁▁▃▁▁▁▁█▁▁▁▁▄▁▁▁▃▁▁▁▁▃▁▁▁▃ ▂
  17 ns Histogram: frequency by time 30 ns <

 Memory estimate: 0 bytes, allocs estimate: 0.

```

---

<div class="post-metadata">

**Author:** ![jipolanco](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jipolanco/32/12129_2.png) [@jipolanco](https://discourse.julialang.org/u/jipolanco)\
**Post date:** [December 17, 2021, 1:33pm UTC](https://discourse.julialang.org/t/setup-benchmark-inside-function-setup-not-defined/73251/5 "2021-12-17T13:33:03Z")

</div>

> [@lmiq](#):
>
> (in that example I think I would need to add `evals=1` )

Oops, I removed it for some reason… Right, you would need the `evals=1`.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [December 17, 2021, 1:42pm UTC](https://discourse.julialang.org/t/setup-benchmark-inside-function-setup-not-defined/73251/6 "2021-12-17T13:42:43Z")

</div>

> [@giordano](#):
>
> ` @benchmark sum(x) setup=(x = deepcopy($a)) evals=1`

Yeah, that is the solution (the interpolation of `a` inside the setup). My problem was more specifically this one (since I was initially trying to use the setup as you are suggesting here):

This works:

```julia
julia> let 
           a = rand(10000)
           b = rand(10000)
           @benchmark sort!(x + y) setup=(x = deepcopy($a); y = deepcopy($b)) evals=1
       end
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max): 436.213 μs … 848.002 μs ┊ GC (min … max): 0.00% … 43.88%
 Time (median): 445.743 μs ┊ GC (median): 0.00%
 Time (mean ± σ): 449.586 μs ± 25.089 μs ┊ GC (mean ± σ): 0.13% ± 1.74%

  ▂▆▅▃▇█▅▂ ▂
  █████████▇▇█▅▇█▇▆▆▆▆▇▆▇███▇█▇▇▆▆▆▆▇▆▅▅▄▅▅▅▄▅▅▆███▇▆▄▄▃▄▃▅▅▃▁▄ █
  436 μs Histogram: log(frequency) by time 552 μs <

 Memory estimate: 78.17 KiB, allocs estimate: 2.

```

But without interpolating it doesn’t (with only one variable it runs, though, that is what was confusing):

```julia
julia> let 
           a = rand(10000)
           b = rand(10000)
           @benchmark sort!(x + y) setup=(x = deepcopy(a); y = deepcopy(b)) evals=1
       end
ERROR: UndefVarError: b not defined

```

I suppose interpolating the the wright thing to do there. I don’t really understand when that is necessary, when not… until now I thought that the setup variables didn’t require that. Probably I need to learn some basics of metaprograming to understand that for good…