# BenchmarkTools setup isn't run between each iteration?
**URL:** https://discourse.julialang.org/t/benchmarktools-setup-isnt-run-between-each-iteration/36258
**Category:** General Usage
**Tags:** question, benchmarktools
**Created:** [March 20, 2020, 12:59pm UTC](https://discourse.julialang.org/t/benchmarktools-setup-isnt-run-between-each-iteration/36258 "2020-03-20T12:59:54Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@DNF](https://discourse.julialang.org/u/DNF)
#### Post date: [March 20, 2020, 12:59pm UTC](https://discourse.julialang.org/t/benchmarktools-setup-isnt-run-between-each-iteration/36258/1 "2020-03-20T12:59:54Z")
Can someone explain to me what’s going on here? I would not expect the error that happens for the short input vector:
```julia
julia> using BenchmarkTools
julia> @btime pop!(v) setup=(v=rand(1000));
8.295 ns (0 allocations: 0 bytes)
julia> @btime pop!(v) setup=(v=rand(100));
ERROR: ArgumentError: array must be non-empty
Stacktrace:
[1] pop! at ./array.jl:1078 [inlined]
[2] ##core#417(::Array{Float64,1}) at /Users/dnf/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:371
[3] ##sample#418(::BenchmarkTools.Parameters) at /Users/dnf/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:379
[4] sample at /Users/dnf/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:394 [inlined]
[5] #_lineartrial#44(::Int64, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(BenchmarkTools._lineartrial), ::BenchmarkTools.Benchmark{Symbol("##benchmark#416")}, ::BenchmarkTools.Parameters) at /Users/dnf/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:133
[6] _lineartrial(::BenchmarkTools.Benchmark{Symbol("##benchmark#416")}, ::BenchmarkTools.Parameters) at /Users/dnf/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:125
[7] #invokelatest#1 at ./essentials.jl:709 [inlined]
[8] invokelatest at ./essentials.jl:708 [inlined]
[9] #lineartrial#38 at /Users/dnf/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:33 [inlined]
[10] lineartrial at /Users/dnf/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:33 [inlined]
[11] #tune!#49(::Nothing, ::Float64, ::Float64, ::Bool, ::String, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(tune!), ::BenchmarkTools.Benchmark{Symbol("##benchmark#416")}, ::BenchmarkTools.Parameters) at /Users/dnf/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:209
[12] tune! at /Users/dnf/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:208 [inlined] (repeats 2 times)
[13] top-level scope at /Users/dnf/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:482
```
It seems like the `@btime` macro runs `pop!` multiple times, without doing the `setup` in-between.
Julia version 1.3.0, BenchmarkTools version 0.5.0.
---
### Author:  [@rdeits](https://discourse.julialang.org/u/rdeits)
#### Post date: [March 20, 2020, 1:02pm UTC](https://discourse.julialang.org/t/benchmarktools-setup-isnt-run-between-each-iteration/36258/2 "2020-03-20T13:02:33Z")
That’s right–it generally runs multiple evaluations for each setup call in order to get more reliable timing data. If you want to avoid that, you can pass `evals=1` to the `@btime` call, which will ensure that there is exactly one evaluation of your function per setup.
---
### Author:  [@DNF](https://discourse.julialang.org/u/DNF)
#### Post date: [March 20, 2020, 1:07pm UTC](https://discourse.julialang.org/t/benchmarktools-setup-isnt-run-between-each-iteration/36258/3 "2020-03-20T13:07:51Z")
Ouch, that means that
```julia
@btime sort!(x) setup=(x=rand(1000));
```
will return results from sorting an already sorted vector! Yikes! This is really bad, I had no idea, and thought setup was done for each iteration. How does this improve reliability? Won’t this completely invalidate every benchmark involving mutating functions?
BTW, how do I pass `evals=1` to `@btime`?
---
### Author:  [@rdeits](https://discourse.julialang.org/u/rdeits)
#### Post date: [March 20, 2020, 1:21pm UTC](https://discourse.julialang.org/t/benchmarktools-setup-isnt-run-between-each-iteration/36258/4 "2020-03-20T13:21:56Z")
Check out this section of the manual, which specifically uses `sort` as an example: [https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/doc/manual.md#setup-and-teardown-phases](https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/doc/manual.md#setup-and-teardown-phases)
---
### Author:  [@DNF](https://discourse.julialang.org/u/DNF)
#### Post date: [March 20, 2020, 1:55pm UTC](https://discourse.julialang.org/t/benchmarktools-setup-isnt-run-between-each-iteration/36258/5 "2020-03-20T13:55:44Z")
Thanks. It seems, though, that there is no obvious way to pass evals=1 directly to `@btime`, or I cannot figure it out.
This made the whole thing a lot more complicated than I knew. I have apparently given a lot of bad advice to people, and performed a number of meaningless benchmarks.
---
### Author:  [@rdeits](https://discourse.julialang.org/u/rdeits)
#### Post date: [March 20, 2020, 2:04pm UTC](https://discourse.julialang.org/t/benchmarktools-setup-isnt-run-between-each-iteration/36258/6 "2020-03-20T14:04:10Z")
`evals` is just another keyword argument, like `setup`:
```julia
julia> @btime sort!(x) setup=(x = rand(5)) evals=1
38.000 ns (0 allocations: 0 bytes)
5-element Array{Float64,1}:
0.27202634588420826
0.3043204769033667
0.38124617952216444
0.44094427015230697
0.4785747961346507
```
or, with more parentheses for clarity:
```julia
julia> @btime(sort!(x), setup=(x = rand(5)), evals=1)
43.000 ns (0 allocations: 0 bytes)
5-element Array{Float64,1}:
0.2818970773419529
0.6447645561336768
0.7669354473160204
0.789996892969342
0.8448189472593695
```
---
### Author:  [@lawless-m](https://discourse.julialang.org/u/lawless-m)
#### Post date: [November 17, 2021, 2:37pm UTC](https://discourse.julialang.org/t/benchmarktools-setup-isnt-run-between-each-iteration/36258/8 "2021-11-17T14:37:02Z")
this link is 404 now, should be
[https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/docs/src/manual.md#setup-and-teardown-phases](https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/docs/src/manual.md#setup-and-teardown-phases)