# BenchmarkTools theory and practice

**URL:** <https://discourse.julialang.org/t/benchmarktools-theory-and-practice/5728>\
**Category:** General Usage\
**Tags:** benchmark, benchmarktools\
**Created:** [September 5, 2017, 2:23pm UTC](https://discourse.julialang.org/t/benchmarktools-theory-and-practice/5728 "2017-09-05T14:23:30Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [September 5, 2017, 2:23pm UTC](https://discourse.julialang.org/t/benchmarktools-theory-and-practice/5728/1 "2017-09-05T14:23:31Z")

</div>

I understand that BenchmarkTools’ job is a hard one, but is it expected that I frequently see several regressions while rerunning the same benchmark (without code change)? Is there some theory that (under certain assumptions) predict how often I should see a false-positive?

The standard benchmarking procedure is, AFAICT:

```julia
bench = @benchmarkable myfun()
tune!(bench)
first_run = run(bench)
# Change code
second_run = run(bench)
judge(second_run, first_run)

```

For reasons I won’t explain here (serialization), I have to do it like this in my package:

```julia
first_run = @benchmark myfun()
# Change code
bench = @benchmarkable myfun()
loadparams!(bench, params(first_run))
second_run = run(bench)
judge(second_run, first_run)

```

Is that equivalent? Furthermore, shouldn’t there be a `warmup(bench)` call before `second_run`? It seems like a reasonable thing to do, to get rid of the JIT from the code change, but [the source](https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/src/execution.jl) doesn’t seem to do that.

cc. @jrevels

---

<div class="post-metadata">

**Author:** ![jrevels](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jrevels/32/10393_2.png) [@jrevels](https://discourse.julialang.org/u/jrevels)\
**Post date:** [September 5, 2017, 3:29pm UTC](https://discourse.julialang.org/t/benchmarktools-theory-and-practice/5728/2 "2017-09-05T15:29:05Z")

</div>

> Is that equivalent?

Yes, they should be (minus JIT overhead for the second run).

For reference, a lot of the theory behind BenchmarkTools is described in [this paper](https://arxiv.org/abs/1608.04295).

> Furthermore, shouldn’t there be a warmup(bench) call before second\_run

Yes. It’s up to the user to add this, though. We should probably add something in the docs that says “if you don’t run the tuning process on a benchmark, and you care about warming up the benchmark to get rid of JIT overhead, you should manually call `warmup` before running the benchmark.”

> It seems like a reasonable thing to do, to get rid of the JIT from the code change, but the source doesn’t seem to do that.

This was a purposeful decision.

For “power users” (e.g. folks that use `@benchmarkable` instead `@benchmark`), BenchmarkTools should always respect user settings instead of making decisions for the user. This is because BenchmarkTools doesn’t have enough knowledge to decide in the general case whether it’s “correct” to execute the benchmark kernel “a hidden extra time” for the sake of getting rid of JIT overhead (e.g. the kernel could be non-idempotent or have side effects).

In the more naive use case (e.g. `@btime` and `@benchmark`), the tuning process takes care of JIT overhead for the users.