# BenchmarkTools with simple, fast-running function

**URL:** <https://discourse.julialang.org/t/benchmarktools-with-simple-fast-running-function/21049>\
**Category:** New to Julia\
**Created:** [February 21, 2019, 2:55pm UTC](https://discourse.julialang.org/t/benchmarktools-with-simple-fast-running-function/21049 "2019-02-21T14:55:51Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)\
**Post date:** [February 21, 2019, 2:55pm UTC](https://discourse.julialang.org/t/benchmarktools-with-simple-fast-running-function/21049/1 "2019-02-21T14:55:51Z")

</div>

I thought that BenchmarkTools was designed to be able to test simple, fast-running functions by evaluating them multiple times to avoid issues with the timing precision.

Using the example from the README, it doesn’t appear to me that it’s actually executing the function for each evaluation. Am I misunderstanding something?

```julia
julia> @benchmark sin(1) evals=1
BenchmarkTools.Trial:
  memory estimate: 0 bytes
  allocs estimate: 0
  --------------
  minimum time: 18.000 ns (0.00% GC)
  median time: 20.000 ns (0.00% GC)
  mean time: 19.785 ns (0.00% GC)
  maximum time: 39.000 ns (0.00% GC)
  --------------
  samples: 10000
  evals/sample: 1

julia> @benchmark sin(1) evals=1000
BenchmarkTools.Trial:
  memory estimate: 0 bytes
  allocs estimate: 0
  --------------
  minimum time: 0.017 ns (0.00% GC)
  median time: 0.020 ns (0.00% GC)
  mean time: 0.020 ns (0.00% GC)
  maximum time: 0.035 ns (0.00% GC)
  --------------
  samples: 10000
  evals/sample: 1000

julia> @benchmark sin(1) evals=1000000
BenchmarkTools.Trial:
  memory estimate: 0 bytes
  allocs estimate: 0
  --------------
  minimum time: 0.001 ns (0.00% GC)
  median time: 0.001 ns (0.00% GC)
  mean time: 0.001 ns (0.00% GC)
  maximum time: 0.001 ns (0.00% GC)
  --------------
  samples: 10000
  evals/sample: 1000000

```

---

<div class="post-metadata">

**Author:** ![bennedich](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bennedich/32/4894_2.png) [@bennedich](https://discourse.julialang.org/u/bennedich)\
**Post date:** [February 21, 2019, 3:15pm UTC](https://discourse.julialang.org/t/benchmarktools-with-simple-fast-running-function/21049/2 "2019-02-21T15:15:40Z")

</div>

Benchmarking is often tricky to get right for tiny functions, since Julia is so good at optimizing your code. Times of ~0.01 ns are the result of the compiler replacing your expression with a constant, and you end up benchmarking nothing at all. (1 nanosecond is ~3 CPU clock cycles on a 3 GHz computer, so 0.01 ns is not enough to do anything.)

To get around that, I usually try to structure the benchmark expression in such a way that the compiler can’t constant-fold it or cheat in any other way. In the example below, I also chose a vector large enough that the CPU can’t learn the branching behavior.

```julia
julia> v = rand(100_000);

julia> a = similar(v);

julia> @btime $a .= sin.($v);
  626.487 μs (0 allocations: 0 bytes)

```

100k calls in 626 μs equals around 6.26 ns per call to `sin` (if called through broadcast).

And yes, the example in the BenchmarkTools manual is now broken, and it would be great if they could talk more about these difficulties. Cf. this issue:

[https://github.com/JuliaCI/BenchmarkTools.jl/issues/130](https://github.com/JuliaCI/BenchmarkTools.jl/issues/130)

---

<div class="post-metadata">

**Author:** ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)\
**Post date:** [February 21, 2019, 9:29pm UTC](https://discourse.julialang.org/t/benchmarktools-with-simple-fast-running-function/21049/3 "2019-02-21T21:29:51Z")

</div>

Thanks. I had suspected that but thought that because it was being used as an example, BenchmarkTools had some way to disable the optimization. For example, Rust has `black_box` that forces evaluation.

I’ve created a issue to hopefully avoid future confusion:  
[https://github.com/JuliaCI/BenchmarkTools.jl/issues/134](https://github.com/JuliaCI/BenchmarkTools.jl/issues/134)

---

<div class="post-metadata">

**Author:** ![bennedich](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bennedich/32/4894_2.png) [@bennedich](https://discourse.julialang.org/u/bennedich)\
**Post date:** [February 21, 2019, 9:36pm UTC](https://discourse.julialang.org/t/benchmarktools-with-simple-fast-running-function/21049/4 "2019-02-21T21:36:58Z")

</div>

I think [this PR](https://github.com/JuliaCI/BenchmarkTools.jl/pull/92) was meant to address that (disabling optimizations), but unfortunately it was never completed.