# How to specify the number of execution and the number of repetitions per execution in BenchmarkTools?
**URL:** https://discourse.julialang.org/t/how-to-specify-the-number-of-execution-and-the-number-of-repetitions-per-execution-in-benchmarktools/34514
**Category:** General Usage
**Tags:** question
**Created:** [February 12, 2020, 1:36pm UTC](https://discourse.julialang.org/t/how-to-specify-the-number-of-execution-and-the-number-of-repetitions-per-execution-in-benchmarktools/34514 "2020-02-12T13:36:21Z")
**Posts on this page:** 17
**Page:** 1
### Author:  [@PyDataBlog](https://discourse.julialang.org/u/PyDataBlog)
#### Post date: [February 12, 2020, 1:36pm UTC](https://discourse.julialang.org/t/how-to-specify-the-number-of-execution-and-the-number-of-repetitions-per-execution-in-benchmarktools/34514/1 "2020-02-12T13:36:21Z")
Hi everyone,
I’m trying to benchmark the speed of a function that I ported to Julia using BenchmarkTools.
In Python with `ipython`, I used `%timeit -n 7 test_speed(X)` to generate the benchmark speeds with this result;
```python
1min 6s ± 2.31 s per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
I want to replicate this benchmark trial to make a fair assessment so I tried this:
```julia
results = @benchmark test_speed(X) evals=7
```
with this result;
```julia
BenchmarkTools.Trial:
memory estimate: 15.91 GiB
allocs estimate: 84727240
--------------
minimum time: 34.098 s (58.05% GC)
median time: 34.098 s (58.05% GC)
mean time: 34.098 s (58.05% GC)
maximum time: 34.098 s (58.05% GC)
--------------
samples: 1
evals/sample: 7
```
I’m, however, not sure if that’s the correct way of doing it since the time is pretty much the same in terms of minimum, median etc.
Am I doing something wrong?
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [February 12, 2020, 1:40pm UTC](https://discourse.julialang.org/t/how-to-specify-the-number-of-execution-and-the-number-of-repetitions-per-execution-in-benchmarktools/34514/2 "2020-02-12T13:40:04Z")
I’m often confused regarding the difference between evals and samples, see if you can sense of it from here
[https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/doc/manual.md#benchmark-parameters](https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/doc/manual.md#benchmark-parameters)
It sounds like it to me that you should specify samples instead.
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [February 12, 2020, 1:50pm UTC](https://discourse.julialang.org/t/how-to-specify-the-number-of-execution-and-the-number-of-repetitions-per-execution-in-benchmarktools/34514/3 "2020-02-12T13:50:33Z")
Here’s the relevant explanation
> The reasoning behind our definition of “sample” may not be obvious to all readers. If the time to execute a benchmark is smaller than the resolution of your timing method, then a single evaluation of the benchmark will generally not produce a valid sample. In that case, one must approximate a valid sample by recording the total time `t` it takes to record `n` evaluations, and estimating the sample’s time per evaluation as `t/n` . For example, if a sample takes 1 second for 1 million evaluations, the approximate time per evaluation for that sample is 1 microsecond. It’s not obvious what the right number of evaluations per sample should be for any given benchmark, so BenchmarkTools provides a mechanism (the `tune!` method) to automatically figure it out for you.
Sonit seems in your case, you should leave evals at the default, 1, and set samples to 7
---
### Author:  [@PyDataBlog](https://discourse.julialang.org/u/PyDataBlog)
#### Post date: [February 12, 2020, 1:58pm UTC](https://discourse.julialang.org/t/how-to-specify-the-number-of-execution-and-the-number-of-repetitions-per-execution-in-benchmarktools/34514/4 "2020-02-12T13:58:16Z")
> [@baggepinnen](#):
>
> It sounds like it to me that you should specify samples instead.
I will try that and report. It’s really confusing for me as well. Trying `r = @benchmark test_speed(X) samples=7` changes nothing as I got this;
```julia
BenchmarkTools.Trial:
memory estimate: 15.99 GiB
allocs estimate: 86012859
--------------
minimum time: 34.815 s (58.44% GC)
median time: 34.815 s (58.44% GC)
mean time: 34.815 s (58.44% GC)
maximum time: 34.815 s (58.44% GC)
--------------
samples: 1
evals/sample: 1
```
Now I’m super confused 😭
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [February 12, 2020, 2:04pm UTC](https://discourse.julialang.org/t/how-to-specify-the-number-of-execution-and-the-number-of-repetitions-per-execution-in-benchmarktools/34514/5 "2020-02-12T14:04:54Z")
Hmm, you might have hit the time limit since you didn’t hit the sample threshold
> Defaults to `BenchmarkTools.DEFAULT_PARAMETERS.seconds = 5` .
You must set time limit to above 7 times the time you just measured.
BTW, there’s probably some room for optimization before you benchmark super accurately since you allocate a huge amount of memory.
---
### Author:  [@PyDataBlog](https://discourse.julialang.org/u/PyDataBlog)
#### Post date: [February 12, 2020, 2:10pm UTC](https://discourse.julialang.org/t/how-to-specify-the-number-of-execution-and-the-number-of-repetitions-per-execution-in-benchmarktools/34514/6 "2020-02-12T14:10:25Z")
> [@baggepinnen](#):
>
> You must set time limit to above 7 times the time you just measured.
>
> BTW, there’s probably some room for optimization before you benchmark super accurately since you allocate a huge amount of memory.
I’m increasing the seconds to 300 and yes, there is so much room for optimization since I’m a julia rookie. Even with that room, it’s actually blowing the python implementation away (takes less than half the time it does in Python).
I will report what the new benchmark results show.
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [February 12, 2020, 2:18pm UTC](https://discourse.julialang.org/t/how-to-specify-the-number-of-execution-and-the-number-of-repetitions-per-execution-in-benchmarktools/34514/7 "2020-02-12T14:18:34Z")
> [@PyDataBlog](#):
>
> Even with that room, it’s actually blowing the python implementation away (takes less than half the time it does in Python).
>
> I will report what the new benchmark results show.
Sweet, I bet you can make that at least 10x 😃
---
### Author:  [@PyDataBlog](https://discourse.julialang.org/u/PyDataBlog)
#### Post date: [February 12, 2020, 2:22pm UTC](https://discourse.julialang.org/t/how-to-specify-the-number-of-execution-and-the-number-of-repetitions-per-execution-in-benchmarktools/34514/8 "2020-02-12T14:22:00Z")
> [@baggepinnen](#):
>
> Sweet, I bet you can make that at least 10x 😃
New results after setting the seconds higher:
```julia
BenchmarkTools.Trial:
memory estimate: 14.17 GiB
allocs estimate: 74011729
--------------
minimum time: 28.436 s (56.82% GC)
median time: 32.221 s (58.45% GC)
mean time: 32.082 s (58.02% GC)
maximum time: 35.245 s (58.26% GC)
--------------
samples: 7
evals/sample: 1
```
Thanks for the tip. It would have been great it the seconds auto adjusted to the number of samples specified 🙂
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [February 12, 2020, 2:24pm UTC](https://discourse.julialang.org/t/how-to-specify-the-number-of-execution-and-the-number-of-repetitions-per-execution-in-benchmarktools/34514/9 "2020-02-12T14:24:43Z")
The time is a benchmarking budget, not a desired time, That’s why benchmarking stops when the first limit is met.
---
### Author:  [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)
#### Post date: [February 12, 2020, 3:36pm UTC](https://discourse.julialang.org/t/how-to-specify-the-number-of-execution-and-the-number-of-repetitions-per-execution-in-benchmarktools/34514/10 "2020-02-12T15:36:29Z")
FWIW, you don’t really need BenchmarkTools if your timing is on the order of seconds. You can just use `@time`.
---
### Author:  [@Skoffer](https://discourse.julialang.org/u/Skoffer)
#### Post date: [February 12, 2020, 3:46pm UTC](https://discourse.julialang.org/t/how-to-specify-the-number-of-execution-and-the-number-of-repetitions-per-execution-in-benchmarktools/34514/11 "2020-02-12T15:46:14Z")
Unlike `@time` BenchmarkTools computes statistics (median, mean time etc) and has nice representation of the result. And it’s easier to use one tool than switching from one to another.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [February 12, 2020, 4:26pm UTC](https://discourse.julialang.org/t/how-to-specify-the-number-of-execution-and-the-number-of-repetitions-per-execution-in-benchmarktools/34514/12 "2020-02-12T16:26:07Z")
You can use BenchmarkTools if preferred, but for long-running calculations where you did a run to compile things already there are diminishing returns to it.
You _will_ get a statistical sample, but it is usually a small one so the benefits are frequently marginal.
---
### Author:  [@PyDataBlog](https://discourse.julialang.org/u/PyDataBlog)
#### Post date: [February 13, 2020, 10:21am UTC](https://discourse.julialang.org/t/how-to-specify-the-number-of-execution-and-the-number-of-repetitions-per-execution-in-benchmarktools/34514/13 "2020-02-13T10:21:39Z")
> [@baggepinnen](#):
>
> Sweet, I bet you can make that at least 10x 😃
I upgraded to Julia 1.3.1 and the speed test is even better now:
```julia
BenchmarkTools.Trial:
memory estimate: 14.40 GiB
allocs estimate: 76011754
--------------
minimum time: 16.140 s (11.75% GC)
median time: 16.620 s (13.68% GC)
mean time: 16.810 s (14.36% GC)
maximum time: 17.806 s (16.92% GC)
--------------
samples: 7
evals/sample: 1
```
while the python implementation in a heavily optimized library like `sklearn` is almost 5x slower:
```julia
1min 21s ± 2.85 s per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
I still can’t believe it~ 😳 😲
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [February 13, 2020, 10:33am UTC](https://discourse.julialang.org/t/how-to-specify-the-number-of-execution-and-the-number-of-repetitions-per-execution-in-benchmarktools/34514/14 "2020-02-13T10:33:57Z")
That’s great news 🙂
I still think you can squeeze some extra performance out of julia by working on the allocations, even though I have no idea what code you are actually benchmarking 😛
---
### Author:  [@PyDataBlog](https://discourse.julialang.org/u/PyDataBlog)
#### Post date: [February 13, 2020, 10:54am UTC](https://discourse.julialang.org/t/how-to-specify-the-number-of-execution-and-the-number-of-repetitions-per-execution-in-benchmarktools/34514/15 "2020-02-13T10:54:14Z")
I’m cleaning up the code. I will make a post soon for optimization tips since I have never really dealt with low level languages so I have no clue about allocations (but I am eager to learn now) 🙂
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [February 13, 2020, 11:16am UTC](https://discourse.julialang.org/t/how-to-specify-the-number-of-execution-and-the-number-of-repetitions-per-execution-in-benchmarktools/34514/16 "2020-02-13T11:16:46Z")
Great 🙂 When you make such a post, make sure to include self-contained code such that anyone can just copy and paste it and get a timing result for themselves to improve upon. That’s a great way to ensure that you get a lot of people hooked on making the code faster 😛
---
### Author:  [@dpinol](https://discourse.julialang.org/u/dpinol)
#### Post date: [September 2, 2021, 9:19am UTC](https://discourse.julialang.org/t/how-to-specify-the-number-of-execution-and-the-number-of-repetitions-per-execution-in-benchmarktools/34514/17 "2021-09-02T09:19:57Z")
Updated link [https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/docs/src/manual.md#benchmark-parameters](https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/docs/src/manual.md#benchmark-parameters)