# Tutorial: How to benchmark and profile your code?

**URL:** <https://discourse.julialang.org/t/tutorial-how-to-benchmark-and-profile-your-code/57790>\
**Category:** Performance\
**Tags:** blog-post\
**Created:** [March 23, 2021, 3:20pm UTC](https://discourse.julialang.org/t/tutorial-how-to-benchmark-and-profile-your-code/57790 "2021-03-23T15:20:25Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Wikunia](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wikunia/32/2180_2.png) [@Wikunia](https://discourse.julialang.org/u/Wikunia)\
**Post date:** [March 23, 2021, 3:20pm UTC](https://discourse.julialang.org/t/tutorial-how-to-benchmark-and-profile-your-code/57790/1 "2021-03-23T15:20:25Z")

</div>

Interested in benchmarking and profiling your code?  
My new blog post walks you through it from highlevel benchmarking to getting deeper with profiling tools.  
It’s quite high level so I avoided explaining the different lower level macros there but will do this in another post if interested:

> **[Benchmarking and Profiling Julia Code](https://opensourc.es/blog/benchmarking-and-profiling-julia-code/)**
>
> Find out how to benchmark and profile your Julia code! Find the spots that aren't as fast as expected to run with the speed.

Let me know your thoughts and enjoy reading 🙂

---

<div class="post-metadata">

**Author:** ![sijo](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@sijo](https://discourse.julialang.org/u/sijo)\
**Post date:** [March 23, 2021, 4:07pm UTC](https://discourse.julialang.org/t/tutorial-how-to-benchmark-and-profile-your-code/57790/2 "2021-03-23T16:07:29Z")

</div>

Very nice! A few remarks:

- You might want to rename this topic: it sounds like you’re asking for help profiling your code… maybe “A new tutorial on benchmarking and profiling” or similar?

- Regarding the first flamegraph screenshots and this part:

- The last runtime plot which “looks quite funny” as you say, can make the reader skeptical that the third solution is really doing what it should do… Maybe a good opportunity to show that a logarithmic scale can be useful?

EDIT: just to clarify, for the first remark I meant the title here on Discourse.

---

<div class="post-metadata">

**Author:** ![Wikunia](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wikunia/32/2180_2.png) [@Wikunia](https://discourse.julialang.org/u/Wikunia)\
**Post date:** [March 23, 2021, 4:16pm UTC](https://discourse.julialang.org/t/tutorial-how-to-benchmark-and-profile-your-code/57790/3 "2021-03-23T16:16:40Z")

</div>

Thanks for your thoughts. Will add those!

---

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [March 23, 2021, 4:34pm UTC](https://discourse.julialang.org/t/tutorial-how-to-benchmark-and-profile-your-code/57790/4 "2021-03-23T16:34:08Z")

</div>

In this code

```julia
    # convert from nano seconds to seconds
    push!(ys, mean(t).time / 10^9)

```

Why are you using `mean`? It’s inconsistent with `@btime` behaviour which uses `min`. And it behave worse compared to `median` which is my second choice in such estimations.

---

<div class="post-metadata">

**Author:** ![Wikunia](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wikunia/32/2180_2.png) [@Wikunia](https://discourse.julialang.org/u/Wikunia)\
**Post date:** [March 23, 2021, 4:45pm UTC](https://discourse.julialang.org/t/tutorial-how-to-benchmark-and-profile-your-code/57790/5 "2021-03-23T16:45:50Z")

</div>

I find `min` a bit strange but it depends on what you want to measure I guess. Is `mean` wrong?  
I chose `mean` to have the average running time of the function. One could add error bars around it in the plot.

---

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [March 23, 2021, 6:27pm UTC](https://discourse.julialang.org/t/tutorial-how-to-benchmark-and-profile-your-code/57790/6 "2021-03-23T18:27:01Z")

</div>

Well, the consensus is that `min` is the most adequate metric to measure actual code performance because the time of the code execution is always “time of the code itself + some random nonnegative noise from the operating system”. Since the second term is always nonnegative, when you take `min` you’ll get the closest estimate to the real time of the code execution. `Median` is slightly worse, `mean` is the worst of them all, since it is very skewed. Imagine, that in 10 runs you get 9 measurements with the time 1ms and 1 with the time 10s. `Mean` time would be 1s, which is definitely not representative of the actual execution time.

But anyway, whether you agree with it or not, it’s inconsistent to use and compare `@btime` and `mean` to profile the same code. It should be either one or another.

---

<div class="post-metadata">

**Author:** ![Wikunia](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wikunia/32/2180_2.png) [@Wikunia](https://discourse.julialang.org/u/Wikunia)\
**Post date:** [March 23, 2021, 6:30pm UTC](https://discourse.julialang.org/t/tutorial-how-to-benchmark-and-profile-your-code/57790/7 "2021-03-23T18:30:33Z")

</div>

Thanks for the clarification @Skoffer . Will make the changes accordingly.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [March 24, 2021, 12:36am UTC](https://discourse.julialang.org/t/tutorial-how-to-benchmark-and-profile-your-code/57790/8 "2021-03-24T00:36:19Z")

</div>

> [@Skoffer](#):
>
> “time of the code itself + some random nonnegative noise

Once I benchmarked a code putting my laptop in the freezer. It was clearly faster. I will test that again and compare the minimum, median and average times obtained relative to room temperature, hopping to show that thermodynamic noise, not only operating system noise, enters into the equation.

---

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [March 24, 2021, 4:10am UTC](https://discourse.julialang.org/t/tutorial-how-to-benchmark-and-profile-your-code/57790/9 "2021-03-24T04:10:29Z")

</div>

Surely my statement is simplification. Another reason for “negative operating system time” can be governor management (I hope this term is correct). Operating system can change cpu frequency on demand, so it is possible, that during benchmark frequency can go up and overall execution time decrease. So yes, this formula is simplification.

---

<div class="post-metadata">

**Author:** ![jzr](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@jzr](https://discourse.julialang.org/u/jzr)\
**Post date:** [March 24, 2021, 5:23am UTC](https://discourse.julialang.org/t/tutorial-how-to-benchmark-and-profile-your-code/57790/10 "2021-03-24T05:23:26Z")

</div>

See: [Minimum times tend to mislead when benchmarking](https://tratt.net/laurie/blog/entries/minimum_times_tend_to_mislead_when_benchmarking.html)

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [March 24, 2021, 6:27am UTC](https://discourse.julialang.org/t/tutorial-how-to-benchmark-and-profile-your-code/57790/11 "2021-03-24T06:27:21Z")

</div>

See this paper conclusion: [results suggest that using the minimum estimator for the true run time of a benchmark, rather than the mean or median, is robust to non-ideal statistics and also provides the smallest error.](https://arxiv.org/abs/1608.04295)

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [March 24, 2021, 7:20am UTC](https://discourse.julialang.org/t/tutorial-how-to-benchmark-and-profile-your-code/57790/12 "2021-03-24T07:20:46Z")

</div>

> [@Skoffer](#):
>
> the consensus is that `min` is the most adequate metric to measure actual code performance because the time of the code execution is always “time of the code itself + some random nonnegative noise from the operating system”

Or, possibly, garbage collection, which occurs in bursts. If some GC is inevitable, it is reasonable to use the median too because it gives you more realistic timing for practical purposes.

These are just informative statistics, there isn’t a single best one. That said, if you have to pick one, then in general minimum is a reasonable choice.