# Identical functions repeated benchmarks show systematic differences

**URL:** <https://discourse.julialang.org/t/identical-functions-repeated-benchmarks-show-systematic-differences/65345>\
**Category:** Performance\
**Tags:** question, sort\
**Created:** [July 27, 2021, 4:08am UTC](https://discourse.julialang.org/t/identical-functions-repeated-benchmarks-show-systematic-differences/65345 "2021-07-27T04:08:27Z")\
**Posts on this page:** 1\
**Showing post:** 36

<div class="post-metadata">

**Author:** ![viraltux](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/viraltux/32/15236_2.png) [@viraltux](https://discourse.julialang.org/u/viraltux)\
**Post date:** [August 2, 2021, 11:54am UTC](https://discourse.julialang.org/t/identical-functions-repeated-benchmarks-show-systematic-differences/65345/36 "2021-08-02T11:54:13Z")

</div>

> [@Sukera](#):
>
> …but rather you’re timing that systemic difference you’re claiming to be an artifact of `BenchmarkTools`

I don’t think anybody is claiming `BenchmarkTools` is introducing any artifacts, rather that `BenchmarkTools` do not account for the Machine/OS artifacts when comparing benchmarks.

> [@Sukera](#):
>
> …and trying to guess what the user wanted in the first place is going to make it much harder to compare different invocations, not easier.

What the users typically may want when using `BenchmarkTools` is to answer these questions:

1. Is **A** faster than **B**?
2. By how much?
3. How confident are we on those differences?

None of these questions are directly addressed by BenchmarkTools as of today.

> [@Sukera](#):
>
> I personally don’t use `judge` for precisely that reason, because it can’t tell me on a per-function basis what’s going on with micro to nanosecond differences, but it’s important to note that `judge` is not intended for that puprpose either. It can only tell me that there _is_ a difference, but not where it comes from or why it’s there in the first place.

`judge` cannot tell you if there is a **real** difference and can not even tell you in which direction the **real** difference goes.

My point though using the **`deepcopy/nice`** settings is not to prove that that’s the best way to do things -in fact I thought about a few other ways do diminish the effect of Machine/OS artifacts that I didn’t use in that example, but rather that **yes we can** reliably detect (or in this case not to detect if there are none) those micro differences.

Anyway, this conversation was really interesting and I have learnt a lot, I found specially useful your link to `leaky abstractions` thank you again @Sukera . However since this Topic has been resolved on the reasons why there are systematic differences for the same function, I think it’s best that I end my participation in this thread and perhaps I open a new Topic in the future with very specific examples of what I am trying to convey.

---

_[View the full topic](https://discourse.julialang.org/t/identical-functions-repeated-benchmarks-show-systematic-differences/65345)._