# Chairmarks.jl

**URL:** <https://discourse.julialang.org/t/chairmarks-jl/111096>\
**Category:** Package Announcements\
**Tags:** memory-allocation, benchmark, speed-optimization\
**Created:** [March 3, 2024, 4:16pm UTC](https://discourse.julialang.org/t/chairmarks-jl/111096 "2024-03-03T16:16:39Z")\
**Posts on this page:** 1\
**Showing post:** 52

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [March 8, 2024, 5:41pm UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/52 "2024-03-08T17:41:45Z")

</div>

> [@kristoffer.carlsson](#):
>
> It would be unfortunate if benchmark packages became a “race to the bottom” where finally the “fastest” benchmarking package would just wrap an alias for `@elapsed`.

I agree a lot with this!

And yet, there is a place for lower precision faster benchmarks, especially in REPL mode.

> [@kristoffer.carlsson](#):
>
> know the authors of BenchmarkTools (Jarett Revels and Jiahao Chen) wrote a paper about the benchmarking strategy employed ([[1608.04295] Robust benchmarking in noisy environments ](https://arxiv.org/abs/1608.04295)) in BenchmarkTools so clearly they thought about this a lot.

Thanks for linking that, interesting. And yet, reading it decreases my trust in benchmarktools, rather than increasing it. The core model is equation (1) on page 3. To excerpt:

> Let P\_0 be a deterministic benchmark program which consists of an instruction tape consisting of k instructions […]  
> Let \tau^{[i]} be the run time of instruction I^{i}. Then, the total run time of P\_0 can be written T\_{P\_0} = \Sigma^N\_{i=1}\tau^{[i]}.

This model may have been accurate in the late 80s, but today it is so indescribably wrong that it seriously makes me doubt benchmarktools.

Look, the CPU executes lots of instructions in parallel, has a giant reorder buffer, and has incredible amounts of microarchitectural state.

Micro-benchmarking is ~~all~~ chiefly about controlling the microarchitectural state such that it correctly simulates the real context you want to simulate.

Look at e.g. [this discussion on how the long memory of the branch predictor interacts with benchmarktools](https://discourse.julialang.org/t/psa-microbenchmarks-remember-branch-history/17436).

---

_[View the full topic](https://discourse.julialang.org/t/chairmarks-jl/111096)._