# Chairmarks.jl

**URL:** <https://discourse.julialang.org/t/chairmarks-jl/111096>\
**Category:** Package Announcements\
**Tags:** memory-allocation, benchmark, speed-optimization\
**Created:** [March 3, 2024, 4:16pm UTC](https://discourse.julialang.org/t/chairmarks-jl/111096 "2024-03-03T16:16:39Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [March 4, 2024, 5:50pm UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/22 "2024-03-04T17:50:41Z")

</div>

There’s no reason to deprecate BenchmarkTools.jl. It works completely fine, and there’s no 1-to-1 replacement for it anyways. Recommending that everyone switch to Chairmarks.jl seems incredibly premature given the stability of BenchmarkTools.jl

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [March 4, 2024, 5:54pm UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/23 "2024-03-04T17:54:11Z")

</div>

It is up to @gdalle this decision as the only maintainer of the package. If he doesn’t want to maintain it anymore, he can let people take his place or announce lack of maintainers.

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [March 4, 2024, 5:54pm UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/24 "2024-03-04T17:54:55Z")

</div>

Announcing a lack of maintainers does not need to involve deprecating a package.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [March 4, 2024, 5:56pm UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/25 "2024-03-04T17:56:44Z")

</div>

Sure. A warning during package load can certainly lead to deprecation, but that is not mandatory. What I want to know as an and user and package developer, is that the packages I am depending on are being actively maintained.

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [March 4, 2024, 6:33pm UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/26 "2024-03-04T18:33:25Z")

</div>

> [@juliohm](#):
>
> A warning during package load […]

Warnings during package load are super annoying. I definitely don’t want this.

Anyways, the topic here is Chairmarks.jl not when/how to announce a potential lack of maintainers.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [March 4, 2024, 6:46pm UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/27 "2024-03-04T18:46:45Z")

</div>

To be clear, I don’t see BenchmarkTools.jl going out of business anytime soon, and I don’t want to deprecate it either. I’m just using the opportunity of a new package announcement to highlight the lack of maintainer energy for this cornerstone of the ecosystem.  
Thus, even though Chairmarks.jl isn’t _meant_ as a replacement, perhaps the idea isn’t as crazy as it sounds.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [March 4, 2024, 7:31pm UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/28 "2024-03-04T19:31:23Z")

</div>

Totally understand it @gdalle. Please make it explicit in the README or elsewhere if you decide to step down from your maintenance role.

---

<div class="post-metadata">

**Author:** ![Lilith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lilith/32/27492_2.png) [@Lilith](https://discourse.julialang.org/u/Lilith)\
**Post date:** [March 4, 2024, 9:43pm UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/29 "2024-03-04T21:43:46Z")

</div>

Update: I’ve released 1.1.0! As requested, it now has interpolation! The interpolation is largely the same as BenchmarkTools, but has different strange interactions with constant propagation:

```julia
julia> @btime ifelse(false, sum(sqrt(i) for i in 1:10000), 0.0);
  2.041 ns (0 allocations: 0 bytes)

julia> @btime ifelse($(false), sum(sqrt(i) for i in 1:10000), 0.0);
  8.569 μs (0 allocations: 0 bytes)

julia> @btime ifelse($(true), sum(sqrt(i) for i in 1:10000), 0.0);
  8.569 μs (0 allocations: 0 bytes)

julia> @b ifelse(false, sum(sqrt(i) for i in 1:10000), 0.0)
0 ns

julia> @b ifelse($(false), sum(sqrt(i) for i in 1:10000), 0.0)
2.731 ns

julia> @b ifelse($(true), sum(sqrt(i) for i in 1:10000), 0.0)
10.083 μs

```

1.1.0 also has a whole new documentation UI using DocumenterVitepress.jl! Thanks to @asinghvi17 for making that possible by helping me set it up. It should be pretty straightforward to use DocumenterVitepress.jl in most packages, but they’re on version 0.0.10, so definitely not stable yet.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [March 5, 2024, 8:22am UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/30 "2024-03-05T08:22:33Z")

</div>

> [@juliohm](#):
>
> Please make it explicit in the README or elsewhere if you decide to step down from your maintenance role.

> [@BenchmarkTools needs a maintainer](https://discourse.julialang.org/t/benchmarktools-needs-a-maintainer/111178):
>
> Hi everyone, I have been maintaining BenchmarkTools.jl for about a year. I didn’t particularly want to do it, nor did the person in charge before me, but we did our best to keep it afloat. Now I am stepping down, and looking for someone to take over. Requirements: having some time available (not that much) knowing a little bit about metaprogramming Job benefits: the limitless pride of contributing to a central package in the Julia ecosystem Perspectives for improvement: drawing inspi…

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [March 5, 2024, 9:31am UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/31 "2024-03-05T09:31:32Z")

</div>

Thank you @gdalle. Can you please add this information to the top of the README on github so that anyone considering usage can learn about the lack of maintainers?

A short message explaining that there are no active maintainers and that you can assign this role to anyone interested would be super helpful.

Why do I care so much about this? People care about software status, especially in industry. I’ll no longer use BenchmarkTools.jl knowing it doesn’t have a maintainer, this information is crucial.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [March 5, 2024, 9:32am UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/32 "2024-03-05T09:32:26Z")

</div>

> [@Lilith](#):
>
> Chairmarks runs benchmarks in the scope of the calling code while BenchmarkTools runs it in global scope.

> [@Lilith](#):
>
> `f(n) = @b rand(n) seconds=.001`

1. Am I correct to think that `@b` can be written much like `@time`, and the only remaining reason to `$`-interpolate is to do it for global variables to get around type instability?
2. Is code being hoisted out of the benchmark loop still a possibility like in BenchmarkTools? The change from running it all in the global scope with heavy interpolation is throwing me off regarding that.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [March 5, 2024, 9:37am UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/33 "2024-03-05T09:37:38Z")

</div>

First I’d like to see if any volunteers pop up here. BenchmarkTools is widely used, there is no need to cause uproar by displaying an “unmaintained” status only to roll it back a couple days later

---

<div class="post-metadata">

**Author:** ![Lilith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lilith/32/27492_2.png) [@Lilith](https://discourse.julialang.org/u/Lilith)\
**Post date:** [March 5, 2024, 2:37pm UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/34 "2024-03-05T14:37:10Z")

</div>

1. Yes.
2. Yes. The same “people doing benchmarks want the compiler to emit code that does useless work”/“everyone else wants the compiler not to do that” tension still exists.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [March 5, 2024, 3:07pm UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/35 "2024-03-05T15:07:24Z")

</div>

BenchmarkTools calls the garbage collector before running a benchmark and between trials. This does not seem to do that. Are you saying that that feature is useless?

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [March 5, 2024, 3:13pm UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/36 "2024-03-05T15:13:57Z")

</div>

> [@cgeoga](#):
>
> My biggest pain point with `BenchmarkTools` is that `@ballocated` in particular feels way slower than it needs to be.

`@ballocated` doesn’t make sense in the first place. It just runs `@benchmark` and grabs the minimum amount of memory. But why would you need to run a function 1000 times to know how much memory it uses? It is not noisy like execution time. Just use `@time` or `@elapsed`.

---

<div class="post-metadata">

**Author:** ![Lilith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lilith/32/27492_2.png) [@Lilith](https://discourse.julialang.org/u/Lilith)\
**Post date:** [March 5, 2024, 3:15pm UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/37 "2024-03-05T15:15:49Z")

</div>

> Are you saying that that feature is useless?

No.

---

<div class="post-metadata">

**Author:** ![matthias314](https://avatars.discourse-cdn.com/v4/letter/m/a88e4f/32.png) [@matthias314](https://discourse.julialang.org/u/matthias314)\
**Post date:** [March 7, 2024, 9:33pm UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/38 "2024-03-07T21:33:05Z")

</div>

The new package looks great! However, I’m getting confused when I compare the results to BenchmarkTools.jl for microbenchmarks. Here is an example, inspired by [this thread](https://discourse.julialang.org/t/how-to-shift-bits-faster/19405):

```julia
f(x, n) = x << n
g(x, n) = x << (n & 63)

```

On my laptop I get

```julia
julia> x = UInt128(1); n = 1;
julia> @btime f($x, $n);
  4.371 ns (0 allocations: 0 bytes)
julia> @b f($x, $n)
1.341 ns
julia> @btime g($x, $n);
  2.736 ns (0 allocations: 0 bytes)
julia> @b g($x, $n)
1.341 ns

```

It’s not just that the absolute timings are different. According to `@btime`, `g` is faster than `f`, but `@b` doesn’t see any difference. How come?

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [March 7, 2024, 9:37pm UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/39 "2024-03-07T21:37:22Z")

</div>

I do think this feature is worse than useless when comparing versions of code that allocate different amounts.  
But if two versions of code have the exact same allocation behavior (which is likely when making minor changes!), it is useful to try and ignore everything other than the difference (such as the GC), so it definitely has merit.

Ideally, folks can make their micro-benchmarks non-allocating, but that’s not always practical.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [March 7, 2024, 11:08pm UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/40 "2024-03-07T23:08:25Z")

</div>

Could that be hoisting? Though I’m used to hoisting showing \<1ns timings. The differing timings also seems consistent with the earlier comment on 1.1.0 introducing `$`-interpolation for globals, but I’m not comfortable with `@b` and `@btime` showing different results because I’d prefer no benchmark artifacts or overheads in the result. What happens when you time it in the local scope, like `let a=x, b=n; @b f(a, b) end`, or compile the benchmark in a function, like `timef(a, b) = @b f(a, b)`?

---

<div class="post-metadata">

**Author:** ![matthias314](https://avatars.discourse-cdn.com/v4/letter/m/a88e4f/32.png) [@matthias314](https://discourse.julialang.org/u/matthias314)\
**Post date:** [March 7, 2024, 11:29pm UTC](https://discourse.julialang.org/t/chairmarks-jl/111096/41 "2024-03-07T23:29:23Z")

</div>

> [@Benny](#):
>
> What happens when you time it in the local scope, like `let a=x, b=n; @b f(a, b) end`, or compile the benchmark in a function, like `timef(a, b) = @b f(a, b)`?

Both variants lead to exactly the same results as before.

[Previous page](https://discourse.julialang.org/t/chairmarks-jl/111096.md?page=1)

[Next page](https://discourse.julialang.org/t/chairmarks-jl/111096.md?page=3)