# Functions with identical native code but different benchmarks

**URL:** <https://discourse.julialang.org/t/functions-with-identical-native-code-but-different-benchmarks/136336>\
**Category:** Performance\
**Created:** [March 22, 2026, 9:08pm UTC](https://discourse.julialang.org/t/functions-with-identical-native-code-but-different-benchmarks/136336 "2026-03-22T21:08:48Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![matthias314](https://avatars.discourse-cdn.com/v4/letter/m/a88e4f/32.png) [@matthias314](https://discourse.julialang.org/u/matthias314)\
**Post date:** [March 22, 2026, 9:08pm UTC](https://discourse.julialang.org/t/functions-with-identical-native-code-but-different-benchmarks/136336/1 "2026-03-22T21:08:48Z")

</div>

I’m currently rewriting some low-level code in one of my packages. I’m seeing that the new function runs 2-3 times slower than the old one (~12ns instead of ~5ns) although `@code_native` shows _exactly_ the same code. The slowdown only appears when benchmarking a single function call. When mapping over a vector of inputs, the difference is gone. Does anybody have an idea what’s going on here?

> ****
>
> I’m comparing the function `reverse` between the ~~master~~ branch `v0.6.0` (also in the Julia repository) and the branch ~~`shuffle128`~~ `discourse-136336` of my package SmallCollections.jl. The native code is different for processors having AVX2 and AVX-512.
> 
> Processor with AVX2 (no AVX-512):
> 
> ```julia-auto
> julia> using Chairmarks, SmallCollections # v0.6.0
> 
> julia> N = 32; T = Int8; v = FixedVector{N,T}(1:N); i = 3; @b reverse($v, $i)
> 5.807 ns
> 
> julia> M = 1000; p = [rand(FixedVector{N,T}) for _ in 1:M]; q = rand(1:N, M);
> 
> julia> @b similar(p) map!(reverse, _, $p, $q)
> 5.201 μs
> 
> ```
> 
> ```julia-auto
> julia> using Chairmarks, SmallCollections # discourse-136336
> 
> julia> N = 32; T = Int8; v = FixedVector{N,T}(1:N); i = 3; @b reverse($v, $i)
> 11.702 ns
> 
> julia> M = 1000; p = [rand(FixedVector{N,T}) for _ in 1:M]; q = rand(1:N, M);
> 
> julia> @b similar(p) map!(reverse, _, $p, $q)
> 5.203 μs
> 
> ```
> 
> Processor with AVX-512:
> 
> ```julia-auto
> julia> using Chairmarks, SmallCollections # v0.6.0
> 
> julia> N = 64; T = UInt8; v = rand(FixedVector{N,T}); i = 3; @b reverse($v, $i)
> 4.873 ns
> 
> julia> M = 1000; p = [rand(FixedVector{N,T}) for _ in 1:M]; q = rand(1:N, M);
> 
> julia> @b similar(p) map!(reverse, _, $p, $q)
> 7.882 μs
> 
> ```
> 
> ```julia-auto
> julia> using Chairmarks, SmallCollections # discourse-136336
> 
> julia> N = 64; T = UInt8; v = rand(FixedVector{N,T}); i = 3; @b reverse($v, $i)
> 12.930 ns
> 
> julia> M = 1000; p = [rand(FixedVector{N,T}) for _ in 1:M]; q = rand(1:N, M);
> 
> julia> @b similar(p) map!(reverse, _, $p, $q)
> 7.884 μs
> 
> ```

EDIT: All functions are defined with `@inline`.

---

<div class="post-metadata">

**Author:** ![matthias314](https://avatars.discourse-cdn.com/v4/letter/m/a88e4f/32.png) [@matthias314](https://discourse.julialang.org/u/matthias314)\
**Post date:** [March 22, 2026, 10:43pm UTC](https://discourse.julialang.org/t/functions-with-identical-native-code-but-different-benchmarks/136336/2 "2026-03-22T22:43:18Z")

</div>

@Lilith Could this be an issue with Chairmarks.jl?

```julia-auto
julia> using SmallCollections, Chairmarks, BenchmarkTools

julia> N = 32; T = Int8; v = FixedVector{N,T}(1:N); i = 3;

julia> @b reverse($v, $i) # Chairmarks
13.206 ns

julia> @btime reverse($v, $i); # BenchmarkTools
  5.167 ns (0 allocations: 0 bytes)

```

---

<div class="post-metadata">

**Author:** ![Lilith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lilith/32/27492_2.png) [@Lilith](https://discourse.julialang.org/u/Lilith)\
**Post date:** [March 22, 2026, 11:43pm UTC](https://discourse.julialang.org/t/functions-with-identical-native-code-but-different-benchmarks/136336/3 "2026-03-22T23:43:24Z")

</div>

Nano-benchmarks are tricky to benchmark. In general you’ll get much more trustworthy results with benchmarks that take \>1μs or ideally ms.

I can’t reproduce this, but that doesn’t mean it isn’t real.

```julia-auto
(@v1.12) pkg> activate --temp
  Activating new project at `/tmp/jl_nu4uCK`

julia> using SmallCollections, Chairmarks, BenchmarkTools
 │ Package SmallCollections not found, but a package named SmallCollections is
 │ available from a registry. 
 │ Install package?
 │ (jl_nu4uCK) pkg> add SmallCollections 
 └ (y/n/o) [y]: 
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
   Installed SmallCollections ─ v0.6.0
    Updating `/tmp/jl_nu4uCK/Project.toml`
  [2b935e18] + SmallCollections v0.6.0
    Updating `/tmp/jl_nu4uCK/Manifest.toml`
  [c3b6d118] + BitIntegers v0.3.7
  [adafc99b] + CpuId v0.3.1
  [2b935e18] + SmallCollections v0.6.0
  [56f22d72] + Artifacts v1.11.0
  [2a0f44e3] + Base64 v1.11.0
  [ac6e5ff7] + JuliaSyntaxHighlighting v1.12.0
  [8f399da3] + Libdl v1.11.0
  [37e2e46d] + LinearAlgebra v1.12.0
  [d6f4376e] + Markdown v1.11.0
  [9a3f8284] + Random v1.11.0
  [ea8e919c] + SHA v0.7.0
  [f489334b] + StyledStrings v1.11.0
  [e66e0078] + CompilerSupportLibraries_jll v1.3.0+1
  [4536629a] + OpenBLAS_jll v0.3.29+0
  [8e850b90] + libblastrampoline_jll v5.15.0+0
Precompiling SmallCollections finished.
  3 dependencies successfully precompiled in 2 seconds. 4 already precompiled.

julia> N = 32; T = Int8; v = FixedVector{N,T}(1:N); i = 3;

julia> @b reverse($v, $i) # Chairmarks
13.697 ns

julia> @btime reverse($v, $i); # BenchmarkTools
  13.639 ns (0 allocations: 0 bytes)

(jl_nu4uCK) pkg> st -m
Status `/tmp/jl_nu4uCK/Manifest.toml`
  [c3b6d118] BitIntegers v0.3.7
  [adafc99b] CpuId v0.3.1
  [2b935e18] SmallCollections v0.6.0
  [56f22d72] Artifacts v1.11.0
  [2a0f44e3] Base64 v1.11.0
  [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0
  [8f399da3] Libdl v1.11.0
  [37e2e46d] LinearAlgebra v1.12.0
  [d6f4376e] Markdown v1.11.0
  [9a3f8284] Random v1.11.0
  [ea8e919c] SHA v0.7.0
  [f489334b] StyledStrings v1.11.0
  [e66e0078] CompilerSupportLibraries_jll v1.3.0+1
  [4536629a] OpenBLAS_jll v0.3.29+0
  [8e850b90] libblastrampoline_jll v5.15.0+0

julia> versioninfo()
Julia Version 1.12.5
Commit 5fe89b8ddc1 (2026-02-09 16:05 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: Linux (aarch64-linux-gnu)
  CPU: 8 × unknown
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, apple-m2)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 8 virtual cores)
Environment:
  JULIA_EDITOR = code

```

If you can produce a MWE for the identical code but different performance issue that doesn’t require the use of unregistered package versions (ideally just benchmarking packages and Base) I’d be happy to take a closer look.

---

<div class="post-metadata">

**Author:** ![matthias314](https://avatars.discourse-cdn.com/v4/letter/m/a88e4f/32.png) [@matthias314](https://discourse.julialang.org/u/matthias314)\
**Post date:** [March 22, 2026, 11:49pm UTC](https://discourse.julialang.org/t/functions-with-identical-native-code-but-different-benchmarks/136336/4 "2026-03-22T23:49:05Z")

</div>

> [@Lilith](#):
>
> that doesn’t require the use of unregistered package versions

OK. Once the new version is released, I’ll let you know.

---

<div class="post-metadata">

**Author:** ![Tyrone](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tyrone/32/221141_2.png) [@Tyrone](https://discourse.julialang.org/u/Tyrone)\
**Post date:** [March 23, 2026, 2:38pm UTC](https://discourse.julialang.org/t/functions-with-identical-native-code-but-different-benchmarks/136336/5 "2026-03-23T14:38:00Z")

</div>

Did you try calling reverse twice to eliminate first time call overhead?

```julia-auto
julia> using Chairmarks, SmallCollections  
julia> N = 32; T = Int8; v = FixedVector{N,T}(1:N); i = 3; 
julia> @b reverse($v, $i)
julia> @b reverse($v, $i)

```

My second time call results ~ 5 μs like expected, when Inspecting the runtime behavior with CodeGlass it also confirms my findings that shuffle128 version requires more warmup.

---

<div class="post-metadata">

**Author:** ![matthias314](https://avatars.discourse-cdn.com/v4/letter/m/a88e4f/32.png) [@matthias314](https://discourse.julialang.org/u/matthias314)\
**Post date:** [March 23, 2026, 10:59pm UTC](https://discourse.julialang.org/t/functions-with-identical-native-code-but-different-benchmarks/136336/6 "2026-03-23T22:59:11Z")

</div>

> [@Tyrone](#):
>
> Did you try calling reverse twice

I’ve tried this on both machines, but it doesn’t change anything. The benchmarks with `@b` change only minimally.

Note: I’ve renamed the branches so that I can modify the branches I was using before. I’ve updated the “Details” section in my first post accordingly.

---

<div class="post-metadata">

**Author:** ![Tyrone](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tyrone/32/221141_2.png) [@Tyrone](https://discourse.julialang.org/u/Tyrone)\
**Post date:** [March 23, 2026, 11:30pm UTC](https://discourse.julialang.org/t/functions-with-identical-native-code-but-different-benchmarks/136336/7 "2026-03-23T23:30:00Z")

</div>

Interesting, I will investigate further tomorrow and validate my results another time 🙂

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [March 24, 2026, 9:45am UTC](https://discourse.julialang.org/t/functions-with-identical-native-code-but-different-benchmarks/136336/8 "2026-03-24T09:45:59Z")

</div>

> [@Tyrone](#):
>
> Did you try calling reverse twice to eliminate first time call overhead?

The benchmark macro (unlike `@time`) already runs multiple times and reports the fastest sample, so it excludes outliers like first-call compilation overhead.

---

<div class="post-metadata">

**Author:** ![Tyrone](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tyrone/32/221141_2.png) [@Tyrone](https://discourse.julialang.org/u/Tyrone)\
**Post date:** [March 24, 2026, 1:05pm UTC](https://discourse.julialang.org/t/functions-with-identical-native-code-but-different-benchmarks/136336/9 "2026-03-24T13:05:02Z")

</div>

@stevengj Thank you, warmup seems the logical answer based on my findings but you are right!  
Though are you sure that @b returns the fastest sample when duration is very short and not more of an average?

I spend some time dinging into this as things did not add up, and there is an issue in [Chairmarks.jl](https://juliaregistries.github.io/General/packages/redirect_to_repo/Chairmarks) but I am unsure if this is related to your question. I will create a [Ticket](https://github.com/LilithHafner/Chairmarks.jl/issues/179) on GitHub for this.

The issue seems that `reverse` can run faster than the smallest amount of time accurately measurable (1 tick) which is on my machine 0.1 μs. ( If I am wrong about this, please let me know, I never have found another reliable accurate way)  
Below that you need to execute the function multiple times and take the average, I assume that this is also what other benchmark tools do, this is probably why @Lilith said nano-benchmarks are tricky (?).  
And when making such an average, it is always good to take “longest single call duration” in consideration.

Below are the measurements I gathered, every time `reverse` is called 400\_001 times and shortest call duration was always 0t.

> **Statistics from CodeGlass**
>
> | branch | Total Duration | Longest single call duration |
> | --- | --- | --- |
> | discourse | 97\_220 t | 156 t |
> | | 96\_188 t | 70 t |
> | | 136\_434 t | 33\_477 t |
> | | 98\_029 t | 375 t |
> | | 162\_303 t | 63\_295 t |
> | | 96\_864 t | 58 t |
> | | 98\_790 t | 418 t |
> | | 113\_411 t | 8\_528 t |
> | | 116\_236 t | 18\_172 t |
> | | 97\_050 t | 139 t |
> | v060 | 97\_119 t | 145 t |
> | | 97\_258 t | 411 t |
> | | 97\_985 t | 235 t |
> | | 106\_862 t | 8\_725 t |
> | | 108\_764 t | 8\_466 t |
> | | 97\_731 t | 117 t |
> | | 97\_903 t | 478 t |
> | | 103\_561 t | 6\_862 t |
> | | 98\_651 t | 417 t |
> | | 116\_864 t | 19\_623 t |

Edit: `reverse` is indeed fully in lined. 🙂

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [March 24, 2026, 5:51pm UTC](https://discourse.julialang.org/t/functions-with-identical-native-code-but-different-benchmarks/136336/10 "2026-03-24T17:51:44Z")

</div>

> [@Tyrone](#):
>
> Though are you sure that @b returns the fastest sample when duration is very short and not more of an average?

Yes. The [documentation for `@b`](https://chairmarks.lilithhafner.com/v1.3.1/reference#Chairmarks.@b) says:

> Benchmark `f` and return the fastest [`Sample`](https://chairmarks.lilithhafner.com/v1.3.1/reference#Chairmarks.Sample).

(That being said, each `Sample` for a very short-duration call is an average time of several function evaluations, the `evals` field, in order to get adequate timing resolution. But this is sampling process is repeated multiple times before returning the minimum, which eliminates outliers like one-time startup costs.)

And you can easily verify from the source code that [`@b` uses the `summarize` function](https://github.com/LilithHafner/Chairmarks.jl/blob/52678207dfb4b4ca7ba33fa4a224d13c1acde4c2/src/public.jl#L44) to fetch the output, which indeed [returns the `minimum`](https://github.com/LilithHafner/Chairmarks.jl/blob/52678207dfb4b4ca7ba33fa4a224d13c1acde4c2/src/statistics.jl#L26).

The justification for using the minimum sampled time is described in [Chen and Revels (2016)](https://arxiv.org/abs/1608.04295): essentially, the noise in benchmarking is always positive — at least for deterministic functions — since system interruptions never make things go _faster_, so the minimum is typically the most robust estimator of execution time. (They were not the first authors to use this methodology — e.g., it was used by the [`lmbench`](https://lmbench.sourceforge.net/) benchmark [(McVoy, 1996)](https://www.usenix.org/legacy/publications/library/proceedings/sd96/full_papers/mcvoy.pdf), and we imitated McVoy in our FFT benchmarks for [FFTW](https://www.fftw.org/) — but previous authors hadn’t attempted a statistical justification to my knowledge.)

---

<div class="post-metadata">

**Author:** ![Tyrone](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tyrone/32/221141_2.png) [@Tyrone](https://discourse.julialang.org/u/Tyrone)\
**Post date:** [March 24, 2026, 11:57pm UTC](https://discourse.julialang.org/t/functions-with-identical-native-code-but-different-benchmarks/136336/11 "2026-03-24T23:57:00Z")

</div>

@stevengj Thank you, this confirms what I assumed and found in the Docs and source code!

I only failed at describing it correctly, in case you like to read my “funny” mistake:

> **Funny misunderstanding....**
>
> I made a wrong assumption here:
> 
> > [@stevengj](#):
> >
> > The benchmark macro (unlike `@time`) already runs multiple times and reports the fastest sample, so it excludes outliers like first-call compilation overhead.
> 
> I assumed with ‘sample’ you meant a single recorded function call duration.
> 
> Which made me go through the source and docs as I was very curious how they managed to do that with such a short running function, but I could only find that they made an average when the duration is this short. At which point I asked you:
> 
> > [@Tyrone](#):
> >
> > Though are you sure that @b returns the fastest sample when duration is very short and not more of an average?
> 
> At which point I should have noticed the ambiguety of using ‘sample’ and that you meant [`Sample`](https://chairmarks.lilithhafner.com/v1.3.1/reference#Chairmarks.Sample) with it.
> 
> I am thankful that this caused you to share the underlying documentation about this, thank you! 🙂

@matthias314 I still don’t have a clear answer for you, however the statistics I proved in my previous post show that there is no actual time difference between branch`v060` and `discourse-136336` when looking at the full time duration over multiple calls.

I did do some more runs and it does seem that `discourse-136336` version does suffer more system interruptions (More frequent and higher longest single call duration) which might explain it, but I am not yet confident in this.

---

<div class="post-metadata">

**Author:** ![matthias314](https://avatars.discourse-cdn.com/v4/letter/m/a88e4f/32.png) [@matthias314](https://discourse.julialang.org/u/matthias314)\
**Post date:** [March 25, 2026, 1:05am UTC](https://discourse.julialang.org/t/functions-with-identical-native-code-but-different-benchmarks/136336/12 "2026-03-25T01:05:52Z")

</div>

@Tyrone Thanks for your help!!

In the meantime I’ve found that a tiny change in the code (passing from `UInt8` to `Int8` somewhere) makes the whole issue miraculously go away. Understanding the differences in the benchmarks would still interesting to avoid similar problems in the future. However, it’s not an issue for the next release of my package anymore.

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [March 25, 2026, 7:53am UTC](https://discourse.julialang.org/t/functions-with-identical-native-code-but-different-benchmarks/136336/13 "2026-03-25T07:53:41Z")

</div>

> [@matthias314](#):
>
> Understanding the differences in the benchmarks would still interesting to avoid similar problems in the future.

A minimal reproducer would help.

I suppose a possible cause is differing alignments in the executable code. I suppose that even when `@code_native` is identical, it is expected that performance characteristics of machine code may vary due to different offsets and stuff, due perhaps to padding (NOP instructions?) being inserted at different places. I do not remember the specifics, but look into profile-guided optimization (PGO) showcases for proof that this is important. In some experiments, such things can cause more than 10% performance differences as far as I remember.

---

<div class="post-metadata">

**Author:** ![matthias314](https://avatars.discourse-cdn.com/v4/letter/m/a88e4f/32.png) [@matthias314](https://discourse.julialang.org/u/matthias314)\
**Post date:** [March 26, 2026, 1:24am UTC](https://discourse.julialang.org/t/functions-with-identical-native-code-but-different-benchmarks/136336/14 "2026-03-26T01:24:23Z")

</div>

> [@nsajko](#):
>
> A minimal reproducer would help.

That’s probably tricky because the benchmarks here seem to be sensitive even to small changes. One could say that the current example is already small in the sense that the function we are talking about has less than 50 assembler instructions (and does not call any other function).

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [March 27, 2026, 4:27am UTC](https://discourse.julialang.org/t/functions-with-identical-native-code-but-different-benchmarks/136336/15 "2026-03-27T04:27:13Z")

</div>

I meant a reproducer that does not depend on any package. Except on Chairmarks.jl / BenchmarkTools.jl.