# How to implement performance regression tests?

**URL:** <https://discourse.julialang.org/t/how-to-implement-performance-regression-tests/132578>\
**Category:** Performance\
**Tags:** question\
**Created:** [September 22, 2025, 5:34pm UTC](https://discourse.julialang.org/t/how-to-implement-performance-regression-tests/132578 "2025-09-22T17:34:23Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [September 22, 2025, 5:34pm UTC](https://discourse.julialang.org/t/how-to-implement-performance-regression-tests/132578/1 "2025-09-22T17:34:23Z")

</div>

I added a performance test to my package KiteModels.jl: [Benchmark simplify operation by ufechner7 · Pull Request #250 · OpenSourceAWE/KiteModels.jl · GitHub](https://github.com/OpenSourceAWE/KiteModels.jl/pull/250).

If you run the script, you get an output like:

```julia-auto
julia> include("test/bench_simplify.jl")
[ Info: Simplifying the system
[ Info: Simplify took 37.283457024 seconds

```

I want a unit test that fails in case of a performance regression of a commit of 20% or more. How can I achieve that?

The absolute time depends on:

- the computer it runs on
- the load on the computer
- If it runs on a laptop, it depends on the question of whether it is connected to power or running on battery

etc

How can I catch regressions independent of the hardware on which the code is running?

Ideas:

- Create a dictionary with reference values per computer
- Disable the test if the computer is running on battery (how can I detect that?)
- Scale the time with the clock frequency of the core on which the main thread is running (how can I detect that?)
- Disable the test if the system load is too high (how can I detect that?)

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [September 22, 2025, 5:42pm UTC](https://discourse.julialang.org/t/how-to-implement-performance-regression-tests/132578/2 "2025-09-22T17:42:02Z")

</div>

I once did something like this

```julia-auto
cpu_model_raw = getproperty(first(Sys.cpu_info()), :model)
cpu_model = lowercase(replace(cpu_model_raw, r"\(.*?\)" => "", ' '=>'-'))
cpu_name = lowercase(replace(Sys.CPU_NAME, '_'=>'-'))
reference_benchmark = "benchmark/profile_$(cpu_name)_$cpu_model.json"

```

it won’t get the battery vs power stuff, or system load, but at least let me partition by cpu

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [September 22, 2025, 5:55pm UTC](https://discourse.julialang.org/t/how-to-implement-performance-regression-tests/132578/3 "2025-09-22T17:55:44Z")

</div>

Take a look into AirSpeedVelocity.jl. You can adapt the benchmarks and GitHub Action we have in Meshes.jl for example. We did setup the action so that every PR triggers benchmark results as comments:

> <https://github.com/JuliaGeometry/Meshes.jl/pull/1229>
>
> I refactored the code to follow the exact same steps of the non-culling branch d…escribed in the Möller, T. & Trumbore, B. 1997 paper. I understand that this branch will compute the correct intersection even when triangles are back-faces.
> 
> The new implementation avoids unnecessary intermediate computations like the variable \`T\` that is only computed after the determinant check. It also follows the notation of the paper more closely to facilitate future maintenance.

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [September 22, 2025, 6:00pm UTC](https://discourse.julialang.org/t/how-to-implement-performance-regression-tests/132578/4 "2025-09-22T18:00:50Z")

</div>

But how do you handle the fact that the results are hardware dependant?

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [September 22, 2025, 6:03pm UTC](https://discourse.julialang.org/t/how-to-implement-performance-regression-tests/132578/5 "2025-09-22T18:03:24Z")

</div>

The benchmarks are run on the same remote machines. But I would double check the exact infrastructure that AirSpeedVelocity.jl provides.

---

<div class="post-metadata">

**Author:** ![Jean\_Michel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jean_michel/32/8282_2.png) [@Jean\_Michel](https://discourse.julialang.org/u/Jean_Michel)\
**Post date:** [September 23, 2025, 9:54pm UTC](https://discourse.julialang.org/t/how-to-implement-performance-regression-tests/132578/6 "2025-09-23T21:54:26Z")

</div>

A simple way I think is to benchmark against a reference benchmark which contains simple code which does not change. Like some king of stupid loop.