# Recommendation for CI benchmark to catch regression

**URL:** <https://discourse.julialang.org/t/recommendation-for-ci-benchmark-to-catch-regression/104698>\
**Category:** Performance\
**Tags:** question, package, ci\
**Created:** [October 7, 2023, 12:28pm UTC](https://discourse.julialang.org/t/recommendation-for-ci-benchmark-to-catch-regression/104698 "2023-10-07T12:28:44Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [October 7, 2023, 12:28pm UTC](https://discourse.julialang.org/t/recommendation-for-ci-benchmark-to-catch-regression/104698/1 "2023-10-07T12:28:44Z")

</div>

It’s pretty common for Julia package developers to want to monitor key usage latency / performance to avoid regression in PRs. Julia itself has [nanosoldier](https://discourse.julialang.org/t/ann-nanosoldier-package-evaluation-with-badges/33339), and some big package such as Makie has their own bot: [Hardcode paths for default fonts to avoid search latency by jkrumbiegel · Pull Request #2531 · MakieOrg/Makie.jl · GitHub](https://github.com/MakieOrg/Makie.jl/pull/2531#issuecomment-1366622636)

Is there a semi-canned solution or workflow people would recommend? I have seen past effort such as: [GitHub - maxbennedich/julia-regression-analysis: Regression Analysis for Julia](https://github.com/maxbennedich/julia-regression-analysis) but didn’t seem to be widely used. Maybe it’s just not worth the effort given stability and cost, I think even very popular packages such as DataFrames.jl don’t have automated performance/latency in CI

---

<div class="post-metadata">

**Author:** ![Krastanov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/krastanov/32/6817_2.png) [@Krastanov](https://discourse.julialang.org/u/Krastanov)\
**Post date:** [October 7, 2023, 6:23pm UTC](https://discourse.julialang.org/t/recommendation-for-ci-benchmark-to-catch-regression/104698/2 "2023-10-07T18:23:09Z")

</div>

Maybe benchmark.yml and benchmark-comment.yml would be useful here? E.g. in JET: [https://github.com/aviatesk/JET.jl/blob/master/.github/workflows/benchmark.yml](https://github.com/aviatesk/JET.jl/blob/master/.github/workflows/benchmark.yml) providing comments like [OptAnalyzer: ignore reports from const-prop when concrete-evaled already by aviatesk · Pull Request #561 · aviatesk/JET.jl · GitHub](https://github.com/aviatesk/JET.jl/pull/561#issuecomment-1687609410)

These track performance regressions but not latency regressions.

For latency regressions a good proxy might be invalidation tracking: [https://github.com/SciML/SymbolicNumericIntegration.jl/blob/main/.github/workflows/Invalidations.yml](https://github.com/SciML/SymbolicNumericIntegration.jl/blob/main/.github/workflows/Invalidations.yml)

Maybe someone can volunteer to add these to the various template package generators…

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [October 10, 2023, 1:33pm UTC](https://discourse.julialang.org/t/recommendation-for-ci-benchmark-to-catch-regression/104698/3 "2023-10-10T13:33:47Z")

</div>

> [@Krastanov](#):
>
> [https://github.com/aviatesk/JET.jl/blob/master/.github/workflows/benchmark.yml](https://github.com/aviatesk/JET.jl/blob/master/.github/workflows/benchmark.yml) providing comments like [OptAnalyzer: ignore reports from const-prop when concrete-evaled already by aviatesk · Pull Request #561 · aviatesk/JET.jl · GitHub](https://github.com/aviatesk/JET.jl/pull/561#issuecomment-1687609410)

this is actually super complicated, it needs two CI yaml to do the dance ☹

* * *

meanwhile, Makie.jl does the whole GitHub API related thing form inside Julia:

> <https://github.com/MakieOrg/Makie.jl/blob/ce5c21411bab77882b317bbeba1b3a204d5d8b78/metrics/ttfp/run-benchmark.jl#L230-L242>

---

<div class="post-metadata">

**Author:** ![filchristou](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/filchristou/32/26760_2.png) [@filchristou](https://discourse.julialang.org/u/filchristou)\
**Post date:** [November 14, 2023, 3:24pm UTC](https://discourse.julialang.org/t/recommendation-for-ci-benchmark-to-catch-regression/104698/4 "2023-11-14T15:24:35Z")

</div>

btw, there is also [Github-action-benchmark with Julia support](https://discourse.julialang.org/t/github-action-benchmark-with-julia-support/72589) and [[ANN] AirspeedVelocity.jl - easily benchmark Julia packages over their lifetime](https://discourse.julialang.org/t/ann-airspeedvelocity-jl-easily-benchmark-julia-packages-over-their-lifetime/97221)

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [November 14, 2023, 5:50pm UTC](https://discourse.julialang.org/t/recommendation-for-ci-benchmark-to-catch-regression/104698/5 "2023-11-14T17:50:41Z")

</div>

> **[GitHub - tkf/BenchmarkCI.jl](https://github.com/tkf/BenchmarkCI.jl)**
>
> Contribute to tkf/BenchmarkCI.jl development by creating an account on GitHub.

Looks unmaintained / experimental but still useful. I think that’s the package used in the workflows linked by Krastanov. But AirspeedVelocity looks ~~better~~ more current.

---

<div class="post-metadata">

**Author:** ![filchristou](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/filchristou/32/26760_2.png) [@filchristou](https://discourse.julialang.org/u/filchristou)\
**Post date:** [November 14, 2023, 7:56pm UTC](https://discourse.julialang.org/t/recommendation-for-ci-benchmark-to-catch-regression/104698/6 "2023-11-14T19:56:40Z")

</div>

For the sake of completeness, the comment from Krastanov before actually mentioned BenchmarkCI.jl . i.e. JET uses BenchmarkCI.jl  
I cannot disagree that it looks unmaintained and experimental, but it works. Moreover, I always found @tkf 's packages very well designed and speak to my personal preference.

---

<div class="post-metadata">

**Author:** ![MilesCranmer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/milescranmer/32/21070_2.png) [@MilesCranmer](https://discourse.julialang.org/u/MilesCranmer)\
**Post date:** [November 19, 2023, 9:33pm UTC](https://discourse.julialang.org/t/recommendation-for-ci-benchmark-to-catch-regression/104698/7 "2023-11-19T21:33:29Z")

</div>

This is what AirspeedVelocity.jl looks like on a typical PR: [Create `AutoFloat` type for units by MilesCranmer · Pull Request #66 · SymbolicML/DynamicQuantities.jl · GitHub](https://github.com/SymbolicML/DynamicQuantities.jl/pull/66#issuecomment-1763978727)

It’s extremely useful for catching regressions:

 ![Screenshot 2023-11-19 at 21.26.26](https://global.discourse-cdn.com/julialang/original/3X/2/2/227481e4b209e33bb1bdfb8039c7f0ff6257b70b.png)

It’s saved me multiple times from introducing performance regression due to some type instability I didn’t notice. Also very useful for monitoring time-to-load. Basically just copy this file into a workflow: [https://github.com/SymbolicML/DynamicQuantities.jl/blob/main/.github/workflows/benchmark\_pr.yml](https://github.com/SymbolicML/DynamicQuantities.jl/blob/main/.github/workflows/benchmark_pr.yml). and make sure you have a file `benchmark/benchmarks.jl` which uses `BenchmarkTools` to define `const SUITE = BenchmarkGroup()`

So far it’s used in my repos, and also SymbolicUtils.jl. (Maybe others too that I’m not aware of.)

* * *

Compared with BenchmarkCI.jl/PkgBenchmark.jl it’s not as extensive, so probably good to check out both. AirspeedVelocity.jl is basically a re-built version (also uses BenchmarkTools) with a significant emphasis on the command line (because it makes it easier to interface with git, especially if you just want to quickly check for regressions against `master` or something).