It’s pretty common for Julia package developers to want to monitor key usage latency / performance to avoid regression in PRs. Julia itself has nanosoldier , and some big package such as Makie has their own bot: Hardcode paths for default fonts to avoid search latency by jkrumbiegel · Pull Request #2…

Maybe benchmark.yml and benchmark-comment.yml would be useful here? E.g. in JET: https://github.com/aviatesk/JET.jl/blob/master/.github/workflows/benchmark.yml providing comments like OptAnalyzer: ignore reports from const-prop when concrete-evaled already by aviatesk · Pull Request #561 · aviatesk/…

[image] Krastanov: https://github.com/aviatesk/JET.jl/blob/master/.github/workflows/benchmark.yml providing comments like OptAnalyzer: ignore reports from const-prop when concrete-evaled already by aviatesk · Pull Request #561 · aviatesk/JET.jl · GitHub this is actually super complicated, i…

[image] GitHub - tkf/BenchmarkCI.jl Contribute to tkf/BenchmarkCI.jl development by creating an account on GitHub. Looks unmaintained / experimental but still useful. I think that’s the package used in the workflows linked by Krastanov. But AirspeedVelocity l…

For the sake of completeness, the comment from Krastanov before actually mentioned BenchmarkCI.jl . i.e. JET uses BenchmarkCI.jl I cannot disagree that it looks unmaintained and experimental, but it works. Moreover, I always found @tkf 's packages very well designed and speak to my personal prefere…

This is what AirspeedVelocity.jl looks like on a typical PR: Create `AutoFloat` type for units by MilesCranmer · Pull Request #66 · SymbolicML/DynamicQuantities.jl · GitHub It’s extremely useful for catching regressions: [Screenshot 2023-11-19 at 21.26.26] It’s saved me multiple times from intro…

Recommendation for CI benchmark to catch regression

General Usage Performance

filchristou November 14, 2023, 3:24pm 4

btw, there is also Github-action-benchmark with Julia support and [ANN] AirspeedVelocity.jl - easily benchmark Julia packages over their lifetime

Topic		Replies	Views
Easy GitHub benchmarking with new AirspeedVelocity.jl Package Announcements package , github , github-actions	18	1391	July 1, 2025
Benchmarking tests to ensure PRs don't introduce regressions Performance benchmark , regression	5	1166	January 28, 2018
Benchmarking with PkgBenchmark.jl General Usage question , testing , performance	15	3253	January 6, 2018
PkgBenchmark.jl workflow General Usage benchmark , benchmarktools	7	1848	April 12, 2020
[ANN] AirspeedVelocity.jl - easily benchmark Julia packages over their lifetime Package Announcements package , announcement , benchmark	7	1394	June 19, 2024

Recommendation for CI benchmark to catch regression

Related topics