# Benchmarking tests to ensure PRs don't introduce regressions

**URL:** <https://discourse.julialang.org/t/benchmarking-tests-to-ensure-prs-dont-introduce-regressions/8630>\
**Category:** Performance\
**Tags:** benchmark, regression\
**Created:** [January 26, 2018, 11:10pm UTC](https://discourse.julialang.org/t/benchmarking-tests-to-ensure-prs-dont-introduce-regressions/8630 "2018-01-26T23:10:34Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon94023334](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@anon94023334](https://discourse.julialang.org/u/anon94023334)\
**Post date:** [January 26, 2018, 11:10pm UTC](https://discourse.julialang.org/t/benchmarking-tests-to-ensure-prs-dont-introduce-regressions/8630/1 "2018-01-26T23:10:34Z")

</div>

Has anyone come up with a clever way to perform benchmarking of specific functions as part of the testing or CI process so that PRs can be checked to make sure no performance regressions are introduced?

---

<div class="post-metadata">

**Author:** ![omus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/omus/32/402_2.png) [@omus](https://discourse.julialang.org/u/omus)\
**Post date:** [January 27, 2018, 3:14am UTC](https://discourse.julialang.org/t/benchmarking-tests-to-ensure-prs-dont-introduce-regressions/8630/2 "2018-01-27T03:14:28Z")

</div>

You may want to checkout [https://github.com/JuliaCI/PkgBenchmark.jl](https://github.com/JuliaCI/PkgBenchmark.jl)

---

<div class="post-metadata">

**Author:** ![mauro3](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mauro3/32/292_2.png) [@mauro3](https://discourse.julialang.org/u/mauro3)\
**Post date:** [January 27, 2018, 8:38am UTC](https://discourse.julialang.org/t/benchmarking-tests-to-ensure-prs-dont-introduce-regressions/8630/3 "2018-01-27T08:38:44Z")

</div>

But I don’t think you can run the benchmarks on the CI machines, as you have no control over what exact machine one gets and how many other processes are running. At least Julia itself has a dedicated machine for this: nanosoldier. But should be ok to run those BM locally.

---

<div class="post-metadata">

**Author:** ![anon94023334](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@anon94023334](https://discourse.julialang.org/u/anon94023334)\
**Post date:** [January 28, 2018, 5:45pm UTC](https://discourse.julialang.org/t/benchmarking-tests-to-ensure-prs-dont-introduce-regressions/8630/4 "2018-01-28T17:45:27Z")

</div>

This is a great point. I was hoping to be able to check out a PR locally, and run the benchmarks on it, comparing to previous benchmarks. I’ve been playing around with PkgBenchmark but am not quite sure how to do this properly.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [January 28, 2018, 6:37pm UTC](https://discourse.julialang.org/t/benchmarking-tests-to-ensure-prs-dont-introduce-regressions/8630/5 "2018-01-28T18:37:22Z")

</div>

Simply time your own tests in your own packages. If they slow down, report it.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [January 28, 2018, 6:44pm UTC](https://discourse.julialang.org/t/benchmarking-tests-to-ensure-prs-dont-introduce-regressions/8630/6 "2018-01-28T18:44:32Z")

</div>

> [@PetrKryslUCSD](#):
>
> Simply time your own tests in your own packages. If they slow down, report it.

I use this method.

> <https://github.com/SciML/OrdinaryDiffEq.jl/blob/master/test/runtests.jl>

If you know the numbers well you’ll see something fishy and it makes it quite easy to bisect to see what happened. Then we have DiffEqBenchmarks.jl which we try to run after any big change to the integrators to make sure it’s the same. I am planning on setting up a local computer with a CRON job to continually update those as well.