# How to run benchmark CI on every commit?

**URL:** <https://discourse.julialang.org/t/how-to-run-benchmark-ci-on-every-commit/27034>\
**Category:** General Usage\
**Tags:** benchmark, ci\
**Created:** [July 31, 2019, 1:40pm UTC](https://discourse.julialang.org/t/how-to-run-benchmark-ci-on-every-commit/27034 "2019-07-31T13:40:28Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![cserteGT3](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/csertegt3/32/8283_2.png) [@cserteGT3](https://discourse.julialang.org/u/cserteGT3)\
**Post date:** [July 31, 2019, 1:40pm UTC](https://discourse.julialang.org/t/how-to-run-benchmark-ci-on-every-commit/27034/1 "2019-07-31T13:40:28Z")

</div>

I’m working on a package, that I want to benchmark at every commit, because I want to see how the performance evolves over time.

### A short note on this idea

I assume that it’s not the best workflow, would be better to run the benchmarks at every PR, but the package is still in an early stage (not yet ready for publication), and I’m working on it alone, so running the benchmarks commit by commit seems fine for now.

### My expectations

I’d like to run the same benchmark for every commit. This doesn’t have to be “real time”, as I run this while I’m coding, but want to have a consistent measurement. I want to store the results somewhere (in git repo in a csv file or something like that). I want to do it free.

### The problem

AFAIK I can’t use GitHub’s and GitLab’s CI, because the jobs can run on any machine (which kills the goal of consistency).

### My concept

I have access to a Linux VM running in our institution’s cloud (I’m waiting for confirmation that the hardware is not changing). I would write a script that clone’s the package and runs the benchmark. It’s ok if I have to start it manually (e.g. at the end of day), but it should traverse through all the commits (on `#master`) that haven’t been benchmarked yet. Then it should commit and push the results to a benchmark repo (generate markdown/html based on the csv and publish on gh/gl pages, etc.).

(Other solution I thought of is installing a GitLab server for myself, but that doesn’t sound like an easy way to do this.)

### My questions

1. What do you think? Is it reasonable? Any other way to do this?
2. How to traverse through commits? (That is my main question.)

I’m open for any help, advice or opinion!

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [July 31, 2019, 1:51pm UTC](https://discourse.julialang.org/t/how-to-run-benchmark-ci-on-every-commit/27034/2 "2019-07-31T13:51:51Z")

</div>

> [@cserteGT3](#):
>
> I have access to a Linux VM running in our institution’s cloud (I’m waiting for confirmation that the hardware is not changing).

It’s not only about the hardware but also about other users running things on the same machine while you are running the benchmarks.

For generating the markdown you could use [GitHub - JuliaCI/PkgBenchmark.jl: Easy benchmark tracking for packages](https://github.com/JuliaCI/PkgBenchmark.jl)

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 1, 2019, 5:52am UTC](https://discourse.julialang.org/t/how-to-run-benchmark-ci-on-every-commit/27034/3 "2019-08-01T05:52:17Z")

</div>

> [@cserteGT3](#):
>
> I can’t use GitHub’s and GitLab’s CI

It’s actually straightforward to setup GitLab CI _runner_ (not entire GitLab service) on machines you control [GitLab Runner | GitLab](https://docs.gitlab.com/runner/). It’s easy if you can use docker [Run GitLab Runner in a container | GitLab](https://docs.gitlab.com/runner/install/docker.html). Once you setup and register the GitLab runner service, you can initiate the job by pushing to [gitlab.com](http://gitlab.com). But this requires the runner service to keep running in the background. So I’m not sure if it fits with your need.

FYI, I setup benchmarks on Travis CI and run a couple of benchmarks: [Home · TransducersBenchmarksReports.jl](https://tkf.github.io/TransducersBenchmarksReports.jl/stable/). I was worried that it may fluctuate a lot. But it turned out Travis is consistent enough for my need. This is because I only care about the performance relative to the baseline implementation I write.

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [August 1, 2019, 8:14am UTC](https://discourse.julialang.org/t/how-to-run-benchmark-ci-on-every-commit/27034/4 "2019-08-01T08:14:00Z")

</div>

Great to see you got PkgBenchmarks working with Travis CI! Only quickly looking over your CI benchmark scripts in Transducers.jl, TransducersBenchmarksReports.jl, and Run.jl it doesn’t seem trivial to set it up, though. I’d really appreciate a couple of minimal examples for this.

Related to this, I never got a response here: [PkgBenchmark.jl workflow](https://discourse.julialang.org/t/pkgbenchmark-jl-workflow/24472)

---

<div class="post-metadata">

**Author:** ![cserteGT3](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/csertegt3/32/8283_2.png) [@cserteGT3](https://discourse.julialang.org/u/cserteGT3)\
**Post date:** [August 1, 2019, 3:09pm UTC](https://discourse.julialang.org/t/how-to-run-benchmark-ci-on-every-commit/27034/5 "2019-08-01T15:09:23Z")

</div>

> [@kristoffer.carlsson](#):
>
> > [@cserteGT3](#):
> >
> > I have access to a Linux VM running in our institution’s cloud (I’m waiting for confirmation that the hardware is not changing).
> 
> It’s not only about the hardware but also about other users running things on the same machine while you are running the benchmarks.

Thanks! I’ll keep that in mind. (As far as I know and understand, our VMs are bound to x number of cores and y RAM so this should not be an issue.)

> [@](#):
>
> For generating the markdown you could use [GitHub - JuliaCI/PkgBenchmark.jl: Easy benchmark tracking for packages](https://github.com/JuliaCI/PkgBenchmark.jl)

I’ll definitely check out!

---

<div class="post-metadata">

**Author:** ![cserteGT3](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/csertegt3/32/8283_2.png) [@cserteGT3](https://discourse.julialang.org/u/cserteGT3)\
**Post date:** [August 1, 2019, 3:13pm UTC](https://discourse.julialang.org/t/how-to-run-benchmark-ci-on-every-commit/27034/6 "2019-08-01T15:13:31Z")

</div>

> [@tkf](#):
>
> > [@cserteGT3](#):
> >
> > I can’t use GitHub’s and GitLab’s CI
> 
> It’s actually straightforward to setup GitLab CI _runner_ (not entire GitLab service) on machines you control [GitLab Runner | GitLab](https://docs.gitlab.com/runner/). It’s easy if you can use docker [Run GitLab Runner in a container | GitLab](https://docs.gitlab.com/runner/install/docker.html). Once you setup and register the GitLab runner service, you can initiate the job by pushing to [gitlab.com](http://gitlab.com). But this requires the runner service to keep running in the background. So I’m not sure if it fits with your need.

Hmm, so that’s how [DiffEqBot](https://julialang.org/blog/2019/06/diffeqbot) works (I guess). Keep it running should not be an issue, I’ll read through the docs how that works.

> [@](#):
>
> FYI, I setup benchmarks on Travis CI and run a couple of benchmarks: [Home · TransducersBenchmarksReports.jl](https://tkf.github.io/TransducersBenchmarksReports.jl/stable/). I was worried that it may fluctuate a lot. But it turned out Travis is consistent enough for my need. This is because I only care about the performance relative to the baseline implementation I write.

I’ll check the code, thank you!

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 1, 2019, 10:05pm UTC](https://discourse.julialang.org/t/how-to-run-benchmark-ci-on-every-commit/27034/7 "2019-08-01T22:05:01Z")

</div>

> [@carstenbauer](#):
>
> Only quickly looking over your CI benchmark scripts in Transducers.jl, TransducersBenchmarksReports.jl, and Run.jl it doesn’t seem trivial to set it up, though.

I’m not doing anything complicated actually. I just use `benchmarkpkg` to run the benchmark, use `readresults` to load it, use `export_markdown` to create a markdown file, and then use Documenter to generate the github page. I’m using `Run.script` to set up a project automatically before running the script, but you can replace it with `Pkg.instantiate` and `Pkg.activate`. Note that the CI for Transducers.jl is not related (even though I run benchmark there as a smoke test).

(A bit more sophisticated approach would be to use `judge` w.r.t. the reference revision (say the last release). But I haven’t gotten there yet.)

---

<div class="post-metadata">

**Author:** ![dilumaluthge](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dilumaluthge/32/29283_2.png) [@dilumaluthge](https://discourse.julialang.org/u/dilumaluthge)\
**Post date:** [August 2, 2019, 12:42am UTC](https://discourse.julialang.org/t/how-to-run-benchmark-ci-on-every-commit/27034/8 "2019-08-02T00:42:51Z")

</div>

> [@tkf](#):
>
> (A bit more sophisticated approach would be to use `judge` w.r.t. the reference revision (say the last release). But I haven’t gotten there yet.)

I’ve been playing around a bit with `PkgBenchmark.judge`, and I highly recommend it. It really makes it easy to, for example, compare the performance of a pull request with the performance of `master`.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 2, 2019, 1:55am UTC](https://discourse.julialang.org/t/how-to-run-benchmark-ci-on-every-commit/27034/9 "2019-08-02T01:55:31Z")

</div>

Yea, `PkgBenchmark.judge` is great for `dev`ed local repositories. But IIRC it doesn’t work well with `Manifest.toml`. I think I know how to set it up. It’s just that I don’t need it at the moment.

But I do use `judge` for comparing with manual implementation (within the same commits): [GEMM · TransducersBenchmarksReports.jl](https://tkf.github.io/TransducersBenchmarksReports.jl/v0.3.3/gemm/#Transducers-vs-manual-implementation-1). This part is a bit tricky and requires to touch `PkgBenchmark` internals a bit [https://github.com/tkf/TransducersBenchmarksReports.jl/blob/cc7e49bec7ad4433145c1e2ec9443057d0f25ac2/docs/src/gemm.md#transducers-vs-manual-implementation](https://github.com/tkf/TransducersBenchmarksReports.jl/blob/cc7e49bec7ad4433145c1e2ec9443057d0f25ac2/docs/src/gemm.md#transducers-vs-manual-implementation)

---

<div class="post-metadata">

**Author:** ![tamasgal](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamasgal/32/27946_2.png) [@tamasgal](https://discourse.julialang.org/u/tamasgal)\
**Post date:** [August 2, 2019, 7:12am UTC](https://discourse.julialang.org/t/how-to-run-benchmark-ci-on-every-commit/27034/10 "2019-08-02T07:12:13Z")

</div>

> [@tkf](#):
>
> It’s actually straightforward to setup GitLab CI _runner_ (not entire GitLab service) on machines you control [https://docs.gitlab.com/runner/](https://docs.gitlab.com/runner/).

We use the same setup and it’s really easy to implement. The nice thing is: you can register a runner on a dedicated machine where you explicitly limit the number of parallel instances to **one** to avoid resource clashes.

---

<div class="post-metadata">

**Author:** ![taqtiqa-mark](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/taqtiqa-mark/32/4383_2.png) [@taqtiqa-mark](https://discourse.julialang.org/u/taqtiqa-mark)\
**Post date:** [August 6, 2019, 8:26am UTC](https://discourse.julialang.org/t/how-to-run-benchmark-ci-on-every-commit/27034/11 "2019-08-06T08:26:16Z")

</div>

If you don’t mind using non-julia the following stackexchange suggestion might be worth exploring. I’m sure there are similar libraries/applications in other languages… this example uses Ruby:

[https://stackoverflow.com/questions/15124193/how-do-you-create-a-guard-configuration-watch-that-will-only-run-the-test-file-t](https://stackoverflow.com/questions/15124193/how-do-you-create-a-guard-configuration-watch-that-will-only-run-the-test-file-t)

---

<div class="post-metadata">

**Author:** ![cserteGT3](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/csertegt3/32/8283_2.png) [@cserteGT3](https://discourse.julialang.org/u/cserteGT3)\
**Post date:** [August 10, 2019, 8:44am UTC](https://discourse.julialang.org/t/how-to-run-benchmark-ci-on-every-commit/27034/12 "2019-08-10T08:44:09Z")

</div>

Finally I wrote a script that goes through all the commits and benchmarks the code with PkgBenchmark. That’s an awesome package, thank you for it! (Although the docs are outdated, I addressed that in a PR).

---

<div class="post-metadata">

**Author:** ![cserteGT3](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/csertegt3/32/8283_2.png) [@cserteGT3](https://discourse.julialang.org/u/cserteGT3)\
**Post date:** [August 10, 2019, 8:46am UTC](https://discourse.julialang.org/t/how-to-run-benchmark-ci-on-every-commit/27034/13 "2019-08-10T08:46:10Z")

</div>

Thank you for the confirmation, I may play with runners in the near future.

---

<div class="post-metadata">

**Author:** ![cserteGT3](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/csertegt3/32/8283_2.png) [@cserteGT3](https://discourse.julialang.org/u/cserteGT3)\
**Post date:** [August 10, 2019, 8:54am UTC](https://discourse.julialang.org/t/how-to-run-benchmark-ci-on-every-commit/27034/14 "2019-08-10T08:54:28Z")

</div>

I wanted to go with julia, because that’s the language I’m most confident with. But thanks for the idea of non-julia scripts, next time I’ll consider a broader set of ideas.