I’m working on a package, that I want to benchmark at every commit, because I want to see how the performance evolves over time. A short note on this idea I assume that it’s not the best workflow, would be better to run the benchmarks at every PR, but the package is still in an early stage (not yet…

[image] cserteGT3: I have access to a Linux VM running in our institution’s cloud (I’m waiting for confirmation that the hardware is not changing). It’s not only about the hardware but also about other users running things on the same machine while you are running the benchmarks. For generat…

[image] cserteGT3: I can’t use GitHub’s and GitLab’s CI It’s actually straightforward to setup GitLab CI runner (not entire GitLab service) on machines you control GitLab Runner | GitLab . It’s easy if you can use docker Run GitLab Runner in a container | GitLab . Once you setup and register t…

Great to see you got PkgBenchmarks working with Travis CI! Only quickly looking over your CI benchmark scripts in Transducers.jl, TransducersBenchmarksReports.jl, and Run.jl it doesn’t seem trivial to set it up, though. I’d really appreciate a couple of minimal examples for this. Related to this, I…

[image] kristoffer.carlsson: [image] cserteGT3: I have access to a Linux VM running in our institution’s cloud (I’m waiting for confirmation that the hardware is not changing). It’s not only about the hardware but also about other users running things on the same machine while you are run…

[image] tkf: [image] cserteGT3: I can’t use GitHub’s and GitLab’s CI It’s actually straightforward to setup GitLab CI runner (not entire GitLab service) on machines you control GitLab Runner | GitLab . It’s easy if you can use docker Run GitLab Runner in a container | GitLab . Once you set…

[image] carstenbauer: Only quickly looking over your CI benchmark scripts in Transducers.jl, TransducersBenchmarksReports.jl, and Run.jl it doesn’t seem trivial to set it up, though. I’m not doing anything complicated actually. I just use benchmarkpkg to run the benchmark, use readresults to…

[image] tkf: (A bit more sophisticated approach would be to use judge w.r.t. the reference revision (say the last release). But I haven’t gotten there yet.) I’ve been playing around a bit with PkgBenchmark.judge, and I highly recommend it. It really makes it easy to, for example, compare the …

Yea, PkgBenchmark.judge is great for deved local repositories. But IIRC it doesn’t work well with Manifest.toml. I think I know how to set it up. It’s just that I don’t need it at the moment. But I do use judge for comparing with manual implementation (within the same commits): GEMM · TransducersBe…

[image] tkf: It’s actually straightforward to setup GitLab CI runner (not entire GitLab service) on machines you control https://docs.gitlab.com/runner/ . We use the same setup and it’s really easy to implement. The nice thing is: you can register a runner on a dedicated machine where you exp…

If you don’t mind using non-julia the following stackexchange suggestion might be worth exploring. I’m sure there are similar libraries/applications in other languages… this example uses Ruby: https://stackoverflow.com/questions/15124193/how-do-you-create-a-guard-configuration-watch-that-will-only…

Finally I wrote a script that goes through all the commits and benchmarks the code with PkgBenchmark. That’s an awesome package, thank you for it! (Although the docs are outdated, I addressed that in a PR).

How to run benchmark CI on every commit?

General Usage

kristoffer.carlsson July 31, 2019, 1:51pm 2

It’s not only about the hardware but also about other users running things on the same machine while you are running the benchmarks.

For generating the markdown you could use GitHub - JuliaCI/PkgBenchmark.jl: Easy benchmark tracking for packages

4 Likes

Topic		Replies	Views
PkgBenchmark.jl workflow General Usage benchmark , benchmarktools	7	1838	April 12, 2020
Easy GitHub benchmarking with new AirspeedVelocity.jl Package Announcements package , github , github-actions	18	1375	July 1, 2025
Benchmarking tests to ensure PRs don't introduce regressions Performance benchmark , regression	5	1163	January 28, 2018
Idiomatic Benchmarking General Usage package , benchmark , ci	6	798	March 2, 2021
Recommendation for CI benchmark to catch regression Performance question , package , ci	6	1008	November 19, 2023

How to run benchmark CI on every commit?

Related topics