# PkgBenchmark.jl workflow
**URL:** https://discourse.julialang.org/t/pkgbenchmark-jl-workflow/24472
**Category:** General Usage
**Tags:** benchmark, benchmarktools
**Created:** [May 22, 2019, 11:58am UTC](https://discourse.julialang.org/t/pkgbenchmark-jl-workflow/24472 "2019-05-22T11:58:59Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)
#### Post date: [May 22, 2019, 11:58am UTC](https://discourse.julialang.org/t/pkgbenchmark-jl-workflow/24472/1 "2019-05-22T11:58:59Z")
This is basically a revival of [Benchmarking with PkgBenchmark.jl](https://discourse.julialang.org/t/benchmarking-with-pkgbenchmark-jl/3038) from two years ago.
Benchmarking packages to detect regressions seems very important, but I don’t quite get how to use PkgBenchmark.jl effectively. In particular, I don’t understand from where I should run the benchmarks defined in my `benchmark/benchmarks.jl`. Naively I put a `run_benchmarks.jl` script (see below) into the same folder. This fails however as soon as I try to benchmark a different branch as it tries to checkout this branch, which fails due to an active process still accessing files/folders in the repo - me running the `run_benchmarks.jl`.
Of course, automizing this in some way would be great (but maybe unrealistic?).
I’d really appreciate any pointers to documentations, tutorials, good examples in existing packages, and other resources.
Thanks in advance!
```julia
# run_benchmarks.jl
# A branch name or commit id can be supplied as an argument.
using Pkg
Pkg.activate(@ __DIR__ )
using PkgBenchmark
branch_or_commit = nothing
if length(ARGS) > 0
branch_or_commit = ARGS[1]
end
out = isnothing(branch_or_commit) ? "master" : string(branch_or_commit)
config = BenchmarkConfig(id = out,
juliacmd = `julia -O3 --project=.`,
env = Dict("JULIA_NUM_THREADS" => 1,
"OMP_NUM_THREADS" => Sys.CPU_THREADS,
"MKL_NUM_THREADS" => Sys.CPU_THREADS))
r = benchmarkpkg("StableDQMC", config;
script = "benchmark/benchmarks.jl",
resultfile = string(out, ".bench"))
export_markdown(string(out, ".md"), r)
```
---
### Author:  [@dilumaluthge](https://discourse.julialang.org/u/dilumaluthge)
#### Post date: [August 1, 2019, 4:22pm UTC](https://discourse.julialang.org/t/pkgbenchmark-jl-workflow/24472/2 "2019-08-01T16:22:14Z")
Have you looked at `PkgBenchmark.judge`? It allows you to run the benchmarks on two different branches/commits/etc and compare the results.
---
### Author:  [@tkf](https://discourse.julialang.org/u/tkf)
#### Post date: [August 19, 2019, 8:45pm UTC](https://discourse.julialang.org/t/pkgbenchmark-jl-workflow/24472/3 "2019-08-19T20:45:33Z")
(Continuing the discussion in [How to run benchmark CI on every commit? - #3 by tkf](https://discourse.julialang.org/t/how-to-run-benchmark-ci-on-every-commit/27034/3))
FYI I just set up CI to run `PkgBenchmark.judge` [https://github.com/tkf/Transducers.jl/pull/32](https://github.com/tkf/Transducers.jl/pull/32) and I find it pretty useful. Feel free to grab some helper script/setting from there. There were some bits that were not super obvious (like how to invoke `git fetch` to pull the `master` branch so that `PkgBenchmark` can switch to it.).
---
### Author:  [@ericphanson](https://discourse.julialang.org/u/ericphanson)
#### Post date: [August 31, 2019, 11:43pm UTC](https://discourse.julialang.org/t/pkgbenchmark-jl-workflow/24472/4 "2019-08-31T23:43:18Z")
I tried mimicking your .travis.yml (and used your awesome pretty printing) for Convex.jl (PR: [Convex#321](https://github.com/JuliaOpt/Convex.jl/pull/321); travis file: [https://github.com/JuliaOpt/Convex.jl/blob/master/.travis.yml](https://github.com/JuliaOpt/Convex.jl/blob/master/.travis.yml)). I think I messed something up though, because CI didn’t run on the next PR I made ([Convex#322](https://github.com/JuliaOpt/Convex.jl/pull/322)). Could I get some assistance? 🙂
---
### Author:  [@tkf](https://discourse.julialang.org/u/tkf)
#### Post date: [September 1, 2019, 1:48am UTC](https://discourse.julialang.org/t/pkgbenchmark-jl-workflow/24472/5 "2019-09-01T01:48:53Z")
So @ericphanson found that there were some quirks in my setup because my setup was relying on that I develop in one repository. If you need to set it up for a GitHub organization (or you have a large flow of PRs), Convex.jl’s setup is probably the best starting point. See: [https://github.com/JuliaOpt/Convex.jl/pull/323](https://github.com/JuliaOpt/Convex.jl/pull/323)
---
### Author:  [@lostella](https://discourse.julialang.org/u/lostella)
#### Post date: [April 12, 2020, 4:35pm UTC](https://discourse.julialang.org/t/pkgbenchmark-jl-workflow/24472/6 "2020-04-12T16:35:28Z")
I’m wondering if anyone has a similar setup using GitHub workflows. I tried mimicking what discussed above by @tkf and @ericphanson but I’m getting the following
```julia
ERROR: LoadError:
/home/runner/work/ProximalOperators.jl/ProximalOperators.jl is dirty.
Please commit/stash your changes before benchmarking a specific commit
```
I think this has to do with the benchmark tuning data being persisted, which prevents PkgBenchmark from being able to switch branches. Any idea how to solve that?
Full log [here](https://pipelines.actions.githubusercontent.com/rWTNbd0YIrwjp7oKGR6Q8HJ3sxjd3MOVoEzOjMMB41i1Oc7xND/_apis/pipelines/1/runs/1549/signedlogcontent/7?urlExpires=2020-04-12T16%3A37%3A24.5308131Z&urlSigningMethod=HMACV1&urlSignature=Bg%2BsGPyfdvO%2FUBgVQPl6LF5LoGeko6OzZsKZ5G%2BUHx4%3D), configuration [here](https://github.com/lostella/ProximalOperators.jl/blob/460f0c4e450c627f5698c0cf179222107e6de0b8/.github/workflows/ci.yml#L49). Any hint is appreciated, thanks! 🙏
---
### Author:  [@ericphanson](https://discourse.julialang.org/u/ericphanson)
#### Post date: [April 12, 2020, 4:44pm UTC](https://discourse.julialang.org/t/pkgbenchmark-jl-workflow/24472/7 "2020-04-12T16:44:41Z")
We ran into this too for Convex.jl but just added the generated files to the gitignore: [https://github.com/JuliaOpt/Convex.jl/blob/5d8f1086949b92c98eaa152976d3caa810c5ec37/.gitignore#L8](https://github.com/JuliaOpt/Convex.jl/blob/5d8f1086949b92c98eaa152976d3caa810c5ec37/.gitignore#L8).
---
### Author:  [@lostella](https://discourse.julialang.org/u/lostella)
#### Post date: [April 12, 2020, 4:48pm UTC](https://discourse.julialang.org/t/pkgbenchmark-jl-workflow/24472/8 "2020-04-12T16:48:26Z")
Of course! 🤦♂️ Thank you so much