This is basically a revival of Benchmarking with PkgBenchmark.jl from two years ago.
Benchmarking packages to detect regressions seems very important, but I don’t quite get how to use PkgBenchmark.jl effectively. In particular, I don’t understand from where I should run the benchmarks defined in my
benchmark/benchmarks.jl. Naively I put a
run_benchmarks.jl script (see below) into the same folder. This fails however as soon as I try to benchmark a different branch as it tries to checkout this branch, which fails due to an active process still accessing files/folders in the repo - me running the
Of course, automizing this in some way would be great (but maybe unrealistic?).
I’d really appreciate any pointers to documentations, tutorials, good examples in existing packages, and other resources.
Thanks in advance!
# A branch name or commit id can be supplied as an argument.
branch_or_commit = nothing
if length(ARGS) > 0
branch_or_commit = ARGS
out = isnothing(branch_or_commit) ? "master" : string(branch_or_commit)
config = BenchmarkConfig(id = out,
juliacmd = `julia -O3 --project=.`,
env = Dict("JULIA_NUM_THREADS" => 1,
"OMP_NUM_THREADS" => Sys.CPU_THREADS,
"MKL_NUM_THREADS" => Sys.CPU_THREADS))
r = benchmarkpkg("StableDQMC", config;
script = "benchmark/benchmarks.jl",
resultfile = string(out, ".bench"))
export_markdown(string(out, ".md"), r)
Have you looked at
PkgBenchmark.judge? It allows you to run the benchmarks on two different branches/commits/etc and compare the results.
(Continuing the discussion in How to run benchmark CI on every commit?)
FYI I just set up CI to run
PkgBenchmark.judge https://github.com/tkf/Transducers.jl/pull/32 and I find it pretty useful. Feel free to grab some helper script/setting from there. There were some bits that were not super obvious (like how to invoke
git fetch to pull the
master branch so that
PkgBenchmark can switch to it.).
I tried mimicking your .travis.yml (and used your awesome pretty printing) for Convex.jl (PR: Convex#321; travis file: https://github.com/JuliaOpt/Convex.jl/blob/master/.travis.yml). I think I messed something up though, because CI didn’t run on the next PR I made (Convex#322). Could I get some assistance?
So @ericphanson found that there were some quirks in my setup because my setup was relying on that I develop in one repository. If you need to set it up for a GitHub organization (or you have a large flow of PRs), Convex.jl’s setup is probably the best starting point. See: https://github.com/JuliaOpt/Convex.jl/pull/323
I’m wondering if anyone has a similar setup using GitHub workflows. I tried mimicking what discussed above by @tkf and @ericphanson but I’m getting the following
/home/runner/work/ProximalOperators.jl/ProximalOperators.jl is dirty.
Please commit/stash your changes before benchmarking a specific commit
I think this has to do with the benchmark tuning data being persisted, which prevents PkgBenchmark from being able to switch branches. Any idea how to solve that?
Full log here, configuration here. Any hint is appreciated, thanks!
Of course! Thank you so much