PkgBenchmark.jl workflow

General Usage
,
1

This is basically a revival of Benchmarking with PkgBenchmark.jl from two years ago.

Benchmarking packages to detect regressions seems very important, but I don’t quite get how to use PkgBenchmark.jl effectively. In particular, I don’t understand from where I should run the benchmarks defined in my benchmark/benchmarks.jl. Naively I put a run_benchmarks.jl script (see below) into the same folder. This fails however as soon as I try to benchmark a different branch as it tries to checkout this branch, which fails due to an active process still accessing files/folders in the repo - me running the run_benchmarks.jl.

Of course, automizing this in some way would be great (but maybe unrealistic?).

I’d really appreciate any pointers to documentations, tutorials, good examples in existing packages, and other resources.

Thanks in advance!

# run_benchmarks.jl
# A branch name or commit id can be supplied as an argument.

using Pkg
Pkg.activate(@__DIR__)

using PkgBenchmark

branch_or_commit = nothing
if length(ARGS) > 0
    branch_or_commit = ARGS[1]
end

out = isnothing(branch_or_commit) ? "master" : string(branch_or_commit)

config = BenchmarkConfig(id = out,
                         juliacmd = `julia -O3 --project=.`,
                         env = Dict("JULIA_NUM_THREADS" => 1,
                                    "OMP_NUM_THREADS" => Sys.CPU_THREADS,
                                    "MKL_NUM_THREADS" => Sys.CPU_THREADS))


r = benchmarkpkg("StableDQMC", config;
             script = "benchmark/benchmarks.jl",
             resultfile = string(out, ".bench"))

export_markdown(string(out, ".md"), r)
6 Likes
2

Have you looked at PkgBenchmark.judge? It allows you to run the benchmarks on two different branches/commits/etc and compare the results.

1 Like
3

(Continuing the discussion in How to run benchmark CI on every commit? - #3 by tkf)

FYI I just set up CI to run PkgBenchmark.judge https://github.com/tkf/Transducers.jl/pull/32 and I find it pretty useful. Feel free to grab some helper script/setting from there. There were some bits that were not super obvious (like how to invoke git fetch to pull the master branch so that PkgBenchmark can switch to it.).

5 Likes
4

I tried mimicking your .travis.yml (and used your awesome pretty printing) for Convex.jl (PR: Convex#321; travis file: https://github.com/JuliaOpt/Convex.jl/blob/master/.travis.yml). I think I messed something up though, because CI didn’t run on the next PR I made (Convex#322). Could I get some assistance? :slight_smile:

5

So @ericphanson found that there were some quirks in my setup because my setup was relying on that I develop in one repository. If you need to set it up for a GitHub organization (or you have a large flow of PRs), Convex.jl’s setup is probably the best starting point. See: https://github.com/JuliaOpt/Convex.jl/pull/323

1 Like
6

I’m wondering if anyone has a similar setup using GitHub workflows. I tried mimicking what discussed above by @tkf and @ericphanson but I’m getting the following

ERROR: LoadError: 
/home/runner/work/ProximalOperators.jl/ProximalOperators.jl is dirty.
Please commit/stash your changes before benchmarking a specific commit

I think this has to do with the benchmark tuning data being persisted, which prevents PkgBenchmark from being able to switch branches. Any idea how to solve that?

Full log here, configuration here. Any hint is appreciated, thanks! :pray:

7

We ran into this too for Convex.jl but just added the generated files to the gitignore: https://github.com/JuliaOpt/Convex.jl/blob/5d8f1086949b92c98eaa152976d3caa810c5ec37/.gitignore#L8.

3 Likes
8

Of course! :man_facepalming: Thank you so much

1 Like