PkgBenchmark.jl workflow

This is basically a revival of Benchmarking with PkgBenchmark.jl from two years ago.

Benchmarking packages to detect regressions seems very important, but I don’t quite get how to use PkgBenchmark.jl effectively. In particular, I don’t understand from where I should run the benchmarks defined in my benchmark/benchmarks.jl. Naively I put a run_benchmarks.jl script (see below) into the same folder. This fails however as soon as I try to benchmark a different branch as it tries to checkout this branch, which fails due to an active process still accessing files/folders in the repo - me running the run_benchmarks.jl.

Of course, automizing this in some way would be great (but maybe unrealistic?).

I’d really appreciate any pointers to documentations, tutorials, good examples in existing packages, and other resources.

Thanks in advance!

# run_benchmarks.jl
# A branch name or commit id can be supplied as an argument.

using Pkg
Pkg.activate(@__DIR__)

using PkgBenchmark

branch_or_commit = nothing
if length(ARGS) > 0
    branch_or_commit = ARGS[1]
end

out = isnothing(branch_or_commit) ? "master" : string(branch_or_commit)

config = BenchmarkConfig(id = out,
                         juliacmd = `julia -O3 --project=.`,
                         env = Dict("JULIA_NUM_THREADS" => 1,
                                    "OMP_NUM_THREADS" => Sys.CPU_THREADS,
                                    "MKL_NUM_THREADS" => Sys.CPU_THREADS))


r = benchmarkpkg("StableDQMC", config;
             script = "benchmark/benchmarks.jl",
             resultfile = string(out, ".bench"))

export_markdown(string(out, ".md"), r)
4 Likes

Have you looked at PkgBenchmark.judge? It allows you to run the benchmarks on two different branches/commits/etc and compare the results.

1 Like

(Continuing the discussion in How to run benchmark CI on every commit?)

FYI I just set up CI to run PkgBenchmark.judge https://github.com/tkf/Transducers.jl/pull/32 and I find it pretty useful. Feel free to grab some helper script/setting from there. There were some bits that were not super obvious (like how to invoke git fetch to pull the master branch so that PkgBenchmark can switch to it.).

4 Likes

I tried mimicking your .travis.yml (and used your awesome pretty printing) for Convex.jl (PR: Convex#321; travis file: https://github.com/JuliaOpt/Convex.jl/blob/master/.travis.yml). I think I messed something up though, because CI didn’t run on the next PR I made (Convex#322). Could I get some assistance? :slight_smile:

So @ericphanson found that there were some quirks in my setup because my setup was relying on that I develop in one repository. If you need to set it up for a GitHub organization (or you have a large flow of PRs), Convex.jl’s setup is probably the best starting point. See: https://github.com/JuliaOpt/Convex.jl/pull/323

1 Like