PkgBenchmark.jl workflow

This is basically a revival of Benchmarking with PkgBenchmark.jl from two years ago.

Benchmarking packages to detect regressions seems very important, but I don’t quite get how to use PkgBenchmark.jl effectively. In particular, I don’t understand from where I should run the benchmarks defined in my benchmark/benchmarks.jl. Naively I put a run_benchmarks.jl script (see below) into the same folder. This fails however as soon as I try to benchmark a different branch as it tries to checkout this branch, which fails due to an active process still accessing files/folders in the repo - me running the run_benchmarks.jl.

Of course, automizing this in some way would be great (but maybe unrealistic?).

I’d really appreciate any pointers to documentations, tutorials, good examples in existing packages, and other resources.

Thanks in advance!

# run_benchmarks.jl
# A branch name or commit id can be supplied as an argument.

using Pkg

using PkgBenchmark

branch_or_commit = nothing
if length(ARGS) > 0
    branch_or_commit = ARGS[1]

out = isnothing(branch_or_commit) ? "master" : string(branch_or_commit)

config = BenchmarkConfig(id = out,
                         juliacmd = `julia -O3 --project=.`,
                         env = Dict("JULIA_NUM_THREADS" => 1,
                                    "OMP_NUM_THREADS" => Sys.CPU_THREADS,
                                    "MKL_NUM_THREADS" => Sys.CPU_THREADS))

r = benchmarkpkg("StableDQMC", config;
             script = "benchmark/benchmarks.jl",
             resultfile = string(out, ".bench"))

export_markdown(string(out, ".md"), r)