This is basically a revival of Benchmarking with PkgBenchmark.jl from two years ago.
Benchmarking packages to detect regressions seems very important, but I don’t quite get how to use PkgBenchmark.jl effectively. In particular, I don’t understand from where I should run the benchmarks defined in my
benchmark/benchmarks.jl. Naively I put a
run_benchmarks.jl script (see below) into the same folder. This fails however as soon as I try to benchmark a different branch as it tries to checkout this branch, which fails due to an active process still accessing files/folders in the repo - me running the
Of course, automizing this in some way would be great (but maybe unrealistic?).
I’d really appreciate any pointers to documentations, tutorials, good examples in existing packages, and other resources.
Thanks in advance!
# run_benchmarks.jl # A branch name or commit id can be supplied as an argument. using Pkg Pkg.activate(@__DIR__) using PkgBenchmark branch_or_commit = nothing if length(ARGS) > 0 branch_or_commit = ARGS end out = isnothing(branch_or_commit) ? "master" : string(branch_or_commit) config = BenchmarkConfig(id = out, juliacmd = `julia -O3 --project=.`, env = Dict("JULIA_NUM_THREADS" => 1, "OMP_NUM_THREADS" => Sys.CPU_THREADS, "MKL_NUM_THREADS" => Sys.CPU_THREADS)) r = benchmarkpkg("StableDQMC", config; script = "benchmark/benchmarks.jl", resultfile = string(out, ".bench")) export_markdown(string(out, ".md"), r)