Use tags, setup, teardown with the `@benchmarkset` macro

General Usage
1

Is there a way to use tags with the @benchmarkset macro? What about setup and teardown? I’ve been searching the docsumentation but to no avail.

2

API Reference mentions setup in @case. Is that what you want?

But you are right. The @benchmarkset is a bit of a mystery and I even think it malfunctions a bit.
E.g. the example code doesn’t produce 5 cases as I would expect but just one:

julia> @benchmarkset "suite" for k in 1:5
           @case "case $k" rand($k, $k)
       end
1-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "suite" => 1-element BenchmarkTools.BenchmarkGroup:
	 tags: []
	 "case 5" => Benchmark(evals=1, seconds=5.0, samples=10000)

I would expect it to be similar to the following code block:

julia> bg = BenchmarkGroup([], "suite" => BenchmarkGroup([]));

julia> foreach(1:5) do k
           bg["suite"]["case $k"] = @benchmarkable rand($k, $k)
       end

julia> bg
1-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "suite" => 5-element BenchmarkTools.BenchmarkGroup:
	 tags: []
	 "case 2" => Benchmark(evals=1, seconds=5.0, samples=10000)
	 "case 4" => Benchmark(evals=1, seconds=5.0, samples=10000)
	 "case 5" => Benchmark(evals=1, seconds=5.0, samples=10000)
	 "case 3" => Benchmark(evals=1, seconds=5.0, samples=10000)
	 "case 1" => Benchmark(evals=1, seconds=5.0, samples=10000)

In the end, you can ofc use the second approach to reliably do all you setup, teardown stuff.

EDIT: opened an issue is `@benchmarkset` usable ? · Issue #343 · JuliaCI/BenchmarkTools.jl · GitHub

3

Yeah, that’s what I’ve been doing. It works quite well with functions that generate the benchmark cases.