Chairmarks supports comparative benchmarking!

Package Announcements
1

Chairmarks version 1.3.0 supports benchmarking two (or more) functions at the same time. Simply provide the functions separated by commas:

julia> @b rand(3) objectid
4.257 ns

julia> @b rand(3) hash
7.360 ns

julia> @b rand(3) objectid,hash
(4.182 ns, 7.342 ns)

This usage and syntax is currently experimental, so please try it out and let me know here or in this thread if you encounter any issues or have feedback to share before I finalize this feature :slight_smile:

When benchmarking two functions simultaneously, they each recieve the same number of evaluations and samples are interleaved randomly to eliminate some sources of noise and bias you would see if you ran the benchmarks serially. It’s intended to be convenient syntax and sound semantics for a common benchmarking usecase.

25 Likes
2

To clarify the syntax here, the argument order remains the same:

@b f
@b setup f
@b setup f teardown
@b init setup f teardown

and if f is a tuple syntactically, then we use comparative benchmarking. Other arguments are still the same for both versions, even if they are syntactic tuplse. So for example,

@b rand(),rand() fld(_...)

Remains unchanged whereas

@b rand(),rand() fld(_...),cld(_...)

used to benchmark the function (x) -> (fld(x...),cld(x...)) and now benchmarks both the function (x) -> fld(x...) and the function (x) -> cld(x...) in parallel but measured separately.

1 Like