Chairmarks.jl

Package Announcements
, ,
82

Huh


julia> using Random, Chairmarks

julia> Random.seed!(0);

julia> @b rand((1, 0.0, 1+5im, big(5), big(1.0))) sin
8.739 μs (14.00 allocs: 432.278 bytes)

julia> Random.seed!(0);

julia> @b rand((1, 0.0, 1+5im, big(5), big(1.0))) sin
16.496 ns

julia> Random.seed!(1);

julia> @b rand((1, 0.0, 1+5im, big(5), big(1.0))) sin
28.667 ns

julia> Random.seed!(1);

julia> @b rand((1, 0.0, 1+5im, big(5), big(1.0))) sin
19.667 ns

julia> versioninfo()
Julia Version 1.10.1
Commit 7790d6f064 (2024-02-13 20:41 UTC)
Platform Info:
  OS: Linux (x86_64-redhat-linux)
  CPU: 28 × Intel(R) Core(TM) i9-9940X CPU @ 3.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, skylake-avx512)
Threads: 28 default, 0 interactive, 14 GC (on 28 virtual cores)
Environment:
  JULIA_PATH = @.
  LD_LIBRARY_PATH = /usr/local/lib/x86_64-unknown-linux-gnu/:/usr/local/lib/:/usr/local/lib/x86_64-unknown-linux-gnu/:/usr/local/lib/
  JULIA_NUM_THREADS = 28
  LD_UN_PATH = /usr/local/lib/x86_64-unknown-linux-gnu/:/usr/local/lib/

EDIT:
Maybe my random stream is simply different.

julia> Random.seed!(2);

julia> @b rand((1, 0.0, 1+5im, big(5), big(1.0))) sin
34.000 ns

julia> Random.seed!(3);

julia> @b rand((1, 0.0, 1+5im, big(5), big(1.0))) sin
29.000 ns

julia> Random.seed!(4);

julia> @b rand((1, 0.0, 1+5im, big(5), big(1.0))) sin
29.000 ns

julia> Random.seed!(5);

julia> @b rand((1, 0.0, 1+5im, big(5), big(1.0))) sin
16.531 ns
83

I really don’t understand what’s happening here:

It’s a bug :slight_smile:

The issue with @b rand((1, 0.0, 1+5im, big(5), big(1.0))) sin is a bug when benchmarking functions with highly variable runtimes. For example, and this should be reproducible on all systems,

julia> f(::Int) = nothing
f (generic function with 1 method)

julia> f(::Float64) = sleep(.01)
f (generic function with 2 methods)

julia> using StableRNGs

julia> rng = StableRNG(0)
StableRNGs.LehmerRNG(state=0x00000000000000000000000000000001)

julia> @be rand(rng, (1, 2.0)) f
[ Info: Loading Chairmarks ...
Benchmark: 17 samples with 1 evaluation
min    0 ns
median 42.000 ns
mean   5.206 ms (1.88 allocs: 52.706 bytes)
max    11.079 ms (4 allocs: 112 bytes)

julia> rng = StableRNG(1)
StableRNGs.LehmerRNG(state=0x00000000000000000000000000000003)

julia> @be rand(rng, (1, 2.0)) f
[hangs for 5 minutes]

This is because calibration seriously mis-estimates the correct number of evals per sample. A low number of evals will result in a sample runtime below clock precision and a high number will hang.

I’ll post back here when the bug is fixed.

3 Likes
84

When writing tuning, I didn’t anticipate benchmarks which have >2 orders of magnitude of runtime variation depending on the result of setup

julia> @b 1.0 sin
4.840 ns

julia> @b big(1.0) sin
1.105 μs (6 allocs: 168 bytes)
2 Likes