Benchmarking function (using `@benchmark`) for various input sizes

General Usage Performance
1

I have a function. I want to benchmark it for various inputs (originally I am solving a differential equations over various timespans). Naturally I would do something like:

using BenchmarkTools
bs = 
for l in [10,100,1000]
    b = @benchmark rand(l).^2
end

(probably compressed using map(), but this makes the mwe clearer)
This, however, yields an error:

UndefVarError: l not defined

Stacktrace:
  [1] var"##core#519"()
    @ Main ~/.julia/packages/BenchmarkTools/7xSXH/src/execution.jl:489
  [2] var"##sample#520"(::Tuple{}, __params::BenchmarkTools.Parameters)
    @ Main ~/.julia/packages/BenchmarkTools/7xSXH/src/execution.jl:495
  [3] _run(b::BenchmarkTools.Benchmark, p::BenchmarkTools.Parameters; verbose::Bool, pad::String, kwargs::Base.Pairs{Symbol, Integer, NTuple{4, Symbol}, NamedTuple{(:samples, :evals, :gctrial, :gcsample), Tuple{Int64, Int64, Bool, Bool}}})
    @ BenchmarkTools ~/.julia/packages/BenchmarkTools/7xSXH/src/execution.jl:99
  [4] #invokelatest#2
    @ ./essentials.jl:718 [inlined]
  [5] #run_result#45
    @ ~/.julia/packages/BenchmarkTools/7xSXH/src/execution.jl:34 [inlined]
  [6] run(b::BenchmarkTools.Benchmark, p::BenchmarkTools.Parameters; progressid::Nothing, nleaves::Float64, ndone::Float64, kwargs::Base.Pairs{Symbol, Integer, NTuple{5, Symbol}, NamedTuple{(:verbose, :samples, :evals, :gctrial, :gcsample), Tuple{Bool, Int64, Int64, Bool, Bool}}})
    @ BenchmarkTools ~/.julia/packages/BenchmarkTools/7xSXH/src/execution.jl:117
  [7] #warmup#54
    @ ~/.julia/packages/BenchmarkTools/7xSXH/src/execution.jl:169 [inlined]
  [8] warmup(item::BenchmarkTools.Benchmark)
    @ BenchmarkTools ~/.julia/packages/BenchmarkTools/7xSXH/src/execution.jl:169
  [9] macro expansion
    @ ~/.julia/packages/BenchmarkTools/7xSXH/src/execution.jl:393 [inlined]
 [10] top-level scope
    @ In[42]:4
 [11] eval
    @ ./boot.jl:373 [inlined]
 [12] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
    @ Base ./loading.jl:1196

Is there a way to get around this, or otherwise accomplish what I want to accomplish?

2

first of, for loop does not return so this won’t become an array as you’d probably want it to, also you want to use $:

julia> using BenchmarkTools

julia> bs = map([10,100,1000]) do l
           @benchmark rand($l)
       end;

julia> bs
3-element Vector{BenchmarkTools.Trial}:
 71.882 ns
 160.522 ns
 1.100 ΞΌs

julia> bs[1]
BenchmarkTools.Trial: 10000 samples with 974 evaluations.
 Range (min … max):  71.882 ns … 644.366 ns  β”Š GC (min … max): 0.00% … 85.38%
 Time  (median):     74.186 ns               β”Š GC (median):    0.00%
 Time  (mean Β± Οƒ):   76.662 ns Β±  25.452 ns  β”Š GC (mean Β± Οƒ):  1.72% Β±  4.54%

   β–ƒβ–†β–ˆβ–ˆβ–‡β–„β–β–‚β–‚β–  ▁▁                                              β–‚
  β–‡β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‡β–†β–‡β–‡β–†β–ƒβ–„β–β–ƒβ–„β–„β–ƒβ–„β–ƒβ–ƒβ–β–β–β–β–ƒβ–β–β–β–β–β–β–ƒβ–„β–…β–†β–‡β–‡β–ˆβ–ˆβ–‡β–†β–…β–†β–…β–†β–‡β–ˆβ–ˆβ–ˆ β–ˆ
  71.9 ns       Histogram: log(frequency) by time       104 ns <

 Memory estimate: 144 bytes, allocs estimate: 1.
1 Like
3

Thanks! Yes, I messed up the example a bit. Anyway, the $ solves it! Lots of thanks!