How to plot benchmarks

General Usage
, ,
1

Hi all,

I would like to plot some benchmarks, but its not clear to me what approach to take. As a simple example, suppose I have:

using BenchmarkTools
using DataFrames
using Statistics 


SUITE = BenchmarkGroup()
SUITE[:benchmark] = BenchmarkGroup()

n1s = [10,100,1000]
n2s = n1s

for n1 ∈ n1s 
    for n2 ∈ n2s 
        SUITE[:benchmark][n1,n2] = @benchmarkable(
            rand($n1, $n2),
            evals = 10,
            samples = 1000)
    end
end

results = run(SUITE)

How do I plot the means as a bar chart (or use violin plots for the distributions)?

My initial inclination was to convert to a dataframe and use StatsPlots.

means = mean(results[:benchmark])
df = DataFrame(mean(results[:benchmark]))
transform!(df, :first => AsTable) # how do I name the columns?

There are two problems. First, the means are TrialEstimate types. Second, the units are not constant.

 Row β”‚ first         second                     x1     x2    
     β”‚ Tuple…        TrialEst…                  Int64  Int64 
─────┼───────────────────────────────────────────────────────
   1 β”‚ (10, 1000)    TrialEstimate(11.478 ΞΌs)      10   1000
   2 β”‚ (1000, 100)   TrialEstimate(106.119 ΞΌs)   1000    100
   3 β”‚ (100, 100)    TrialEstimate(13.545 ΞΌs)     100    100
   4 β”‚ (10, 10)      TrialEstimate(442.292 ns)     10     10
   5 β”‚ (100, 1000)   TrialEstimate(109.565 ΞΌs)    100   1000
   6 β”‚ (1000, 1000)  TrialEstimate(1.636 ms)     1000   1000
   7 β”‚ (10, 100)     TrialEstimate(1.706 ΞΌs)       10    100
   8 β”‚ (1000, 10)    TrialEstimate(12.028 ΞΌs)    1000     10
   9 β”‚ (100, 10)     TrialEstimate(1.913 ΞΌs)      100     10
2

This should work:

transform(df,
          :first => ByRow(identity) => [:n1, :n2],
          :second => (t -> time.(t)) => "Time [ns]")
3

No need to cross the stream for water:

using StatsPlots
res = run(SUITE)
plot(res)
4

Thank you for your replies. The solution provided by bertschi works. I think I can extend it to violin plots, or something else to represent distributional information.

@gustaphe, I was hoping your solution would work, but unfortunately it resulted in the following error:

julia> plot(results)
ERROR: Cannot convert BenchmarkGroup to series data for plotting
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] _prepare_series_data(x::BenchmarkGroup)
    @ RecipesPipeline ~/.julia/packages/RecipesPipeline/BGM3l/src/series.jl:8
  [3] _series_data_vector(x::BenchmarkGroup, plotattributes::Dict{Symbol, Any})
    @ RecipesPipeline ~/.julia/packages/RecipesPipeline/BGM3l/src/series.jl:36
  [4] macro expansion
    @ ~/.julia/packages/RecipesPipeline/BGM3l/src/series.jl:129 [inlined]
  [5] apply_recipe(plotattributes::AbstractDict{Symbol, Any}, #unused#::Type{RecipesPipeline.SliceIt}, x::Any, y::Any, z::Any)
    @ RecipesPipeline ~/.julia/packages/RecipesBase/BRe07/src/RecipesBase.jl:300
  [6] _process_userrecipes!(plt::Any, plotattributes::Any, args::Any)
    @ RecipesPipeline ~/.julia/packages/RecipesPipeline/BGM3l/src/user_recipe.jl:38
  [7] recipe_pipeline!(plt::Any, plotattributes::Any, args::Any)
    @ RecipesPipeline ~/.julia/packages/RecipesPipeline/BGM3l/src/RecipesPipeline.jl:72
  [8] _plot!(plt::Plots.Plot, plotattributes::Any, args::Any)
    @ Plots ~/.julia/packages/Plots/sxUvK/src/plot.jl:223
  [9] #plot#188
    @ ~/.julia/packages/Plots/sxUvK/src/plot.jl:102 [inlined]
 [10] plot(args::Any)
    @ Plots ~/.julia/packages/Plots/sxUvK/src/plot.jl:93
 [11] top-level scope
    @ REPL[17]:1
5

Sorry, I though we had changed that to an extension, you also need using BenchmarkPlots

6

Do you think it would make sense as an extension?

7

Thank you. That worked.

Note for someone who is using BenchmarkGroup, you need to call the group:

plot(results[:group])

8

Yeah, loading Plots and BenchmarkTools should just work. But the deprecation process would be a pita :smiley: