# How to plot benchmarks

Hi all,

I would like to plot some benchmarks, but its not clear to me what approach to take. As a simple example, suppose I have:

``````using BenchmarkTools
using DataFrames
using Statistics

SUITE = BenchmarkGroup()
SUITE[:benchmark] = BenchmarkGroup()

n1s = [10,100,1000]
n2s = n1s

for n1 β n1s
for n2 β n2s
SUITE[:benchmark][n1,n2] = @benchmarkable(
rand(\$n1, \$n2),
evals = 10,
samples = 1000)
end
end

results = run(SUITE)
``````

How do I plot the means as a bar chart (or use violin plots for the distributions)?

My initial inclination was to convert to a dataframe and use StatsPlots.

``````means = mean(results[:benchmark])
df = DataFrame(mean(results[:benchmark]))
transform!(df, :first => AsTable) # how do I name the columns?
``````

There are two problems. First, the means are TrialEstimate types. Second, the units are not constant.

`````` Row β first         second                     x1     x2
β Tupleβ¦        TrialEstβ¦                  Int64  Int64
ββββββΌβββββββββββββββββββββββββββββββββββββββββββββββββββββββ
1 β (10, 1000)    TrialEstimate(11.478 ΞΌs)      10   1000
2 β (1000, 100)   TrialEstimate(106.119 ΞΌs)   1000    100
3 β (100, 100)    TrialEstimate(13.545 ΞΌs)     100    100
4 β (10, 10)      TrialEstimate(442.292 ns)     10     10
5 β (100, 1000)   TrialEstimate(109.565 ΞΌs)    100   1000
6 β (1000, 1000)  TrialEstimate(1.636 ms)     1000   1000
7 β (10, 100)     TrialEstimate(1.706 ΞΌs)       10    100
8 β (1000, 10)    TrialEstimate(12.028 ΞΌs)    1000     10
9 β (100, 10)     TrialEstimate(1.913 ΞΌs)      100     10
``````

This should work:

``````transform(df,
:first => ByRow(identity) => [:n1, :n2],
:second => (t -> time.(t)) => "Time [ns]")
``````
2 Likes

No need to cross the stream for water:

``````using StatsPlots
res = run(SUITE)
plot(res)
``````
3 Likes

Thank you for your replies. The solution provided by bertschi works. I think I can extend it to violin plots, or something else to represent distributional information.

@gustaphe, I was hoping your solution would work, but unfortunately it resulted in the following error:

``````julia> plot(results)
ERROR: Cannot convert BenchmarkGroup to series data for plotting
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:35
[2] _prepare_series_data(x::BenchmarkGroup)
@ RecipesPipeline ~/.julia/packages/RecipesPipeline/BGM3l/src/series.jl:8
[3] _series_data_vector(x::BenchmarkGroup, plotattributes::Dict{Symbol, Any})
@ RecipesPipeline ~/.julia/packages/RecipesPipeline/BGM3l/src/series.jl:36
[4] macro expansion
@ ~/.julia/packages/RecipesPipeline/BGM3l/src/series.jl:129 [inlined]
[5] apply_recipe(plotattributes::AbstractDict{Symbol, Any}, #unused#::Type{RecipesPipeline.SliceIt}, x::Any, y::Any, z::Any)
@ RecipesPipeline ~/.julia/packages/RecipesBase/BRe07/src/RecipesBase.jl:300
[6] _process_userrecipes!(plt::Any, plotattributes::Any, args::Any)
@ RecipesPipeline ~/.julia/packages/RecipesPipeline/BGM3l/src/user_recipe.jl:38
[7] recipe_pipeline!(plt::Any, plotattributes::Any, args::Any)
@ RecipesPipeline ~/.julia/packages/RecipesPipeline/BGM3l/src/RecipesPipeline.jl:72
[8] _plot!(plt::Plots.Plot, plotattributes::Any, args::Any)
@ Plots ~/.julia/packages/Plots/sxUvK/src/plot.jl:223
[9] #plot#188
@ ~/.julia/packages/Plots/sxUvK/src/plot.jl:102 [inlined]
[10] plot(args::Any)
@ Plots ~/.julia/packages/Plots/sxUvK/src/plot.jl:93
[11] top-level scope
@ REPL[17]:1
``````
1 Like

Sorry, I though we had changed that to an extension, you also need `using BenchmarkPlots`

3 Likes

Do you think it would make sense as an extension?

Thank you. That worked.

Note for someone who is using `BenchmarkGroup`, you need to call the group:

`plot(results[:group])`

Yeah, loading Plots and BenchmarkTools should just work. But the deprecation process would be a pita