Using Benchmarktools with in-place functions and a default mutable struct

General Usage
1

I want to benchmark a function by every time using the default value of a mutable struct. I tried to recreate a minimal working example of my code and what goes wrong:

mutable struct myNum{T<:Real} 
     n::T; 
     myNum()=new{Real}(2) 
end

function addOne!(myNum::p) 
    p.n=p.n+1 
    println(p) 
end

@benchmark addOne!(p) setup = (p=myNum())

This produces an output similar to

myNum{Real}(3)
myNum{Real}(3)
myNum{Real}(3)
myNum{Real}(3)
myNum{Real}(4)
myNum{Real}(3)
myNum{Real}(4)
myNum{Real}(5)
myNum{Real}(3)
myNum{Real}(4)
myNum{Real}(5)
myNum{Real}(6)
myNum{Real}(3)
myNum{Real}(4)
myNum{Real}(5)
myNum{Real}(6)
myNum{Real}(7)
myNum{Real}(3)
myNum{Real}(4)
myNum{Real}(5)
myNum{Real}(6)
myNum{Real}(7)
myNum{Real}(8)
myNum{Real}(3)
myNum{Real}(4)
myNum{Real}(5)
myNum{Real}(6)
myNum{Real}(7)
myNum{Real}(8)
myNum{Real}(9)
myNum{Real}(3)
myNum{Real}(4)
myNum{Real}(5)
myNum{Real}(6)
myNum{Real}(7)
myNum{Real}(8)
myNum{Real}(9)
myNum{Real}(10)
myNum{Real}(3)
myNum{Real}(4)
myNum{Real}(5)
myNum{Real}(6)
myNum{Real}(7)
myNum{Real}(8)
myNum{Real}(9)
myNum{Real}(10)
etc.

while I would want myNum{Real}(3) every time. I tried other things like @benchmark addOne!(p) setup = (p=$myNum()) and q=myNum(); @benchmark addOne!(p) setup = (p=$q) but this doesn’t seem to fix anything. I wonder why this is and how to fix it?

2 
@benchmark addOne!(p) setup = (p=myNum()) evals=1

Should do it, but also see `tune!` on benchmarkable with evals set. · Issue #328 · JuliaCI/BenchmarkTools.jl · GitHub

1 Like
3

Funny enough I asked about a similar issue on Slack the other day. It was pointed out to me by @Sukera that @benchmark can be thought of as something like

for _ in 1:nsamples
    # setup runs here
    for _ in 1:nevals
         # the benchmarking expression runs here
    end
end

So multiple calls to your code can happen between each call to the setup code you provide. The solution @vchuravy gave with evals=1 ensures you only run the inner loop once per setup call.

2 Likes