Making @benchmark outputs statistically meaningful, and actionable

It would appear that randomizing the heap doesn’t eliminate the disparate timings. The higher 3rd run from the previous reloading runs is gone (strange that it’s the 3rd both times), but that disappearance could be by chance. I wonder if samplewise randomization of function addresses can be done by putting include("/user.jl") in the setup expression. Might run into world age issues so maybe data = @invokelatest MyData() could help there.

2 Likes