Should Julia's microbenchmark include a benchmark Julia crushes at?

Community
1

According to Chris’s recent blog Julia’s real strength doesn’t show up in microbenchmarks. Any competent language with that can figure out what type things are should basically be identical in microbenchmarks (letting slide factors of 10% or so). But Julia lets a domain expert write an awesome algorithm that might be 10 times faster for some problems, and lets random users take advantage of that even with non-Base types, like GPUArray or Measurement. This comes down to zero cost abstraction that works across modules written independently. So could there be a benchmark in the Julia microbenchmarks that shows off this strength, that cython and numba would look worse at? Can we make a microbenchmark that tells story? The microbenchmarks are a tool to show off the strengths of Julia, they should show cases where Julia wins.

Anyway, here is my attempt. I make an Uncertain type that supports addition in one module, and a sum function in another module, then call sum on a Vector{Uncertain} in a 3rd module. I think for this to foil cython/numba there would have to be an additional rule that the implementation has to be such that these modules could live in 3 separate packages with no specific glue code between Uncertains and Sums. Just like would be the case when a new user downloads two packages and tries to use them. This behavior is easy to achieve in pure python, so I think it (or something like it) would be a nice demonstration that julia can do the generic programming that pure python can do, but fast.

module Uncertains
struct Uncertain{T}
    v::T
    σ::T
end
Base.:+(x::Uncertain,y::Uncertain) = Uncertain(x.v+y.v,sqrt(x.σ*x.σ+y.σ*y.σ))
Base.zero(::Type{Uncertain{T}}) where T = Uncertain(zero(T),zero(T))
end

module Sums
function sum(v)
    s = zero(eltype(v))
    for x in v
        s+=x
    end
    s
end
end

module Usercode
import Sums, Uncertains
function getdata()
    v = [Uncertains.Uncertain(i,0.1) for i=1.0:10000.0]
end
function processdata(v)
    return Sums.sum(v)
end
end

using BenchmarkTools
v = Usercode.getdata();
@btime Usercode.processdata($v)
8 Likes