Unintuitive behaviour with BenchmarkTools

General Usage
1

I’m baffled by this error. Here is the relevant code fragment:

  function f1(p)
    total = 0.
    
    for i in 1:length(p)-2
      q = p[i]
      a = p[i+1]
      b = p[i+2]
    
      dx = b.x - a.x
      dy = b.y - a.y
      total += abs(q.x * dy - q.y * dx + b.x * a.y - b.y * a.x);
    end

    return total
  end

  println(f1(pointsxy))
  @benchmark f1($pointsxy)

and the REPL says:

156.06603667791933
ERROR: LoadError: UndefVarError: f1 not defined
Stacktrace:
 [1] ##core#261() at /home/paul/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:371
 [2] ##sample#262(::BenchmarkTools.Parameters) at /home/paul/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:377
 [3] _run(::BenchmarkTools.Benchmark{Symbol("##benchmark#260")}, ::BenchmarkTools.Parameters; verbose::Bool, pad::String, kwargs::Base.Iterators.Pairs{Symbol,Integer,NTuple{4,Symbol},NamedTuple{(:samples, :evals, :gctrial, :gcsample),Tuple{Int64,Int64,Bool,Bool}}}) at /home/paul/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:405
 [4] (::Base.var"#inner#2"{Base.Iterators.Pairs{Symbol,Integer,NTuple{5,Symbol},NamedTuple{(:verbose, :samples, :evals, :gctrial, :gcsample),Tuple{Bool,Int64,Int64,Bool,Bool}}},typeof(BenchmarkTools._run),Tuple{BenchmarkTools.Benchmark{Symbol("##benchmark#260")},BenchmarkTools.Parameters}})() at ./essentials.jl:715
 [5] #invokelatest#1 at ./essentials.jl:716 [inlined]
 [6] #run_result#37 at /home/paul/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:32 [inlined]
 [7] run(::BenchmarkTools.Benchmark{Symbol("##benchmark#260")}, ::BenchmarkTools.Parameters; progressid::Nothing, nleaves::Float64, ndone::Float64, kwargs::Base.Iterators.Pairs{Symbol,Integer,NTuple{5,Symbol},NamedTuple{(:verbose, :samples, :evals, :gctrial, :gcsample),Tuple{Bool,Int64,Int64,Bool,Bool}}}) at /home/paul/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:94
 [8] #warmup#45 at /home/paul/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:141 [inlined]
 [9] warmup(::BenchmarkTools.Benchmark{Symbol("##benchmark#260")}) at /home/paul/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:141
 [10] top-level scope at /home/paul/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:287
 [11] top-level scope at /home/paul/st/julia/bench6.jl:35
 [12] include_string(::Module, ::String, ::String) at ./loading.jl:1080
in expression starting at /home/paul/st/julia/bench6.jl:4

f1 is defined and produces the result shown in the first line. What is happening here?

2

Works for me.

julia> pointsxy = [p(1.0, 2.0), p(3.0, 4.0), p(-2.0, -1.0), p(2.2, 1.1)]
4-element Array{p,1}:
 p(1.0, 2.0)
 p(3.0, 4.0)
 p(-2.0, -1.0)
 p(2.2, 1.1)

julia>   println(f1(pointsxy))
10.5

julia>   @benchmark f1($pointsxy)
BenchmarkTools.Trial:
  memory estimate:  352 bytes
  allocs estimate:  22
  --------------
  minimum time:     589.266 ns (0.00% GC)
  median time:      614.119 ns (0.00% GC)
  mean time:        803.968 ns (0.71% GC)
  maximum time:     7.509 μs (82.94% GC)
  --------------
  samples:          10000
  evals/sample:     177

julia>     

julia> versioninfo()
Julia Version 1.6.0-DEV.222
Commit 5ecb8e2c15 (2020-06-13 18:08 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-6300U CPU @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)

(@v1.6) pkg> st
Status `C:\Users\pkonl\.julia\environments\v1.6\Project.toml`
  [6e4b80f9] BenchmarkTools v0.5.0
  [682c06a0] JSON v0.21.0
  [c3e4b0f8] Pluto v0.10.5
  [90137ffa] StaticArrays v0.12.4
  [b8865327] UnicodePlots v1.2.0
  [8bb1440f] DelimitedFiles

(@v1.6) pkg>
1 Like
3

After I created another, simpler example it started to work. What is puzzling, the first time I’ve seen this error I did:

  1. Reload the REPL - no effect
  2. Reload VSCode - no effect
  3. Created simpler example - works fine
  4. Tried the original file - works fine

I can’t find any logical explanation for this. The error itself was absurd enough.

4

To make sure I’m not hallucinating, I reloaded VSCode and run this file:

using BenchmarkTools, Random
using StaticArrays, LinearAlgebra

let 
  struct Point
    x::Float32
    y::Float32
  end
  
  Point((x, y)::Tuple{Real, Real}) = Point(x, y)
  
  Vec2f = SVector{2, Float32}
  np = 1000
  points2d = rand(MersenneTwister(12345), Vec2f, np)
  pointsxy = reinterpret(Point, points2d)

  function f1(p)
    total = 0.
    
    for i in 1:length(p)-2
      q = p[i]
      a = p[i+1]
      b = p[i+2]
    
      dx = b.x - a.x
      dy = b.y - a.y
      total += abs(q.x * dy - q.y * dx + b.x * a.y - b.y * a.x);
    end

    return total
  end

  println(f1(pointsxy))
  @benchmark f1($pointsxy)
end

I’m getting the same error again:

156.06603667791933
ERROR: LoadError: UndefVarError: f1 not defined
5

Your function is scoped to the let block.

1 Like
6

I’ve never used a let block. Why are you using it here?

7

But @benchmark is called inside let block.

8

I’m experimenting with different types and there are name collisions when I do it in global space.

9

I see. I’ve always had trouble when I tried using BenchmarkTools in a local scope. Not sure why…

1 Like
10

@benchmark calls eval. Arguments to eval are evaluated in global scope.

3 Likes
11

There is the explanation, then. Thanks @oheil.

1 Like
12

I made a pull request: https://github.com/JuliaCI/BenchmarkTools.jl/pull/172

Thanks everyone for your help in solving this puzzle.

4 Likes
13

May be I was not exact enough. Looking at the code ( @edit @benchmark ) it is Core.eval(m::Module, expr) which is called. But in this case with let...end block, m::Module scope is again Global. :slight_smile:
Anyways, the pull request is a good idea and the experts will probably make it correct.

1 Like