# Unintuitive behaviour with BenchmarkTools

**URL:** <https://discourse.julialang.org/t/unintuitive-behaviour-with-benchmarktools/43291>\
**Category:** General Usage\
**Created:** [July 18, 2020, 3:50pm UTC](https://discourse.julialang.org/t/unintuitive-behaviour-with-benchmarktools/43291 "2020-07-18T15:50:57Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![pauljurczak](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pauljurczak/32/921_2.png) [@pauljurczak](https://discourse.julialang.org/u/pauljurczak)\
**Post date:** [July 18, 2020, 3:50pm UTC](https://discourse.julialang.org/t/unintuitive-behaviour-with-benchmarktools/43291/1 "2020-07-18T15:50:57Z")

</div>

I’m baffled by this error. Here is the relevant code fragment:

```julia
  function f1(p)
    total = 0.
    
    for i in 1:length(p)-2
      q = p[i]
      a = p[i+1]
      b = p[i+2]
    
      dx = b.x - a.x
      dy = b.y - a.y
      total += abs(q.x * dy - q.y * dx + b.x * a.y - b.y * a.x);
    end

    return total
  end

  println(f1(pointsxy))
  @benchmark f1($pointsxy)

```

and the REPL says:

```julia
156.06603667791933
ERROR: LoadError: UndefVarError: f1 not defined
Stacktrace:
 [1] ##core#261() at /home/paul/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:371
 [2] ##sample#262(::BenchmarkTools.Parameters) at /home/paul/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:377
 [3] _run(::BenchmarkTools.Benchmark{Symbol("##benchmark#260")}, ::BenchmarkTools.Parameters; verbose::Bool, pad::String, kwargs::Base.Iterators.Pairs{Symbol,Integer,NTuple{4,Symbol},NamedTuple{(:samples, :evals, :gctrial, :gcsample),Tuple{Int64,Int64,Bool,Bool}}}) at /home/paul/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:405
 [4] (::Base.var"#inner#2"{Base.Iterators.Pairs{Symbol,Integer,NTuple{5,Symbol},NamedTuple{(:verbose, :samples, :evals, :gctrial, :gcsample),Tuple{Bool,Int64,Int64,Bool,Bool}}},typeof(BenchmarkTools._run),Tuple{BenchmarkTools.Benchmark{Symbol("##benchmark#260")},BenchmarkTools.Parameters}})() at ./essentials.jl:715
 [5] #invokelatest#1 at ./essentials.jl:716 [inlined]
 [6] #run_result#37 at /home/paul/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:32 [inlined]
 [7] run(::BenchmarkTools.Benchmark{Symbol("##benchmark#260")}, ::BenchmarkTools.Parameters; progressid::Nothing, nleaves::Float64, ndone::Float64, kwargs::Base.Iterators.Pairs{Symbol,Integer,NTuple{5,Symbol},NamedTuple{(:verbose, :samples, :evals, :gctrial, :gcsample),Tuple{Bool,Int64,Int64,Bool,Bool}}}) at /home/paul/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:94
 [8] #warmup#45 at /home/paul/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:141 [inlined]
 [9] warmup(::BenchmarkTools.Benchmark{Symbol("##benchmark#260")}) at /home/paul/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:141
 [10] top-level scope at /home/paul/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:287
 [11] top-level scope at /home/paul/st/julia/bench6.jl:35
 [12] include_string(::Module, ::String, ::String) at ./loading.jl:1080
in expression starting at /home/paul/st/julia/bench6.jl:4

```

`f1` **is** defined and produces the result shown in the first line. What is happening here?

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [July 18, 2020, 4:11pm UTC](https://discourse.julialang.org/t/unintuitive-behaviour-with-benchmarktools/43291/2 "2020-07-18T16:11:12Z")

</div>

Works for me.

```julia
julia> pointsxy = [p(1.0, 2.0), p(3.0, 4.0), p(-2.0, -1.0), p(2.2, 1.1)]
4-element Array{p,1}:
 p(1.0, 2.0)
 p(3.0, 4.0)
 p(-2.0, -1.0)
 p(2.2, 1.1)

julia> println(f1(pointsxy))
10.5

julia> @benchmark f1($pointsxy)
BenchmarkTools.Trial:
  memory estimate: 352 bytes
  allocs estimate: 22
  --------------
  minimum time: 589.266 ns (0.00% GC)
  median time: 614.119 ns (0.00% GC)
  mean time: 803.968 ns (0.71% GC)
  maximum time: 7.509 μs (82.94% GC)
  --------------
  samples: 10000
  evals/sample: 177

julia>     

julia> versioninfo()
Julia Version 1.6.0-DEV.222
Commit 5ecb8e2c15 (2020-06-13 18:08 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-6300U CPU @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)

(@v1.6) pkg> st
Status `C:\Users\pkonl\.julia\environments\v1.6\Project.toml`
  [6e4b80f9] BenchmarkTools v0.5.0
  [682c06a0] JSON v0.21.0
  [c3e4b0f8] Pluto v0.10.5
  [90137ffa] StaticArrays v0.12.4
  [b8865327] UnicodePlots v1.2.0
  [8bb1440f] DelimitedFiles

(@v1.6) pkg>                   

```

---

<div class="post-metadata">

**Author:** ![pauljurczak](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pauljurczak/32/921_2.png) [@pauljurczak](https://discourse.julialang.org/u/pauljurczak)\
**Post date:** [July 18, 2020, 4:33pm UTC](https://discourse.julialang.org/t/unintuitive-behaviour-with-benchmarktools/43291/3 "2020-07-18T16:33:27Z")

</div>

> [@PetrKryslUCSD](#):
>
> Works for me

After I created another, simpler example it started to work. What is puzzling, the first time I’ve seen this error I did:

1. Reload the REPL - no effect
2. Reload VSCode - no effect
3. Created simpler example - works fine
4. Tried the original file - works fine

I can’t find any logical explanation for this. The error itself was absurd enough.

---

<div class="post-metadata">

**Author:** ![pauljurczak](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pauljurczak/32/921_2.png) [@pauljurczak](https://discourse.julialang.org/u/pauljurczak)\
**Post date:** [July 18, 2020, 4:42pm UTC](https://discourse.julialang.org/t/unintuitive-behaviour-with-benchmarktools/43291/4 "2020-07-18T16:42:00Z")

</div>

To make sure I’m not hallucinating, I reloaded VSCode and run this file:

```julia
using BenchmarkTools, Random
using StaticArrays, LinearAlgebra

let 
  struct Point
    x::Float32
    y::Float32
  end
  
  Point((x, y)::Tuple{Real, Real}) = Point(x, y)
  
  Vec2f = SVector{2, Float32}
  np = 1000
  points2d = rand(MersenneTwister(12345), Vec2f, np)
  pointsxy = reinterpret(Point, points2d)

  function f1(p)
    total = 0.
    
    for i in 1:length(p)-2
      q = p[i]
      a = p[i+1]
      b = p[i+2]
    
      dx = b.x - a.x
      dy = b.y - a.y
      total += abs(q.x * dy - q.y * dx + b.x * a.y - b.y * a.x);
    end

    return total
  end

  println(f1(pointsxy))
  @benchmark f1($pointsxy)
end

```

I’m getting the same error again:

```julia
156.06603667791933
ERROR: LoadError: UndefVarError: f1 not defined

```

---

<div class="post-metadata">

**Author:** ![yurivish](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yurivish/32/307_2.png) [@yurivish](https://discourse.julialang.org/u/yurivish)\
**Post date:** [July 18, 2020, 4:45pm UTC](https://discourse.julialang.org/t/unintuitive-behaviour-with-benchmarktools/43291/5 "2020-07-18T16:45:50Z")

</div>

Your function is scoped to the `let` block.

---

<div class="post-metadata">

**Author:** ![klaff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/klaff/32/7637_2.png) [@klaff](https://discourse.julialang.org/u/klaff)\
**Post date:** [July 18, 2020, 4:49pm UTC](https://discourse.julialang.org/t/unintuitive-behaviour-with-benchmarktools/43291/6 "2020-07-18T16:49:04Z")

</div>

I’ve never used a `let` block. Why are you using it here?

---

<div class="post-metadata">

**Author:** ![pauljurczak](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pauljurczak/32/921_2.png) [@pauljurczak](https://discourse.julialang.org/u/pauljurczak)\
**Post date:** [July 18, 2020, 4:52pm UTC](https://discourse.julialang.org/t/unintuitive-behaviour-with-benchmarktools/43291/7 "2020-07-18T16:52:17Z")

</div>

But `@benchmark` is called inside `let` block.

---

<div class="post-metadata">

**Author:** ![pauljurczak](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pauljurczak/32/921_2.png) [@pauljurczak](https://discourse.julialang.org/u/pauljurczak)\
**Post date:** [July 18, 2020, 4:53pm UTC](https://discourse.julialang.org/t/unintuitive-behaviour-with-benchmarktools/43291/8 "2020-07-18T16:53:51Z")

</div>

I’m experimenting with different types and there are name collisions when I do it in global space.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [July 18, 2020, 5:05pm UTC](https://discourse.julialang.org/t/unintuitive-behaviour-with-benchmarktools/43291/9 "2020-07-18T17:05:33Z")

</div>

I see. I’ve always had trouble when I tried using BenchmarkTools in a local scope. Not sure why…

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [July 18, 2020, 5:06pm UTC](https://discourse.julialang.org/t/unintuitive-behaviour-with-benchmarktools/43291/10 "2020-07-18T17:06:17Z")

</div>

@benchmark calls `eval`. Arguments to `eval` are evaluated in global scope.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [July 18, 2020, 5:06pm UTC](https://discourse.julialang.org/t/unintuitive-behaviour-with-benchmarktools/43291/11 "2020-07-18T17:06:43Z")

</div>

There is the explanation, then. Thanks @oheil.

---

<div class="post-metadata">

**Author:** ![pauljurczak](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pauljurczak/32/921_2.png) [@pauljurczak](https://discourse.julialang.org/u/pauljurczak)\
**Post date:** [July 18, 2020, 5:13pm UTC](https://discourse.julialang.org/t/unintuitive-behaviour-with-benchmarktools/43291/12 "2020-07-18T17:13:25Z")

</div>

I made a pull request: [https://github.com/JuliaCI/BenchmarkTools.jl/pull/172](https://github.com/JuliaCI/BenchmarkTools.jl/pull/172)

Thanks everyone for your help in solving this puzzle.

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [July 18, 2020, 5:39pm UTC](https://discourse.julialang.org/t/unintuitive-behaviour-with-benchmarktools/43291/13 "2020-07-18T17:39:03Z")

</div>

> [@oheil](#):
>
> @benchmark calls `eval` . Arguments to `eval` are evaluated in global scope.

May be I was not exact enough. Looking at the code ( `@edit @benchmark` ) it is `Core.eval(m::Module, expr)` which is called. But in this case with `let...end` block, `m::Module` scope is again `Global`. 🙂  
Anyways, the pull request is a good idea and the experts will probably make it correct.