Benchmark is moving target?

General Usage Performance
1

I am doing Advent of Code again this year (not sure how long I’d stay haha) and tried to optimize a very simple function. Surprisingly, @btime is showing different number of allocations on 2nd run. Also, when I redefined my function then the number of allocations dropped to zero. What’s going on? Can I still trust BenchmarkTools?

Source: AdventOfCode2024/day1.jl at main · tk3369/AdventOfCode2024 · GitHub

My REPL:

julia> include("day1.jl");

julia> @btime part2_fast($x, $y)
  139.041 μs (2 allocations: 32 bytes)
21024792

julia> @btime part2_fast($x, $y)
  139.000 μs (1 allocation: 16 bytes)
21024792

julia> part2_fast(x, y) = sum(value * count(==(value), y) for value in x)
part2_fast (generic function with 1 method)

julia> @btime part2_fast($x, $y)
  139.000 μs (0 allocations: 0 bytes)
21024792
2

It seems it has something to do with the generator inside the sum argument. The allocations go away reliably when I switch the sum method:

f(x, y) = sum(v -> v * count(==(v), y), x)
3

you can plot the distrubition of times using @benchmark instead of @btime