Using @btime and printing a Vector{Vector} yields infinite loop

General Usage
1

The following code produces an infinite loop

using BenchmarkTools
v = Vector{Vector}(undef,1)
@btime println(v)

Is this a bug or intended behaviour?

2

I can’t replicate this, I get (after a lot of output, because it’s not really sensible to call @btime on println):

(...)
Vector[#undef]
Vector[#undef]
Vector[#undef]
Vector[#undef]
  293.800 μs (340 allocations: 69.92 KiB)

Chairmarks is a bit faster here, so I guess BenchmarkTools uses too many samples which are expensive as you’re spamming the console:

using Chairmarks

@b println($v)
3

Ah, right. The loop is not infinite, but will stop after quite a lot of output.

Note that I did not call @btime on println directly, but on a function that (beside other things) also calls println

function foo(v)
   # do stuff ...
   println(v)
end
@btime foo(v)

So what is the recommended way to @btime such functions? Erase all println statements, or use Chairmarks instead?

4

There’s not really a difference between BenchmarkTools and Chairmarks here, except that for your example Chairmarks chooses fewer samples and finishes faster (although that’s generally a goal of Chairmarks).

In my view, if you’re using @btime or @b you’re doing so because you are looking at a performance-critical part of your code. That is inherently incompatible with printing to the terminal, in general I would say but especially in Julia, see:

So yes, I’d say if you want to benchmark the runtime of some piece of code for which you care about performance, don’t let that code print to the terminal.