# @time and @btime report different values?

**URL:** <https://discourse.julialang.org/t/time-and-btime-report-different-values/80087>\
**Category:** General Usage\
**Created:** [April 26, 2022, 6:37pm UTC](https://discourse.julialang.org/t/time-and-btime-report-different-values/80087 "2022-04-26T18:37:56Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![Seif\_Shebl](https://avatars.discourse-cdn.com/v4/letter/s/eada6e/32.png) [@Seif\_Shebl](https://discourse.julialang.org/u/Seif_Shebl)\
**Post date:** [April 26, 2022, 6:37pm UTC](https://discourse.julialang.org/t/time-and-btime-report-different-values/80087/1 "2022-04-26T18:37:56Z")

</div>

When timing the following function, I noticed that the `@time` macro shows a value, that I assume is correct, whereas `@btime` shows a different value. This only occurs in Julia 1.8.0-DEV and 1.9.0-DEV. Did I make a mistake?

```julia
function integrate(func, a, b, N)
    sum = 0.0
    step = (b - a) / N
    for i = 1:N
        sum += func(a + i*step) * step
    end
    return sum
end

@time integrate(sin, 0, 1, 10^8) # 0.736943 seconds
@btime integrate(sin, 0, 1, 10^8) # 1.115 s (0 allocations: 0 bytes)

```

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [April 26, 2022, 6:55pm UTC](https://discourse.julialang.org/t/time-and-btime-report-different-values/80087/2 "2022-04-26T18:55:48Z")

</div>

Confirmed on 1.9.0-DEV. This seems to be a recent change, on 1.6.6 I see

```julia
using BenchmarkTools

function integrate(func, a, b, N)
    sum = 0.0
    step = (b - a) / N
    for i = 1:N
        sum += func(a + i*step) * step
    end
    return sum
end

@time integrate(sin, 0, 1, 10^8)
@time integrate(sin, 0, 1, 10^8)
@btime integrate(sin, 0, 1, 10^8)

```

yielding

```julia
  1.080566 seconds
  1.082476 seconds
  1.080 s (0 allocations: 0 bytes)

```

Edit: in case platform is relevant: I’m on Windows 10, Intel(R) Core™ i7

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [April 26, 2022, 7:19pm UTC](https://discourse.julialang.org/t/time-and-btime-report-different-values/80087/3 "2022-04-26T19:19:00Z")

</div>

Funny, with Julia 1.9 I get

```julia
julia> @time integrate(sin, 0, 1, 10^8)
  0.775296 seconds
0.45969769833927326

julia> @btime integrate(sin, 0, 1, 10^8)
  316.236 ms (0 allocations: 0 bytes)
0.45969769833927326

```

Similar `@time`, but `@btime` is way faster for me.

Note that with Julia 1.9 `@btime` can sometimes show larger times than before because of [Make use of Base.donotdelete if available by Keno · Pull Request #275 · JuliaCI/BenchmarkTools.jl · GitHub](https://github.com/JuliaCI/BenchmarkTools.jl/pull/275) (but I’m not sure that’s relevant here).

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [April 26, 2022, 7:50pm UTC](https://discourse.julialang.org/t/time-and-btime-report-different-values/80087/4 "2022-04-26T19:50:10Z")

</div>

It was my understanding that `@time` ran once and `@btime` ran potentially thousands of times and reported the minimum time.

Why would you expect them to show the same value?

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [April 26, 2022, 7:51pm UTC](https://discourse.julialang.org/t/time-and-btime-report-different-values/80087/5 "2022-04-26T19:51:06Z")

</div>

> [@tbeason](#):
>
> It was my understanding that `@time` ran once and `@btime` ran potentially thousands of times and reported the minimum time.

Exactly, but why should `@btime` be slower then?

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [April 26, 2022, 7:58pm UTC](https://discourse.julialang.org/t/time-and-btime-report-different-values/80087/6 "2022-04-26T19:58:35Z")

</div>

First, I cannot reproduce this.

Second, my first guess as to why `@btime` would give larger values is that nothing was interpolated in the call. I always (literally always) interpolate.

```julia
julia> @benchmark integrate($sin, $0, $1, $(10^8)) 
BenchmarkTools.Trial: 9 samples with 1 evaluation.
 Range (min … max): 604.252 ms … 622.429 ms ┊ GC (min … max): 0.00% … 0.00%
 Time (median): 609.033 ms ┊ GC (median): 0.00%
 Time (mean ± σ): 609.482 ms ± 6.008 ms ┊ GC (mean ± σ): 0.00% ± 0.00%

  █▁ ▁ ▁▁ ▁ ▁ ▁
  ██▁█▁▁▁▁▁▁▁▁▁▁▁▁██▁▁▁▁▁▁▁▁█▁▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█ ▁
  604 ms Histogram: frequency by time 622 ms <

 Memory estimate: 0 bytes, allocs estimate: 0.

```

That is on 1.8-beta3 but I get identical timing on 1.7.2

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [April 26, 2022, 8:00pm UTC](https://discourse.julialang.org/t/time-and-btime-report-different-values/80087/7 "2022-04-26T20:00:31Z")

</div>

> [@tbeason](#):
>
> Second, my first guess as to why `@btime` would give larger values is that nothing was interpolated in the call. I always (literally always) interpolate.

Tried interpolation, to no avail. I suspect a platform problem (Linux vs. Windows?)

And interestingly enough it is `@time` which decreases for me from 1.6.6 to 1.9.0-DEV.

Edit: `@benchmark integrate($sin, $0, $1, $10^8)` shows

```julia
BenchmarkTools.Trial: 6 samples with 1 evaluation.
 Range (min … max): 946.139 ms … 947.561 ms ┊ GC (min … max): 0.00% … 0.00%
 Time (median): 946.902 ms ┊ GC (median): 0.00%        
 Time (mean ± σ): 946.887 ms ± 516.072 μs ┊ GC (mean ± σ): 0.00% ± 0.00%

  █ █ █ █ █ █  
  █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁█ ▁
  946 ms Histogram: frequency by time 948 ms <

 Memory estimate: 0 bytes, allocs estimate: 0.

```

---

<div class="post-metadata">

**Author:** ![Seif\_Shebl](https://avatars.discourse-cdn.com/v4/letter/s/eada6e/32.png) [@Seif\_Shebl](https://discourse.julialang.org/u/Seif_Shebl)\
**Post date:** [April 26, 2022, 8:39pm UTC](https://discourse.julialang.org/t/time-and-btime-report-different-values/80087/8 "2022-04-26T20:39:01Z")

</div>

Your understanding is correct, by saying different I mean significantly different, what I expect is that `@time` for a second run would show approximately the same value as `@btime` does. Also, interpolation has no effect here as @goerch reported. It may be related to the issue linked to above by @giordano. Furthermore, I’m on Windows 10 also as @goerch if it makes a difference.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [April 26, 2022, 9:04pm UTC](https://discourse.julialang.org/t/time-and-btime-report-different-values/80087/9 "2022-04-26T21:04:44Z")

</div>

> [@Seif\_Shebl](#):
>
> It may be related to the issue linked to above by @giordano.

Reading the discussion of the PR, @kristoffer.carlsson writes

> [@](#):
>
> A better intention seems to me to have BenchmarkTools be as useful as possible. And when it optimizes away the whole benchmark, it is not being useful. The hoops you have to jump through now with interpolating Ref etc just to get a useful answer is annoying. This seems like an improvement.

So this would mean `@btime` is more conservative than `@time` from now on? Interesting change.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [April 27, 2022, 12:53pm UTC](https://discourse.julialang.org/t/time-and-btime-report-different-values/80087/10 "2022-04-27T12:53:35Z")

</div>

I do not think that PR is relevant here because this function does return its value. That quote is in reference to benchmarks that have no side effects so the entire program is just compiled away.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [April 27, 2022, 1:54pm UTC](https://discourse.julialang.org/t/time-and-btime-report-different-values/80087/11 "2022-04-27T13:54:30Z")

</div>

> [@tbeason](#):
>
> I do not think that PR is relevant here because this function does return its value.

@Seif_Shebl, does this deserve an issue then?

---

<div class="post-metadata">

**Author:** ![ppalmes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ppalmes/32/6838_2.png) [@ppalmes](https://discourse.julialang.org/u/ppalmes)\
**Post date:** [April 29, 2022, 10:20am UTC](https://discourse.julialang.org/t/time-and-btime-report-different-values/80087/12 "2022-04-29T10:20:13Z")

</div>

@benchmark gives you mean performance with standard deviation while @time or @btime is just single point performance. also, if you want to measure performance, you need average value and its standard deviation. a single point measurement is a bias measurement. you only use @time to get a feel of the performance but it is not conclusive. the @benchmark is more conclusive because it runs several times and get the average and standard deviation.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [April 29, 2022, 10:36am UTC](https://discourse.julialang.org/t/time-and-btime-report-different-values/80087/13 "2022-04-29T10:36:17Z")

</div>

From the [documentation](https://juliaci.github.io/BenchmarkTools.jl/dev/manual/#Defining-and-executing-benchmarks):

> `@btime` prints the minimum time and memory allocation before returning the value of the expression

---

<div class="post-metadata">

**Author:** ![ppalmes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ppalmes/32/6838_2.png) [@ppalmes](https://discourse.julialang.org/u/ppalmes)\
**Post date:** [April 29, 2022, 1:52pm UTC](https://discourse.julialang.org/t/time-and-btime-report-different-values/80087/14 "2022-04-29T13:52:11Z")

</div>

ok, stand to be corrected. @btime or @time should not be used then if you want to measure unbiased performance. use @benchmark to get average with standard deviation performance. i corrected the @btime explanation above and instead replaced it with @benchmark which i’m originally referring to for unbiased performance measure.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [April 30, 2022, 12:07pm UTC](https://discourse.julialang.org/t/time-and-btime-report-different-values/80087/15 "2022-04-30T12:07:15Z")

</div>

> [@goerch](#):
>
> @Seif_Shebl, does this deserve an issue then?

I searched the issues and found [#39760](https://github.com/JuliaLang/julia/issues/39760). It seems the behavior is intentional. Testing [this proposal](https://github.com/JuliaLang/julia/issues/39760#issuecomment-783738806) with

```julia
using BenchmarkTools

function integrate(func, a, b, N)
    sum = 0.0
    step = (b - a) / N
    for i = 1:N
        sum += func(a + i*step) * step
    end
    return sum
end

@time @eval integrate($sin, 0, 1, 10^8)
@time @eval integrate($sin, 0, 1, 10^8)
@btime integrate($sin, 0, 1, 10^8)

```

I see

```julia
  0.996603 seconds (229.57 k allocations: 14.926 MiB, 8.53% compilation time)
  0.915735 seconds (56 allocations: 2.797 KiB)
  1.067 s (0 allocations: 0 bytes)

```

---

<div class="post-metadata">

**Author:** ![Seif\_Shebl](https://avatars.discourse-cdn.com/v4/letter/s/eada6e/32.png) [@Seif\_Shebl](https://discourse.julialang.org/u/Seif_Shebl)\
**Post date:** [April 30, 2022, 7:19pm UTC](https://discourse.julialang.org/t/time-and-btime-report-different-values/80087/16 "2022-04-30T19:19:46Z")

</div>

Now I’m totally confused, which one of the following represents the most correct timing and no. of allocations for my function? In Jeff’s own words, " most people want `@time f()` to mean **how long does `f` take to run** , not the latency around it". What is the way to go? Especially with the flood of questions by new users stating incorrectly that lang x is faster than Julia doing y job, and it turns out they also measure the compile time of their code.

```julia
@time integrate(sin, 0, 1, 10^8) # 0.742815 seconds (No alloc. count?)
@time integrate(sin, 0, 1, 10^8) # 0.742608 seconds (No alloc. count?)
@time @eval integrate(sin, 0, 1, 10^8) # 1.005283 seconds (303 allocations: 15.688 KiB, 1.33% compilation time)
@time @eval integrate(sin, 0, 1, 10^8) # 0.992475 seconds (53 allocations: 2.641 KiB)
@btime integrate(sin, 0, 1, 10^8) # 1.127 s (0 allocations: 0 bytes)
@benchmark integrate(sin, 0, 1, 10^8)
BenchmarkTools.Trial: 5 samples with 1 evaluation.
 Range (min … max): 1.114 s … 1.116 s ┊ GC (min … max): 0.00% … 0.00%
 Time (median): 1.114 s ┊ GC (median): 0.00%
 Time (mean ± σ): 1.114 s ± 679.540 μs ┊ GC (mean ± σ): 0.00% ± 0.00%

  █ ███ █
  █▁▁███▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█ ▁
  1.11 s Histogram: frequency by time 1.12 s <

 Memory estimate: 0 bytes, allocs estimate: 0.

```

---

<div class="post-metadata">

**Author:** ![Ahmed\_Salih](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ahmed_salih/32/206579_2.png) [@Ahmed\_Salih](https://discourse.julialang.org/u/Ahmed_Salih)\
**Post date:** [May 1, 2022, 1:33am UTC](https://discourse.julialang.org/t/time-and-btime-report-different-values/80087/17 "2022-05-01T01:33:12Z")

</div>

@time does what it says, it times how long a line took to run. This includes as far as I know anything from start up time, first compilation of code etc. Therefore one should be careful using this, but it is indeed nice to have in an interactive terminal.

@btime from BenchmarkTools gives you the measurement of how the function performs, without startup or first compilation of code. Use this when you want to know more precisely how your function performs.

The allocations can be due to two things; not interpolating variables i.e. using `$` infront and by writing variable inputs directly, i.e. “10^8”, instead of `value = 10^8`, and then inputting that as `$value`. There is another way which allows for direct input of numerical numbers, but I forgot how.

Kind regards

---

<div class="post-metadata">

**Author:** ![Seif\_Shebl](https://avatars.discourse-cdn.com/v4/letter/s/eada6e/32.png) [@Seif\_Shebl](https://discourse.julialang.org/u/Seif_Shebl)\
**Post date:** [May 1, 2022, 5:55am UTC](https://discourse.julialang.org/t/time-and-btime-report-different-values/80087/18 "2022-05-01T05:55:23Z")

</div>

This was my understanding as well. Until recently, I believe 1.7.0+ or so, some PRs were merged that modified the original functionality. I didn’t pay careful attention to all of it to see which one was responsible for these changes. As far as I can tell, currently the only method to get the most accurate timing of a function is to manually run `@time` several times and take the average excluding the first trial. All other variants are imprecise as shown by the examples above.