# Does @benchmark accurately track PyCall allocations?

**URL:** <https://discourse.julialang.org/t/does-benchmark-accurately-track-pycall-allocations/57375>\
**Category:** Performance\
**Created:** [March 17, 2021, 12:16pm UTC](https://discourse.julialang.org/t/does-benchmark-accurately-track-pycall-allocations/57375 "2021-03-17T12:16:15Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mihalybaci](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mihalybaci/32/13528_2.png) [@mihalybaci](https://discourse.julialang.org/u/mihalybaci)\
**Post date:** [March 17, 2021, 12:16pm UTC](https://discourse.julialang.org/t/does-benchmark-accurately-track-pycall-allocations/57375/1 "2021-03-17T12:16:15Z")

</div>

In the few instances I have tried to compare PyCall+`@benchmark` versus `timteit` within Python, I get runtimes that are very similar (occasionally timing Python from Julia is 1-2 ms _faster_ for some reason). But, does `@benchmark` accurately track memory use and allocations when used with PyCall?

Consider this:

```julia
using BenchmarkTools
using PyCall

np = pyimport("numpy")

a = rand(1_000_000)

@benchmark sin.($a)
@benchmark np.sin($a)

----------------------------------

julia> @benchmark sin.($abc)
BenchmarkTools.Trial: 
  memory estimate: 781.33 KiB
  allocs estimate: 2
  --------------
  minimum time: 672.060 μs (0.00% GC)
  median time: 674.188 μs (0.00% GC)
  mean time: 703.750 μs (1.67% GC)
  maximum time: 3.737 ms (78.29% GC)
  --------------
  samples: 7100
  evals/sample: 1

julia> @benchmark np.sin($abc)
BenchmarkTools.Trial: 
  memory estimate: 782.94 KiB
  allocs estimate: 32
  --------------
  minimum time: 844.317 μs (0.00% GC)
  median time: 883.844 μs (0.00% GC)
  mean time: 911.998 μs (1.41% GC)
  maximum time: 3.835 ms (52.47% GC)
  --------------
  samples: 5477
  evals/sample: 1

```

The memory estimate is very similar, but Julia seems to make fewer allocations. Can I trust these results?

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [March 17, 2021, 12:33pm UTC](https://discourse.julialang.org/t/does-benchmark-accurately-track-pycall-allocations/57375/2 "2021-03-17T12:33:23Z")

</div>

> [@mihalybaci](#):
>
> But, does `@benchmark` accurately track memory use and allocations when used with PyCall?

It will track memory made by the Julia runtime. This will not include eventual allocations made by the Python runtime.

---

<div class="post-metadata">

**Author:** ![mihalybaci](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mihalybaci/32/13528_2.png) [@mihalybaci](https://discourse.julialang.org/u/mihalybaci)\
**Post date:** [March 17, 2021, 12:41pm UTC](https://discourse.julialang.org/t/does-benchmark-accurately-track-pycall-allocations/57375/3 "2021-03-17T12:41:37Z")

</div>

Okay, that’s what pretty much what I had figured. I just wanted to be sure since I don’t have a solid understanding of how `@benchmark` works. Thanks!