# \--track-allocation and @threads

**URL:** <https://discourse.julialang.org/t/track-allocation-and-threads/84729>\
**Category:** General Usage\
**Tags:** question, multithreading\
**Created:** [July 25, 2022, 4:06am UTC](https://discourse.julialang.org/t/track-allocation-and-threads/84729 "2022-07-25T04:06:37Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![jwake](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jwake/32/36661_2.png) [@jwake](https://discourse.julialang.org/u/jwake)\
**Post date:** [July 25, 2022, 4:06am UTC](https://discourse.julialang.org/t/track-allocation-and-threads/84729/1 "2022-07-25T04:06:37Z")

</div>

Essentially, I call a function within nested loops. In serial, no heap allocations occur within the function. When run on multiple threads via `@threads`, allocations are reported within the function. Because the threaded case runs much slower, I suspect this is [not only a reporting issue](https://discourse.julialang.org/t/can-the-output-of-track-allocation-be-trusted-with-multi-threading-enabled/78744/2).

I know there are some related issues to this elsewhere, but I haven’t really seen anything that would immediately solve this problem. In particular, it would be good to know whether this is something subtle in the code `@threads` is operating on or if this is a more fundamental problem with `@threads` and I need to switch to some other threading construct. [I have tried `@batch`, which has the same issue.](https://discourse.julialang.org/t/overhead-of-threads-threads/53964)

The following example may fail to be ‘minimal’, but it is complete and demonstrates the issue.

```julia
  1 #!/usr/bin/env julia
  2 
  3 
  4 using Profile
  5 
  6 using .Threads
  7 
  8 using StaticArrays
  9 
 10 using BenchmarkTools
 11 
 12 
 13 function nonsense_work(v::SVector{3,F}, i::Int) where {F}
 14 return v.^i ./ SVector{3,F}(1, 2, 3)
 15 end
 16 
 17 
 18 function body()
 19 m, n = 100, 200
 20 some_vectors = [SVector(i, i-1, i+1) for i in 1:n]
 21 result = Array{SVector{3,Float64},2}(undef, m, n)
 22 @threads for i in 1:m
 23 for (j, v) in enumerate(some_vectors)
 24 result[i,j] = nonsense_work(v, i)
 25 end
 26 end
 27 end
 28 
 29 
 30 function main()
 31 body()
 32 Profile.clear_malloc_data()
 33 body()
 34 @time body()
 35 @btime body()
 36 end
 37 
 38 
 39 main()
 40 
 41 

```

```julia
 $ JULIA_NUM_THREADS=1
 $ julia --track-allocation=user threadallocmwe.jl 
  0.009075 seconds (9 allocations: 474.188 KiB)
  8.823 ms (9 allocations: 474.19 KiB)
 $ JULIA_NUM_THREADS=4
 $ julia --track-allocation=user threadallocmwe.jl 
  0.040592 seconds (25 allocations: 475.547 KiB)
  29.747 ms (24 allocations: 475.52 KiB)

```

In the first case:

```julia
 13 - function nonsense_work(v::SVector{3,F}, i::Int) where {F}
 14 0 return v.^i ./ SVector{3,F}(1, 2, 3)
 15 - end
 16 -
 17 -
 18 - function body()
 19 - m, n = 100, 200
 20 0 some_vectors = [SVector(i, i-1, i+1) for i in 1:n]
 21 534809120 result = Array{SVector{3,Float64},2}(undef, m, n)
 22 53472 @threads for i in 1:m
 23 - for (j, v) in enumerate(some_vectors)
 24 - result[i,j] = nonsense_work(v, i)
 25 - end
 26 - end
 27 - end

```

and in the second:

```julia
 13 - function nonsense_work(v::SVector{3,F}, i::Int) where {F}
 14 15584 return v.^i ./ SVector{3,F}(1, 2, 3)
 15 - end
 16 -
 17 -
 18 - function body()
 19 - m, n = 100, 200
 20 0 some_vectors = [SVector(i, i-1, i+1) for i in 1:n]
 21 127701280 result = Array{SVector{3,Float64},2}(undef, m, n)
 22 12768 @threads for i in 1:m
 23 - for (j, v) in enumerate(some_vectors)
 24 - result[i,j] = nonsense_work(v, i)
 25 - end
 26 - end
 27 - end

```

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [July 25, 2022, 7:35am UTC](https://discourse.julialang.org/t/track-allocation-and-threads/84729/2 "2022-07-25T07:35:57Z")

</div>

> [@jwake](#):
>
> When run on multiple threads via `@threads`, allocations are reported within the function.

What `@threads` is effectively doing is creating an anonymous function for the loop body, which captures the used variables from the surrounding scope. You should be able to see that if you do `@macroexpand`. This capturing is tricky and hard for the compiler to optimize, which can often lead to additional allocations happening.

One way around that may be to use a `Ref` of the captured variable instead, which you then access via `r[]` in the loop body, to get the original object back. This explicit boxing is much easier on the compiler.

A different approach would be asking whether the overhead of interthread communication alone is too much to make the parallelization worthwhile - that requires a more careful analysis of your original code though.

---

<div class="post-metadata">

**Author:** ![fft](https://avatars.discourse-cdn.com/v4/letter/f/e8c25b/32.png) [@fft](https://discourse.julialang.org/u/fft)\
**Post date:** [July 25, 2022, 1:31pm UTC](https://discourse.julialang.org/t/track-allocation-and-threads/84729/3 "2022-07-25T13:31:46Z")

</div>

@Sukera

Can you give an example of what you mean by use a Ref to capture variables to avoid allocations. Here is a nonsense example that doesn’t allocate without Threads.@threads, but does with it.

```julia
@views function test!(a, b)
    n = size(a,1)
    Threads.@threads for i = 1:n-1
        a[i] = cos(atan(log10((a[i]^2 + b[i+1]^2)^2)))
    end
end

a = ones(Float32, 100000)
b = ones(Float32, 100000)

@time test!(a,b)

```

---

<div class="post-metadata">

**Author:** ![jwake](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jwake/32/36661_2.png) [@jwake](https://discourse.julialang.org/u/jwake)\
**Post date:** [July 25, 2022, 1:44pm UTC](https://discourse.julialang.org/t/track-allocation-and-threads/84729/4 "2022-07-25T13:44:59Z")

</div>

@Sukera, correct me if I’m wrong, but I think you mean this:

```julia
 18 function body()
 19 m, n = 100, 200
 20 some_vectors = [SVector(i, i-1, i+1) for i in 1:n]
 21 result = Array{SVector{3,Float64},2}(undef, m, n)
 22 res_ref = Ref(result)
 23 @batch for i in 1:m
 24 for (j, v) in enumerate(some_vectors)
 25 res_ref[][i,j] = nonsense_work(v, i)
 26 end
 27 end
 28 end

```

I have yet to try this in my larger code, but this works for the MWE. Thank you so much for providing a simple solution to this problem—I spend a long time trying to figure this out, assuming it was some issue with the functions being called.

```julia
 $ JULIA_NUM_THREADS=1
 $ julia --track-allocation=user threadallocmwe.jl 
  0.009055 seconds (3 allocations: 473.641 KiB)
  9.317 ms (3 allocations: 473.64 KiB)
 $ julia -O3 threadallocmwe.jl 
  0.000553 seconds (3 allocations: 473.641 KiB)
  387.229 μs (3 allocations: 473.64 KiB)
 $ JULIA_NUM_THREADS=4
 $ julia --track-allocation=user threadallocmwe.jl 
  0.056792 seconds (5 allocations: 473.688 KiB)
  54.899 ms (5 allocations: 473.69 KiB)
 $ julia -O3 threadallocmwe.jl 
  0.000384 seconds (5 allocations: 473.688 KiB)
  107.344 μs (5 allocations: 473.69 KiB)

```

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [July 25, 2022, 2:59pm UTC](https://discourse.julialang.org/t/track-allocation-and-threads/84729/5 "2022-07-25T14:59:58Z")

</div>

Yes, that is exactly what I meant.

For more background about why that helps - the `Ref(result)` produces a `RefValue{Matrix{...}}`, so a typed reference. Capturing variables produces a generic, untyped `Box` (more or less a derefable `Any`). Accessing that box means you have to put the result somewhere, while also introducing type instabilities - which can notoriously lead to allocations.

---

<div class="post-metadata">

**Author:** ![jwake](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jwake/32/36661_2.png) [@jwake](https://discourse.julialang.org/u/jwake)\
**Post date:** [July 25, 2022, 3:36pm UTC](https://discourse.julialang.org/t/track-allocation-and-threads/84729/6 "2022-07-25T15:36:25Z")

</div>

I don’t know if I understand why that would be the case. I think what you’re saying is that `typeof(val_ref[])` where `val_ref = Ref(val)` is known a-priori, whereas the type of `val` outside this context could change. Since the type of `val_ref` could change, I am not sure how this leads to any optimization.

Thought this might be a bit outside my abilities, is there a reason `@threads` doesn’t do this automatically for every variable not declared within the loop? In other words, is there a reason

```julia
@threads for i in range
    dothings(a, b, i)
end

```

is not automatically turned into the equivalent of

```julia
a_ref, b_ref = Ref(a), Ref(b)
@threads for i in range
    dothings(a[], b[], i)
end

```

?

PS - The fix you gave doesn’t fix the problem for my use case, but hopefully something like it could work.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [July 25, 2022, 3:41pm UTC](https://discourse.julialang.org/t/track-allocation-and-threads/84729/7 "2022-07-25T15:41:37Z")

</div>

> [@jwake](#):
>
> whereas the type of `val` outside this context could change. Since the type of `val_ref` could change, I am not sure how this leads to any optimization.

No, I’m not talking about changing types of anything. The objects still stay the same type. I’m saying that the type of `val` _inside_ the function created by `@threads` is a `Box` (accessing which will insert dynamic lookups everywhere) in the bad case, and a `RefValue{Matrix{...}}` in the good case. That’s the difference.

> [@jwake](#):
>
> Thought this might be a bit outside my abilities, is there a reason `@threads` doesn’t do this automatically for every variable not declared within the loop?

How should a macro reach outside of its captured expression and know which variables are captured and which are not? Some may even be global variables, which would probably be stable anyway if they’re marked `const`. A macro is just missing a lot of extra information (and it can’t reach outside of its domain to change surrounding code).

> [@jwake](#):
>
> PS - The fix you gave doesn’t fix the problem for my use case, but hopefully something like it could work.

You mean in your real code? That’s unfortunate, but without knowing what that code is, it’s kind of impossible to diagnose…

The canonical issue for this is [https://github.com/JuliaLang/julia/issues/15276](https://github.com/JuliaLang/julia/issues/15276)

---

<div class="post-metadata">

**Author:** ![jwake](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jwake/32/36661_2.png) [@jwake](https://discourse.julialang.org/u/jwake)\
**Post date:** [July 25, 2022, 3:55pm UTC](https://discourse.julialang.org/t/track-allocation-and-threads/84729/8 "2022-07-25T15:55:27Z")

</div>

> [@Sukera](#):
>
> > [@jwake](#):
> >
> > whereas the type of `val` outside this context could change. Since the type of `val_ref` could change, I am not sure how this leads to any optimization.
> 
> No, I’m not talking about changing types of anything. The objects still stay the same type. I’m saying that the type of `val` _inside_ the function created by `@threads` is a `Box` (accessing which will insert dynamic lookups everywhere) in the bad case, and a `RefValue{Matrix{...}}` in the good case. That’s the difference.

I see, thanks.

> [@Sukera](#):
>
> > [@jwake](#):
> >
> > Thought this might be a bit outside my abilities, is there a reason `@threads` doesn’t do this automatically for every variable not declared within the loop?
> 
> How should a macro reach outside of its captured expression and know which variables are captured and which are not? Some may even be global variables, which would probably be stable anyway if they’re marked `const`. A macro is just missing a lot of extra information (and it can’t reach outside of its domain to change surrounding code).

I understand it can’t reach outside of the block it is operating on, but you make a good point that doing this to _everything_ not defined within the block could be a bad idea.

> [@Sukera](#):
>
> > [@jwake](#):
> >
> > PS - The fix you gave doesn’t fix the problem for my use case, but hopefully something like it could work.
> 
> You mean in your real code? That’s unfortunate, but without knowing what that code is, it’s kind of impossible to diagnose…

It wouldn’t be reasonable to ask anyone to look through it even if I could share it.

> [@Sukera](#):
>
> The canonical issue for this is [performance of captured variables in closures · Issue #15276 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/15276)

I saw this issue, but wasn’t able to turn that information into a solution. I also assumed everything before ~2020 may no longer be the case.

Thank you again for your help.

---

<div class="post-metadata">

**Author:** ![jwake](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jwake/32/36661_2.png) [@jwake](https://discourse.julialang.org/u/jwake)\
**Post date:** [July 31, 2022, 1:45am UTC](https://discourse.julialang.org/t/track-allocation-and-threads/84729/9 "2022-07-31T01:45:47Z")

</div>

Earlier I mentioned this didn’t work for my more general code, and I have found what I think is the difference between the MWE I gave and my larger code. In my larger code the data structure passed to `nonsense_work` is complex and created outside of the function, it looks, for example, more like the following (where the result of `--track-allocations=user` have been included):

```julia
        - function nonsense_work(::Type{F}, v::V, i::Int) where {V,F}
       32 return v[].^i ./ SVector{3,F}(1, 2, 3)
        - end
        - 
        - 
        - function body(m::Int, n::Int, some_vectors::Vector{Base.RefValue{SVector{3,Int}}})
   480080 result = Array{SVector{3,Float64},2}(undef, m, n)
   480080 result2 = Array{SVector{3,Float64},2}(undef, m, n)
       16 res_ref = Ref(result)
       48 @threads for i in 1:m
        - for (j, v) in enumerate(some_vectors)
        - res_ref[][i,j] = nonsense_work(Int, v, i)
        - end
        - end
        0 for i in 1:m
        0 for (j, v) in enumerate(some_vectors)
        0 result2[i,j] = nonsense_work(Int, v, i)
        - end
        - end
     7024 @assert all(result .== result2)
        - end
        - 
        - 
        - function main()
        - m, n = 100, 200
        0 some_vectors = [Ref(SVector(i, i-1, i+1)) for i in 1:n]
        0 body(m, n, some_vectors)
        0 Profile.clear_malloc_data()
        0 @time body(m, n, some_vectors)
        - end
        - 
        - 

```

Wildly speculating that the issue has something to do with these added function barriers, `@inline`ing functions seems to largely (although not entirely) solve the problem.

This doesn’t seem like the best solution, but it seems to partially solve my problem. Hopefully this is useful to someone else.

I am entirely open to better solutions to this problem.

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [August 1, 2022, 7:40pm UTC](https://discourse.julialang.org/t/track-allocation-and-threads/84729/10 "2022-08-01T19:40:11Z")

</div>

> [@jwake](#):
>
> Wildly speculating that the issue has something to do with these added function barriers, `@inline`ing functions seems to largely (although not entirely) solve the problem.

Is your current question about why `@inline` seems to help? How much does it help?

Also you refer to a large structure. I suspect we have an issue where this large structure may not be concretely typed and thus we are experiencing significant overhead from dynamic dispatch.

Have we tried running `@code_warntype` to analyze type stability issues and other type inference issues?

---

<div class="post-metadata">

**Author:** ![jwake](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jwake/32/36661_2.png) [@jwake](https://discourse.julialang.org/u/jwake)\
**Post date:** [August 1, 2022, 7:57pm UTC](https://discourse.julialang.org/t/track-allocation-and-threads/84729/11 "2022-08-01T19:57:54Z")

</div>

`@inline` reduces allocations; I am not sure if it has made a huge timing difference.

My goal is for the large structure to be concretely typed; according to `isconcretetype` it is.

There are a couple unions with nothing (yellow) from iterators, but nothing red.

I may post it publicly at some point, but it is currently not on github.