# Slowdown when multi-threading is used

**URL:** <https://discourse.julialang.org/t/slowdown-when-multi-threading-is-used/1149>\
**Category:** General Usage\
**Created:** [December 25, 2016, 9:19pm UTC](https://discourse.julialang.org/t/slowdown-when-multi-threading-is-used/1149 "2016-12-25T21:19:28Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [December 25, 2016, 9:19pm UTC](https://discourse.julialang.org/t/slowdown-when-multi-threading-is-used/1149/1 "2016-12-25T21:19:28Z")

</div>

I’m trying to improve performance of [`LombScargle.jl`](https://github.com/giordano/LombScargle.jl) package. Functions used in two methods involves for-loops that can be computed in parallel, thus I thought it would have been a nice idea to use `Threads.@threads`, however I experience a large performance degradation when I do so.

For example, without multi-threading (current `master`):

```julia
julia> using LombScargle, BenchmarkTools

julia> t = collect(linspace(0.01, 10, 100)); s = sin.(t);

julia> @benchmark lombscargle(t, s, fast = false, fit_mean = false)
BenchmarkTools.Trial: 
  memory estimate: 13.36 kb
  allocs estimate: 18
  --------------
  minimum time: 4.187 ms (0.00% GC)
  median time: 4.192 ms (0.00% GC)
  mean time: 4.205 ms (0.00% GC)
  maximum time: 5.395 ms (0.00% GC)
  --------------
  samples: 1189
  evals/sample: 1
  time tolerance: 5.00%
  memory tolerance: 1.00%

```

Add `Threads.@threads` before `for` in line

> <https://github.com/JuliaAstro/LombScargle.jl/blob/master/src/LombScargle.jl#L61>

and repeat the same benchmark with just one thread:

```julia
julia> @benchmark lombscargle(t, s, fast = false, fit_mean = false)
BenchmarkTools.Trial: 
  memory estimate: 15.67 mb
  allocs estimate: 1026273
  --------------
  minimum time: 21.778 ms (0.00% GC)
  median time: 24.138 ms (7.90% GC)
  mean time: 24.079 ms (5.88% GC)
  maximum time: 32.270 ms (7.19% GC)
  --------------
  samples: 208
  evals/sample: 1
  time tolerance: 5.00%
  memory tolerance: 1.00%

```

This is a pretty large slowdown. What’s most interesting is that if I move the code inside the loop to a function on its own and enable multi-threading, there is improvement when comparing to the latter case (this is done in current `multi-threading` branch):

```julia
julia> @benchmark lombscargle(t, s, fast = false, fit_mean = false)
BenchmarkTools.Trial: 
  memory estimate: 91.66 kb
  allocs estimate: 5023
  --------------
  minimum time: 4.504 ms (0.00% GC)
  median time: 4.555 ms (0.00% GC)
  mean time: 4.583 ms (0.17% GC)
  maximum time: 6.786 ms (31.55% GC)
  --------------
  samples: 1091
  evals/sample: 1
  time tolerance: 5.00%
  memory tolerance: 1.00%

```

This has the same performance of non-threaded code, using more threads speed-up calculations.

Why moving body of the for-loop to a function changes the result of benchmark?

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [December 26, 2016, 3:13am UTC](https://discourse.julialang.org/t/slowdown-when-multi-threading-is-used/1149/2 "2016-12-26T03:13:59Z")

</div>

The most likely reason is [performance of captured variables in closures · Issue #15276 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/15276) . See workarounds I’ve mentioned in the comments.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [December 26, 2016, 12:43pm UTC](https://discourse.julialang.org/t/slowdown-when-multi-threading-is-used/1149/3 "2016-12-26T12:43:07Z")

</div>

You might want to see this?

[https://github.com/JuliaLang/julia/issues/17395#issuecomment-241829216](https://github.com/JuliaLang/julia/issues/17395#issuecomment-241829216)

It may be related to [performance of captured variables in closures · Issue #15276 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/15276) . Last time I checked, there were still a few inference bugs around multithreading, so test things out and see how function barriers change the results.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [December 26, 2016, 6:07pm UTC](https://discourse.julialang.org/t/slowdown-when-multi-threading-is-used/1149/4 "2016-12-26T18:07:48Z")

</div>

By moving the whole loop to a function on its own I get the same performance as when moving only the body of the loop to a function. Then I assume I’m hitting bug #15276.

I have no rush, I can wait for this bug to be fixed before adding multi-threading to the package (because the bug will be eventually fixed, right? ;-)).

**Edit** : I stand corrected: the timing of first benchmark in the first post of the thread (which I’m going to update now) was wrong, the time should have been of the order of ~4 ms, that is the same as threaded code with body loop or whole loop moved to a function. Then I think this is exactly bug #15276 and your workaround indeed fixes it.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [December 26, 2016, 6:12pm UTC](https://discourse.julialang.org/t/slowdown-when-multi-threading-is-used/1149/5 "2016-12-26T18:12:29Z")

</div>

Thank you. Not sure I got your suggestion right. I changed

```julia
    @inbounds Threads.@threads for n in eachindex(freqs)

```

to

```julia
    local uidx::Base.OneTo{Int64} = eachindex(freqs)
    @inbounds Threads.@threads for n in uidx

```

but the code is still 24× slower than non-threaded code (i.e., there is no difference with `@inbounds Threads.@threads for n in eachindex(freqs)`.

You said that it may be a problem with `eachindex`, then I tried to replace `eachindex(freqs)` with `1:length(freqs)`, to no avail.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [January 19, 2017, 1:51am UTC](https://discourse.julialang.org/t/slowdown-when-multi-threading-is-used/1149/6 "2017-01-19T01:51:54Z")

</div>

@yuyichao After your post, I implemented the trick (moving the body of the loop to an external function) in `lombscargle` and I had a nice linear scaling with the number of used threads (tested with up to 4 threads). I didn’t change that part of the code after that.

However, after commit (found with git bisect)  
[https://github.com/JuliaLang/julia/commit/673e6f0fdb852c462ea67069c560b5c722b4aa69](https://github.com/JuliaLang/julia/commit/673e6f0fdb852c462ea67069c560b5c722b4aa69)  
I’m experiencing a slowdown of computation of `lombscargle` function when multi-threading is used, even when using your trick At least, after fixing [bug #19876](https://github.com/JuliaLang/julia/issues/19876) the results are correct (thank you for that!), but the degradation when running 4 threads is of ~20%, compared to single-threaded mode. Any hint?

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [January 19, 2017, 12:06pm UTC](https://discourse.julialang.org/t/slowdown-when-multi-threading-is-used/1149/7 "2017-01-19T12:06:47Z")

</div>

Ok, I managed to produce a very minimal working example. The culprit seems to be trigonometric functions. Consider the following function and variable

```julia
function loop!(P)
    @inbounds Threads.@threads for n in eachindex(P)
        P[n] = sin(2.0)
    end
end

P = Vector{Float64}(1000000)

```

In Julia 0.5 with 1 thread:

```julia
julia> @benchmark loop!($P)
BenchmarkTools.Trial: 
  memory estimate: 48.00 bytes
  allocs estimate: 2
  --------------
  minimum time: 11.080 ms (0.00% GC)
  median time: 11.477 ms (0.00% GC)
  mean time: 11.549 ms (0.00% GC)
  maximum time: 13.508 ms (0.00% GC)
  --------------
  samples: 433
  evals/sample: 1
  time tolerance: 5.00%
  memory tolerance: 1.00%

```

With 4 threads:

```julia
julia> @benchmark loop!($P)
BenchmarkTools.Trial: 
  memory estimate: 48.00 bytes
  allocs estimate: 2
  --------------
  minimum time: 2.969 ms (0.00% GC)
  median time: 2.996 ms (0.00% GC)
  mean time: 3.030 ms (0.00% GC)
  maximum time: 5.497 ms (0.00% GC)
  --------------
  samples: 1650
  evals/sample: 1
  time tolerance: 5.00%
  memory tolerance: 1.00%

```

In Julia master (clean build: `git clean -fxd && make`) with 1 thread

```julia
julia> versioninfo()
Julia Version 0.6.0-dev.2181
Commit 71fa0a60f (2017-01-19 08:01 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7-4700MQ CPU @ 2.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, haswell)

julia> @benchmark loop!($P)
BenchmarkTools.Trial: 
  memory estimate: 48.00 bytes
  allocs estimate: 2
  --------------
  minimum time: 11.093 ms (0.00% GC)
  median time: 11.302 ms (0.00% GC)
  mean time: 11.379 ms (0.00% GC)
  maximum time: 12.902 ms (0.00% GC)
  --------------
  samples: 440
  evals/sample: 1
  time tolerance: 5.00%
  memory tolerance: 1.00%

```

with 4 threads

```julia
julia> @benchmark loop!($P)
BenchmarkTools.Trial: 
  memory estimate: 48.00 bytes
  allocs estimate: 2
  --------------
  minimum time: 20.141 ms (0.00% GC)
  median time: 20.149 ms (0.00% GC)
  mean time: 20.163 ms (0.00% GC)
  maximum time: 21.095 ms (0.00% GC)
  --------------
  samples: 248
  evals/sample: 1
  time tolerance: 5.00%
  memory tolerance: 1.00%

```

If I replace `sin` inside `loop!` with other functions (I tried `exp`, `log`, and `gamma`, or also direct assignment to a constant) there is no slowdown, instead there is the expected nice scaling.

Can someone reproduce this issue?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [January 19, 2017, 1:56pm UTC](https://discourse.julialang.org/t/slowdown-when-multi-threading-is-used/1149/8 "2017-01-19T13:56:17Z")

</div>

I think it has to be related to

[https://github.com/JuliaLang/julia/issues/17395](https://github.com/JuliaLang/julia/issues/17395)

then. Or:

> **[Google](https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=int+box+julia)**
>
> Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [January 19, 2017, 2:36pm UTC](https://discourse.julialang.org/t/slowdown-when-multi-threading-is-used/1149/9 "2017-01-19T14:36:06Z")

</div>

@ChrisRackauckas That actually seems to be the answer. Using system libm (replace `sin(2.0)` with `ccall(:sin, Float64, (Float64,), 2.0)`), as suggested by @yuyichao there, does enable scaling.

I wonder why on my system the behavior seems to have changed after [https://github.com/JuliaLang/julia/commit/673e6f0fdb852c462ea67069c560b5c722b4aa69](https://github.com/JuliaLang/julia/commit/673e6f0fdb852c462ea67069c560b5c722b4aa69).