# Multi-threading changing results

**URL:** <https://discourse.julialang.org/t/multi-threading-changing-results/44671>\
**Category:** New to Julia\
**Created:** [August 10, 2020, 10:16am UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671 "2020-08-10T10:16:32Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![Joao\_Barata](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joao_barata/32/10082_2.png) [@Joao\_Barata](https://discourse.julialang.org/u/Joao_Barata)\
**Post date:** [August 10, 2020, 10:16am UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671/1 "2020-08-10T10:16:32Z")

</div>

Hello,

I’m doing some parallel loops in my code using multi-threading. As I was reviewing my results, I noticed that I kept having different results in a piece of code, although the input data was always the same. So instead of having a parallel version, I moved to a serial version, which confirmed that multi-threading was somehow producing random results.

Here is a MWV of what i’m doing:

```julia
Sim = randn(50,12,10,2)
lev = zeros(10)
for ij = 1:10
    suma = 0.0
    counta = 0.0
    Threads.@threads for i = 1:12
        for j = 1:50
            if Sim[j,i,ij,1] > 0.05 && Sim[j,i,ij,2] > 0.0
                suma = suma + Sim[j,i,ij,1]/Sim[j,i,ij,2]
                counta = counta + 1
            end
        end
    end
    lev[ij] = suma/counta
end

```

After running Sim once and then running the loop multiple times, one gets a different result every time, for some reason. Not so in the serial version. Does anybody have any explanation for this?

---

<div class="post-metadata">

**Author:** ![fbanning](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fbanning/32/14972_2.png) [@fbanning](https://discourse.julialang.org/u/fbanning)\
**Post date:** [August 10, 2020, 10:34am UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671/2 "2020-08-10T10:34:01Z")

</div>

> After running Sim once and then running the loop multiple times, one gets a different result every time

Cannot reproduce this:

```julia
julia> Sim = randn(50,12,10,2)
50×12×10×2 Array{Float64,4}:
LOTS OF NUMBERS

julia> lev = zeros(10)
10-element Array{Float64,1}:
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0

julia> for ij = 1:10
          suma = 0.0
          counta = 0.0
          Threads.@threads for i = 1:12
              for j = 1:50
                  if Sim[j,i,ij,1] > 0.05 && Sim[j,i,ij,2] > 0.0
                      suma = suma + Sim[j,i,ij,1]/Sim[j,i,ij,2]
                                      counta = counta + 1
                  end
              end
          end
          lev[ij] = suma/counta
      end

julia> lev
10-element Array{Float64,1}:
3.602256205117252
6.076811055555966
4.482345464058883
2.6988466255185792
4.717570071596839
4.040129042647047
5.85015237256851
2.3409702687463794
6.324949754983045
3.2099170858509183

julia> for ij = 1:10
          suma = 0.0
          counta = 0.0
          Threads.@threads for i = 1:12
              for j = 1:50
                  if Sim[j,i,ij,1] > 0.05 && Sim[j,i,ij,2] > 0.0
                      suma = suma + Sim[j,i,ij,1]/Sim[j,i,ij,2]
                                      counta = counta + 1
                  end
              end
          end
          lev[ij] = suma/counta
      end

julia> lev
10-element Array{Float64,1}:
3.602256205117252
6.076811055555966
4.482345464058883
2.6988466255185792
4.717570071596839
4.040129042647047
5.85015237256851
2.3409702687463794
6.324949754983045
3.2099170858509183

```

---

<div class="post-metadata">

**Author:** ![LaurentPlagne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/laurentplagne/32/10103_2.png) [@LaurentPlagne](https://discourse.julialang.org/u/LaurentPlagne)\
**Post date:** [August 10, 2020, 10:39am UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671/3 "2020-08-10T10:39:12Z")

</div>

It seems that you are performing a reduction on `suma` then you probably encounter data races.  
You may accumulate on a different variable on each thread (or use atomic).  
[https://docs.julialang.org/en/v1/manual/multi-threading/#Atomic-Operations-1](https://docs.julialang.org/en/v1/manual/multi-threading/#Atomic-Operations-1)

---

<div class="post-metadata">

**Author:** ![Joao\_Barata](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joao_barata/32/10082_2.png) [@Joao\_Barata](https://discourse.julialang.org/u/Joao_Barata)\
**Post date:** [August 10, 2020, 10:44am UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671/4 "2020-08-10T10:44:29Z")

</div>

I can show you what it looks like in mine:

```julia
lev
10-element Array{Float64,1}:
 3.7538904363191454
 1.4806699022971184
 3.380346307030155
 1.5777107005713031
 ⋮
 1.3869407970389027
 0.9888372404478836
 5.494930442090792
 1.5369875031486322

```

second run:

```julia
lev
10-element Array{Float64,1}:
 8.871407349458194
 2.305630998574381
 2.9425926928342405
 2.682961326061693
 ⋮
 1.6146984391226809
 1.5934577478658278
 2.603503063167422
 2.510691292496566

```

---

<div class="post-metadata">

**Author:** ![LaurentPlagne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/laurentplagne/32/10103_2.png) [@LaurentPlagne](https://discourse.julialang.org/u/LaurentPlagne)\
**Post date:** [August 10, 2020, 10:48am UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671/5 "2020-08-10T10:48:56Z")

</div>

Mind try

```julia
suma = Threads.Atomic{Float64}(0); 
counta =Threads.Atomic{Int}(0);

```

?

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [August 10, 2020, 10:53am UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671/6 "2020-08-10T10:53:11Z")

</div>

I’m not an expert (to say the least), but won’t it be much faster to aggregate into an array ~~or tuple~~? I seem to remember that atomics are slow.

Edit: I guess tuples, being immutable, aren’t suited.

---

<div class="post-metadata">

**Author:** ![LaurentPlagne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/laurentplagne/32/10103_2.png) [@LaurentPlagne](https://discourse.julialang.org/u/LaurentPlagne)\
**Post date:** [August 10, 2020, 11:06am UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671/7 "2020-08-10T11:06:05Z")

</div>

You are right and it was my first suggestion. I propose atomics because the fix is easier can confirm data race.

BTW this kind of bugs can be really awful because they may happen rarely. Explicit thread programming is really hard and should be avoided by using parallel patterns when possible.

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [August 10, 2020, 11:14am UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671/8 "2020-08-10T11:14:08Z")

</div>

Moving the threading to the outermost loop removes the data races, and also runs twice as fast (on my laptop):

```julia
function bar!(S, lev)
    Threads.@threads for ij in axes(S, 3)
        suma = 0.0
        counta = 0
        for i in axes(S, 2)
            for j in axes(S, 1)
                if S[j,i,ij,1] > 0.05 && S[j,i,ij,2] > 0
                    suma += S[j,i,ij,1] / S[j,i,ij,2]
                    counta += 1
                end
            end
        end
        lev[ij] = suma / counta
    end
    return lev
end

```

Edit: Sorry! Bad typo in original code!!

The speedup of moving the `@threads` macro call is actually \>12x.

---

<div class="post-metadata">

**Author:** ![Joao\_Barata](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joao_barata/32/10082_2.png) [@Joao\_Barata](https://discourse.julialang.org/u/Joao_Barata)\
**Post date:** [August 10, 2020, 12:47pm UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671/9 "2020-08-10T12:47:24Z")

</div>

Thanks that’s really helpful! That formulation makes much more sense.

---

<div class="post-metadata">

**Author:** ![Joao\_Barata](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joao_barata/32/10082_2.png) [@Joao\_Barata](https://discourse.julialang.org/u/Joao_Barata)\
**Post date:** [August 10, 2020, 12:48pm UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671/10 "2020-08-10T12:48:55Z")

</div>

I’m sorry, I’m not familiar with the terms you’re using. What is the difference between explicit thread programming and parallel patterns?

---

<div class="post-metadata">

**Author:** ![LaurentPlagne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/laurentplagne/32/10103_2.png) [@LaurentPlagne](https://discourse.julialang.org/u/LaurentPlagne)\
**Post date:** [August 10, 2020, 1:35pm UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671/11 "2020-08-10T13:35:16Z")

</div>

My bad: I was rather unclear especially in the Julia context.

**Parallel patterns** are parallel versions of high order functions (functions of function) like `map`,`reduce`,`scan`,`foldl/r`… You can find good example of these in C++ [TBBs](https://github.com/oneapi-src/oneTBB) or even better in the fantastic (but confidential) [Futhark](https://futhark-lang.org/).

- a parallel map operation  
`parallel_map(f, c...) -> collection` with f being a pure function (with no side effect and no mutation on `c...`.)
- a parallel reduce operation  
`parallel_reduce(op, itr) -> result` with `op` being a pure operator and no mutation on `c...`.)

Using such constructs requires the user to identify what he is doing (a map, a reduction, a scan…) and to ensure that the passed function/operator has no side effects.

- The first advantage is that these parallel higher order functions can be carefully designed and tuned and reused in multiple contexts.

- The second advantage is that it forces the user to think about its computation (I am doing a reduction ?).

- Finally, it could help to evolve toward distributed computing or GPU computing more easily.

IMHO, the biggest problem is to ensure that a function has no side effect in Julia. This is why I open a new thread on this topic.

I think that I have seen somewhere (where ?) that some Julia core dev were considering implicit parallelism (a standard map could be executed in parallel). This would be a crazy idea in other languages because you can never be sure that the passed function is not parallel. Since Julia looks exceptionally good at handling nested parallelism it may me sound less crazy… but the problem of possible function side effects remain.

**Explicit thread programming:**  
On the contrary, it looks very easy (too easy) to add a @thread on a parallel loop to see if this brings some speed-up. It often does, and it is also likely that potential data races remain silent for a while because the user didn’t even consider this potential issue. Spawning task can result in very difficult bugs (e.g. random deadlocks).

Because Julia gathers users from different community (math/science and CS) (which is trully wonderful), it will expose more users to these advanced technical difficulties. Hence my concern for strong warning in the documentation and parallel debug tools.

---

<div class="post-metadata">

**Author:** ![Joao\_Barata](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joao_barata/32/10082_2.png) [@Joao\_Barata](https://discourse.julialang.org/u/Joao_Barata)\
**Post date:** [August 10, 2020, 3:05pm UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671/12 "2020-08-10T15:05:03Z")

</div>

Thanks for the extremely detailed and useful answer. It is indeed extremely easy to just put @threads in front of loops for easy speed gains.

Edit: at the expense of possible (nearly) invisible errors, of course.

---

<div class="post-metadata">

**Author:** ![eaubanel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eaubanel/32/13327_2.png) [@eaubanel](https://discourse.julialang.org/u/eaubanel)\
**Post date:** [August 10, 2020, 3:19pm UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671/13 "2020-08-10T15:19:04Z")

</div>

I agree with the need of a strong warning. Parallel programming with language extensions, such as OpenMP or CUDA, makes it clear to the programmer that they’re doing something different, and there’s tons of supporting documentation. In Julia, it’s almost too easy to slap on `@threads` to a loop. More examples of good uses of thread parallelization would be useful. Steering programmers towards implicit parallelism is also a good idea. It would be nice to have a good set of concurrent data structures that programmers could use safely, something like Java’s concurrent collections.

---

<div class="post-metadata">

**Author:** ![Joao\_Barata](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joao_barata/32/10082_2.png) [@Joao\_Barata](https://discourse.julialang.org/u/Joao_Barata)\
**Post date:** [August 12, 2020, 6:54pm UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671/14 "2020-08-12T18:54:55Z")

</div>

Does anybody have a suggestion of a good source to read up on the issue of parallel patterns in Julia, given the post by @LaurentPlagne? The way that this is dealt with the the official Julia documentation is, from a beginner’s point of view, not sufficient both in terms of examples and in terms of the presentation of the tools of multi-threading to a person who wants to begin using that feature (or parallel computing in general). Thanks

---

<div class="post-metadata">

**Author:** ![LaurentPlagne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/laurentplagne/32/10103_2.png) [@LaurentPlagne](https://discourse.julialang.org/u/LaurentPlagne)\
**Post date:** [August 13, 2020, 12:52am UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671/15 "2020-08-13T00:52:55Z")

</div>

Hi Joao, I think that @mbauman lecture on parallel computing in Julia is quite nice ([https://juliaacademy.com](https://juliaacademy.com)).

Please forgive me for the following non solicited piece of advice (I spent many years teaching HPC in an industrial context and often had strong difficulties to warn against early use of parallel programming). They are probably irrelevant to your situation.

I would advice to spend some time to learn about code optimization and computer architecture (in general (and in Julia with the performance tips of the documentation)) before tackling parallel computing. One should have a clear knowledge of what are _memory bound_ and _compute bound_ problems. About the _memory wall_ and the _roof-line model_. Have a basic idea about how many cycles are necessary for +,-,\*,/,sqrt,exp,… operations on common numerical types, for access to contiguous and non contiguous piece of memory. Basic knowledge of cache hierarchy. Some knowledge of the cost of (predictable, and unpredictable) code branches.

If these notions are clear , then one should be able to assess the (absolute) performance of a given algorithm implementation and decide if (s)he should spend some more time to optimize it, look for better algorithm or … enter parallel programming.

Concerning your initial data race problem, the wikipedia article is quite complete (I just learned that Go have a tool to detect data races).

> **[Race condition](https://en.m.wikipedia.org/wiki/Race_condition)**
>
> A race condition or race hazard is the condition of an electronics, software, or other system where the system's substantive behavior is dependent on the sequence or timing of other uncontrollable events. It becomes a bug when one or more of the possible behaviors is undesirable.
> The term race condition was already in use by 1954, for example in David A. Huffman's doctoral thesis "The synthesis of sequential switching circuits".
> Race conditions can occur especially in logic circuits, multithrea...

What is sometime poorly foreseen is that, although these kind of bugs are simple to explain and to spot on small examples, their non-deterministic nature makes them extremely dangerous at scale. If you consider to add a @thread on a loop calling a function, then you should be able _ **to prove** _ that the function is _thread safe_. For example, a simple IO operation, or a tiny counter inside the function can be forgotten and you have built a ticking bomb. Or the function is really thread safe and have locks on these IO or counter and you will spend time to understand why the SpeedUp is not here (much better situation).

Parallel computing is really interesting and useful but it is rather hard and shared memory parallelism is in my opinion quite dangerous for developers (and users) and as such should be handled with great care.

I think that @arch.d.robison have made important contribution to simd introduction in Julia and designed the C++ Intel’s Threading building blocks. Maybe he could add something one safe MT programming in Julia 😉

---

<div class="post-metadata">

**Author:** ![Joao\_Barata](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joao_barata/32/10082_2.png) [@Joao\_Barata](https://discourse.julialang.org/u/Joao_Barata)\
**Post date:** [August 13, 2020, 3:31pm UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671/16 "2020-08-13T15:31:35Z")

</div>

Thanks for the excellent suggestions Laurent. They are extremely useful for a beginner like me.

---

<div class="post-metadata">

**Author:** ![LaurentPlagne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/laurentplagne/32/10103_2.png) [@LaurentPlagne](https://discourse.julialang.org/u/LaurentPlagne)\
**Post date:** [August 20, 2020, 6:38am UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671/17 "2020-08-20T06:38:47Z")

</div>

I just read this thread from @tkf who provides neat packages for Julia parallel patterns.

> [@\[ANN\] FileTrees.jl -- easy everyday parallelism on trees of files](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891/17):
>
> Hi, thanks for your quick intro to Dagger.jl! I’ve been wanting to look into it. I think now I can see that how it is out-of-core friendly. BTW, I think it was a bit of exaggeration when I said “scheduler”. It was just a mechanism to hook different implementation of reduce (sequential, threaded, distributed, unordered variant of them, etc.) into the for loop syntax. So it’s not a scheduler in the sense of e.g., partr. Just as a fair warning(?), I don’t think halve is tested outside of my pack…

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 21, 2020, 3:54am UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671/18 "2020-08-21T03:54:30Z")

</div>

FYI [FLoops.jl](https://github.com/JuliaFolds/FLoops.jl) is trying to provide a “user-friendly” interface to parallel (and sequential) loops. In particular, it can detect the error as in the OP:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/5/2/527f67f5fda3d547e1172f5074ea50f32fbbb40e.png)

(But I should say that I had to fix [two bugs](https://github.com/JuliaFolds/FLoops.jl/releases/tag/v0.1.2) for this example to work. FLoops.jl is a quite new package. It may find your mistakes but it may contain its own bugs 😂.)

A data race-free implementation using FLoops.jl is something like

```julia
function demo()
    Sim = randn(50,12,10,2)
    lev = zeros(10)
    for ij = 1:10
        @floop for i = 1:12
            for j = 1:50
                if Sim[j,i,ij,1] > 0.05 && Sim[j,i,ij,2] > 0.0
                    @reduce(
                        suma = 0.0 + Sim[j,i,ij,1]/Sim[j,i,ij,2],
                        counta = 0.0 + 1,
                    )
                end
            end
        end
        lev[ij] = suma/counta
    end
    Sim
end

```

---

<div class="post-metadata">

**Author:** ![LaurentPlagne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/laurentplagne/32/10103_2.png) [@LaurentPlagne](https://discourse.julialang.org/u/LaurentPlagne)\
**Post date:** [August 21, 2020, 6:51am UTC](https://discourse.julialang.org/t/multi-threading-changing-results/44671/19 "2020-08-21T06:51:28Z")

</div>

I did not knew about this data race diagnostic feature: awesome !!!