# Scope in multithreading

**URL:** <https://discourse.julialang.org/t/scope-in-multithreading/78276>\
**Category:** General Usage\
**Tags:** multithreading, scope\
**Created:** [March 22, 2022, 2:05pm UTC](https://discourse.julialang.org/t/scope-in-multithreading/78276 "2022-03-22T14:05:01Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![tomtom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomtom/32/5106_2.png) [@tomtom](https://discourse.julialang.org/u/tomtom)\
**Post date:** [March 22, 2022, 2:05pm UTC](https://discourse.julialang.org/t/scope-in-multithreading/78276/1 "2022-03-22T14:05:01Z")

</div>

a rather basic question about scope in multithreading:

```julia
a = 0.0
Threads.@threads for i in 1:10
    b = fun(i)
    a += b
end

```

in the above code:

1. is `a` global and shared by all threads?
2. is `b` local to each thread? i.e. a separate `b` is maintained inside each thread?

thanks

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [March 22, 2022, 2:10pm UTC](https://discourse.julialang.org/t/scope-in-multithreading/78276/2 "2022-03-22T14:10:39Z")

</div>

1. Yes
2. Yes

Those are true for any loop, actually.

(of course, note that `a` will be possibly wrongly updated there)

---

<div class="post-metadata">

**Author:** ![tomtom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomtom/32/5106_2.png) [@tomtom](https://discourse.julialang.org/u/tomtom)\
**Post date:** [March 22, 2022, 2:13pm UTC](https://discourse.julialang.org/t/scope-in-multithreading/78276/3 "2022-03-22T14:13:26Z")

</div>

thanks.

what confuses me is that “scope” seems quite different when moving from simple single thread to multi threads… and I cannot found relevant documentations talking specially for this topic…

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [March 22, 2022, 2:16pm UTC](https://discourse.julialang.org/t/scope-in-multithreading/78276/4 "2022-03-22T14:16:18Z")

</div>

Not really in that case. Each iteration of the loop creates a new scope, and newly defined variables inside the loop iteration are local. The variables of the outer scope of the loop are shared by the loop iterations. This is the same in both cases.

The problem with the multi-threading there is that the iterations of the loop won’t run necessarily in the same sequence, and maybe try to update `a` concurrently, so you can get the wrong result, and need to make the access of `a` safe.

---

<div class="post-metadata">

**Author:** ![tomtom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomtom/32/5106_2.png) [@tomtom](https://discourse.julialang.org/u/tomtom)\
**Post date:** [March 22, 2022, 2:21pm UTC](https://discourse.julialang.org/t/scope-in-multithreading/78276/5 "2022-03-22T14:21:05Z")

</div>

> [@lmiq](#):
>
> the loop won’t run necessarily in the same sequence

it’s expected and not a problem

> [@lmiq](#):
>
> maybe try to update `a` concurrently,

but this one sounds like a problem. what does “concurrently” mean? does it mean some of the `+=` operations may be interrupted and cannot complete?

if it’s the case, please kindly advise how to “safely” access `a`. Thanks.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [March 22, 2022, 2:36pm UTC](https://discourse.julialang.org/t/scope-in-multithreading/78276/6 "2022-03-22T14:36:03Z")

</div>

It means that two threads may try to update the variable _at the same time_, reading from memory the same value, and thus the updates will occur at the same time and the result will be wrong.

This is a simple pattern that does the correct thing:

```julia
julia> ntasks = Threads.nthreads()
       at = zeros(ntasks)
       Threads.@threads for it in 1:ntasks
           for i in it:ntasks:100 # simple splitter
               b = sin(i)
               at[it] += b
           end
       end
       a = sum(at)
-0.1271710136604196

julia> sum(sin(i) for i in 1:100)
-0.12717101366041972

```

The key is to split `a` into independent variables to be updated by each thread independently, and reduce the result at the end.

But you can also use `Floops.jl`, `ThreadX`, `Tullio`, or other less “manual” strategies for multi-threading.

---

<div class="post-metadata">

**Author:** ![tomtom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomtom/32/5106_2.png) [@tomtom](https://discourse.julialang.org/u/tomtom)\
**Post date:** [March 22, 2022, 2:50pm UTC](https://discourse.julialang.org/t/scope-in-multithreading/78276/7 "2022-03-22T14:50:05Z")

</div>

split-and-reduce sounds good.

what about below? is it correct? is it the “formal” method?

```julia
lk = ReentrantLock()
a = 0.0
Threads.@threads for i in 1:10
    b = fun(i)

    lock(lk)
    try
        a += b
    finally
        unlock(lk)
    end
end

```

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [March 22, 2022, 2:55pm UTC](https://discourse.julialang.org/t/scope-in-multithreading/78276/8 "2022-03-22T14:55:10Z")

</div>

It works, but it will be slower (because of the locks).

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [March 22, 2022, 3:06pm UTC](https://discourse.julialang.org/t/scope-in-multithreading/78276/9 "2022-03-22T15:06:22Z")

</div>

> [@tomtom](#):
>
> is it correct?

It depends on what you do outside of the code you write. The following code is data race free (given some other reasonable assumptions like `fun(i)` does not introduce data races, `ok` itself is not a closure, etc.):

```julia
function ok()
    lk = ReentrantLock()
    a = 0.0
    Threads.@threads for i in 1:10
        b = fun(i)

        lock(lk)
        try
            a += b
        finally
            unlock(lk)
        end
    end
end

```

On the other hand, the following function has a data race

```julia
function bad()
    b = nothing # added
    lk = ReentrantLock()
    a = 0.0
    Threads.@threads for i in 1:10
        b = fun(i)

        lock(lk)
        try
            a += b
        finally
            unlock(lk)
        end
    end
end

```

But the `ok` function is still bad. Don’t use lock for reduction. Also, `a` is not type stable. So, `fun` has to be very slow for multi-threading to be beneficial for this code.

FYI, for a more high-level overview, I wrote tutorials like [A quick introduction to data parallelism in Julia](https://juliafolds.github.io/data-parallelism/tutorials/quick-introduction/) and [Efficient and safe approaches to mutation in data parallelism](https://juliafolds.github.io/data-parallelism/tutorials/mutations/) to make parallelism easy. If you like these tutorials, you can look at [How to avoid Box · FLoops](https://juliafolds.github.io/FLoops.jl/dev/howto/avoid-box/) to avoid the problems I described above.

> [@lmiq](#):
>
> `for i in it:ntasks:100 # simple splitter`

It is very important to note that this requires a commutative (and associative) operator. However, parallel reduction only requires associativity and there are many useful non-commutative reductions (e.g., concatenation).

---

<div class="post-metadata">

**Author:** ![tomtom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomtom/32/5106_2.png) [@tomtom](https://discourse.julialang.org/u/tomtom)\
**Post date:** [March 22, 2022, 3:14pm UTC](https://discourse.julialang.org/t/scope-in-multithreading/78276/10 "2022-03-22T15:14:12Z")

</div>

> [@tkf](#):
>
> `a` is not type stable

why? isn’t it should be type stable given that `b` is also `Float64`?