# How to implement multi-threading with external in-place mutable variables?

**URL:** <https://discourse.julialang.org/t/how-to-implement-multi-threading-with-external-in-place-mutable-variables/62610>\
**Category:** General Usage\
**Tags:** question\
**Created:** [June 9, 2021, 7:55am UTC](https://discourse.julialang.org/t/how-to-implement-multi-threading-with-external-in-place-mutable-variables/62610 "2021-06-09T07:55:32Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mlanghinrichs](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mlanghinrichs/32/50371_2.png) [@mlanghinrichs](https://discourse.julialang.org/u/mlanghinrichs)\
**Post date:** [June 9, 2021, 7:55am UTC](https://discourse.julialang.org/t/how-to-implement-multi-threading-with-external-in-place-mutable-variables/62610/1 "2021-06-09T07:55:32Z")

</div>

Hey,  
I’m wondering how to implement a method with multi-threading that also allows for mutating external variables in-place. The idea is that each thread/base requires their “own” external variable for data race free mutations.

One solution I came up with is based on [`FLoops.jl`](https://juliafolds.github.io/FLoops.jl/dev/) (down below). The `@floop` macro can run the for-loop in single or multiple threads (`ex=SequentialEx()` or `ex=ThreadedEx()`, respectively). The `@init` macro, together with a `deepcopy()`, should handle data race free mutating variables. Does anyone understand what I want and is the code correct and safe?

```julia
using FLoops

# method to emulate some "runtime"
sleep2 = t -> (b=time(); while b+t > time() end)

function FL_ex3!(y, varexternal, niter; ex=ThreadedEx())
   @floop ex for i in 1:niter
       @init c = deepcopy(varexternal)
       sleep2(0.25)
       c .= (1.0*i, 2.0*i, 3.0*i)
       y[i, :] .= c
   end
   y
end

### sequential and multi-threading give the same output (as a check)
FL_ex3!(zeros(10, 3), zeros(3), 10, ex=SequentialEx()) == FL_ex3!(zeros(10, 3), zeros(3), 10)
# true

### compare performance multi-threaded vs. single-threaded
Threads.nthreads() # 10

using BenchmarkTools
@btime FL_ex3!(zeros(10, 3), zeros(3), 10)
# 250.124 ms (150 allocations: 10.45 KiB)

@btime FL_ex3!(zeros(10, 3), zeros(3), 10, ex=SequentialEx())
# 2.500 s (7 allocations: 960 bytes)

```

Multi-threading is a bit new to me, so sorry if the terminology is a bit off. Would be very glad for input!

---

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [June 9, 2021, 8:46am UTC](https://discourse.julialang.org/t/how-to-implement-multi-threading-with-external-in-place-mutable-variables/62610/2 "2021-06-09T08:46:22Z")

</div>

Just in case, have you seen this tutorial: [Tutorial: Efficient and safe approaches to mutation in data parallelism](https://discourse.julialang.org/t/tutorial-efficient-and-safe-approaches-to-mutation-in-data-parallelism/62105) ?

---

<div class="post-metadata">

**Author:** ![mlanghinrichs](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mlanghinrichs/32/50371_2.png) [@mlanghinrichs](https://discourse.julialang.org/u/mlanghinrichs)\
**Post date:** [June 9, 2021, 10:39am UTC](https://discourse.julialang.org/t/how-to-implement-multi-threading-with-external-in-place-mutable-variables/62610/3 "2021-06-09T10:39:21Z")

</div>

This is a great resource thanks!  
There are two points that are still a bit unclear for me:

1. In the tutorial and docs, the `@init` macro is always used in combination with `@floop` and the `@reduce` macro. I’m wondering if it also works when being used with `@floop` alone as for my function.
2. The parallel pattern I use in my function above is of the kind [“filling pre-allocated output”](https://juliafolds.github.io/data-parallelism/tutorials/mutations/#filling_outputs) as described in the tutorial (output refers to `y` in my case). It says that this pattern may be unsafe for `view`s, `BitArray`s, `SparseMatrixCSC`/`SparseVector`, `Dict` and potentially more. However it indicates that this pattern is data race free for `Array`s. I’m wondering if this is true for _any_ array that contains some custom type such as `y = Vector{SomeType}()`, when this vector is indexed/mutated on the most-outer level (`y[i]`).

Glad for any help!

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [June 11, 2021, 8:11pm UTC](https://discourse.julialang.org/t/how-to-implement-multi-threading-with-external-in-place-mutable-variables/62610/4 "2021-06-11T20:11:36Z")

</div>

> [@mlanghinrichs](#):
>
> The `@init` macro, together with a `deepcopy()` , should handle data race free mutating variables. Does anyone understand what I want and is the code correct and safe?

Yes, `@init` works for this purpose. It looks safe to me.

> [@mlanghinrichs](#):
>
> I’m wondering if this is true for _any_ array that contains some custom type such as `y = Vector{SomeType}()` , when this vector is indexed/mutated on the most-outer level ( `y[i]` ).

It’s data race-free if `SomeType` and its components are immutable or union of immutable types (`isbitstype` or `Base.isbitsunion`). If `SomeType` is mutable, it’s slightly tricky. Consider:

```julia
x1 = Ref(0)
xs = [x1, x1]

```

Now, concurrent update of the memory location _of `x1`_ (e.g., `xs[1][] = 1` in task 1 and `xs[2][] = 2` in task 2) is a data race even though they use different indices for `xs`. On the other hand, updating `xs[i]` _itself_ (e.g., `xs[1] = Ref(1)` in task 1 and `xs[2] = Ref(2)` in task 2) is not a data race.

---

<div class="post-metadata">

**Author:** ![mlanghinrichs](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mlanghinrichs/32/50371_2.png) [@mlanghinrichs](https://discourse.julialang.org/u/mlanghinrichs)\
**Post date:** [June 14, 2021, 11:30am UTC](https://discourse.julialang.org/t/how-to-implement-multi-threading-with-external-in-place-mutable-variables/62610/5 "2021-06-14T11:30:00Z")

</div>

Thanks a lot, extremely helpful!! My `SomeType` is a (immutable) `struct` with `tuple` as fields, so this should be covered by your first case.