# Multi-threading

**URL:** <https://discourse.julialang.org/t/multi-threading/22708>\
**Category:** General Usage\
**Created:** [April 3, 2019, 5:44pm UTC](https://discourse.julialang.org/t/multi-threading/22708 "2019-04-03T17:44:20Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![krenova](https://avatars.discourse-cdn.com/v4/letter/k/53a042/32.png) [@krenova](https://discourse.julialang.org/u/krenova)\
**Post date:** [April 3, 2019, 5:44pm UTC](https://discourse.julialang.org/t/multi-threading/22708/1 "2019-04-03T17:44:20Z")

</div>

I’m using windows 10 and julia 1.0.3 and testing out the multi-threaded function.

I understand that when generating random numbers we would need a separate seed for each thread but ignoring that for the purpose of a very preliminary test, I created the following code. In the following im trying to assess the overheads of calling a multi-threaded for loop multiple times.

```julia
#set JULIA_NUM_THREADS=4

test = zeros(10000,1000);

@time for i in 1:10000
    Threads.@threads for j in 1:1000
        test[i,j] = rand(1)[1]
    end
end

```

When I ran the above code, I was returned a continuous stream of error messages, their counts likely correlated to the number of times the threaded loop was called. Sample of the errors are:

> Error thrown in threaded loop on thread 0: InexactError(func=:check\_top\_bit, T=Int64, val=-8)  
> Error thrown in threaded loop on thread 2: InexactError(func=:check\_top\_bit, T=Int64, val=-8)  
> Error thrown in threaded loop on thread 1: InexactError(func=:check\_top\_bit, T=Int64, val=-8)  
> Error thrown in threaded loop on thread 0: InexactError(func=:check\_top\_bit, T=Int64, val=-8)  
> Error thrown in threaded loop on thread 3: InexactError(func=:check\_top\_bit, T=Int64, val=-8)  
> Error thrown in threaded loop on thread 1: InexactError(func=:check\_top\_bit, T=Int64, val=-8)  
> Error thrown in threaded loop on thread 2: InexactError(func=:check\_top\_bit, T=Int64, val=-8)  
> Error thrown in threaded loop on thread 3: InexactError(func=:check\_top\_bit, T=Int64, val=-8)

True enough, when i did a sum to check the results, `sum(sum(test,dims=1),dims=2)`, my output was about 40081, way lesser than the expected value of 0.5 \times 10000 \times 1000.

Wondering what mistake I may have made…

Edit1: To think of it I suspect it’s sychronization issues, some threads may have finished and moved on to the next loop?

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [April 3, 2019, 5:59pm UTC](https://discourse.julialang.org/t/multi-threading/22708/2 "2019-04-03T17:59:04Z")

</div>

> [@krenova](#):
>
> I understand that when generating random numbers we would need a separate seed for each thread but ignoring that for the purpose of a very preliminary test

That’s precisely the problem. It’s not just that you need separate seeds — it’s that you need wholly distinct random number generators! Remember that calling `rand` doesn’t just return some value, it’s also _mutating_ the state of your generator to move to the next value. When multiple threads try to mutate this state at the same time, you can rapidly run into inconsistencies.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [April 3, 2019, 6:00pm UTC](https://discourse.julialang.org/t/multi-threading/22708/3 "2019-04-03T18:00:16Z")

</div>

> [@krenova](#):
>
> ignoring that for the purpose of a very preliminary test,

You can’t. Calling the same random number generator from multiple thread is undefined behavior and anything can happen.

Practically, this at least mean that you can corrupt/have corrupted the random number generate (giving you the error) and you’ll not get the correct performance measure. You’ll get very bad performance even without any error.

---

<div class="post-metadata">

**Author:** ![krenova](https://avatars.discourse-cdn.com/v4/letter/k/53a042/32.png) [@krenova](https://discourse.julialang.org/u/krenova)\
**Post date:** [April 3, 2019, 6:01pm UTC](https://discourse.julialang.org/t/multi-threading/22708/4 "2019-04-03T18:01:52Z")

</div>

Ah gosh, I see, this makes sense now. Thanks @mbauman and @yuyichao

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [April 3, 2019, 7:33pm UTC](https://discourse.julialang.org/t/multi-threading/22708/5 "2019-04-03T19:33:40Z")

</div>

I’d take a look at [trandjump](https://github.com/mohamed82008/KissThreading.jl/blob/master/src/KissThreading.jl#L23) of you’d like solutions.

---

<div class="post-metadata">

**Author:** ![zymbuzz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zymbuzz/32/5482_2.png) [@zymbuzz](https://discourse.julialang.org/u/zymbuzz)\
**Post date:** [April 3, 2019, 8:01pm UTC](https://discourse.julialang.org/t/multi-threading/22708/6 "2019-04-03T20:01:20Z")

</div>

Does this also apply to distributed computing across cores? If let’s say instead of Threads.@threads, I use @distributed?

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [April 3, 2019, 8:08pm UTC](https://discourse.julialang.org/t/multi-threading/22708/7 "2019-04-03T20:08:54Z")

</div>

No

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [April 3, 2019, 8:27pm UTC](https://discourse.julialang.org/t/multi-threading/22708/8 "2019-04-03T20:27:16Z")

</div>

> [@krenova](#):
>
> `test[i,j] = rand(1)[1]`

Byt the way, and incidentally, when people write `rand(1)[1]`, what they almost always want is just `rand()`.

`rand(1)[1]` allocates a length-1 vector and then reads out the first number, while `rand()` just returns a random value.

---

<div class="post-metadata">

**Author:** ![Aquaman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aquaman/32/6586_2.png) [@Aquaman](https://discourse.julialang.org/u/Aquaman)\
**Post date:** [April 4, 2019, 2:01am UTC](https://discourse.julialang.org/t/multi-threading/22708/9 "2019-04-04T02:01:51Z")

</div>

Hi @DNF , @mbauman ,@Elrod,

is ` Threads.@threads` equal to the command `parfor` in MATLAB?

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [April 4, 2019, 2:43am UTC](https://discourse.julialang.org/t/multi-threading/22708/10 "2019-04-04T02:43:01Z")

</div>

No afaict MATLAB do not have user controlled thread.

---

<div class="post-metadata">

**Author:** ![krenova](https://avatars.discourse-cdn.com/v4/letter/k/53a042/32.png) [@krenova](https://discourse.julialang.org/u/krenova)\
**Post date:** [April 4, 2019, 2:46am UTC](https://discourse.julialang.org/t/multi-threading/22708/11 "2019-04-04T02:46:13Z")

</div>

Good question.  
@yuyichao, while there maybe no race conditions issue with generating random numbers with `@distributed`, I guess its safe to say that the random numbers also dont run the risk of overlaps since no race condition issues should mean that there is some synchronization involved between workers?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [April 4, 2019, 11:06am UTC](https://discourse.julialang.org/t/multi-threading/22708/12 "2019-04-04T11:06:07Z")

</div>

> [@mbauman](#):
>
> Remember that calling `rand` doesn’t just return some value, it’s also _mutating_ the state of your generator to move to the next value.

I sometimes toy with the idea of calling it `rand!`. Strictly speaking, `rand` is a lie.

---

<div class="post-metadata">

**Author:** ![Aquaman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aquaman/32/6586_2.png) [@Aquaman](https://discourse.julialang.org/u/Aquaman)\
**Post date:** [April 4, 2019, 11:27am UTC](https://discourse.julialang.org/t/multi-threading/22708/13 "2019-04-04T11:27:28Z")

</div>

> # parfor
> 
> Execute `for` -loop iterations in parallel on workers

`in Julia Threads.@threads + for == parfor in Matlab` ???

---

<div class="post-metadata">

**Author:** ![Bernard\_GODARD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bernard_godard/32/4155_2.png) [@Bernard\_GODARD](https://discourse.julialang.org/u/Bernard_GODARD)\
**Post date:** [April 4, 2019, 11:59am UTC](https://discourse.julialang.org/t/multi-threading/22708/14 "2019-04-04T11:59:15Z")

</div>

This works:

```julia
test = zeros(10000,1000);

using Random

const generators = Dict{Int,Any}()
const mutex = Threads.Mutex()

function randomNumber()
   i = Threads.threadid()
   if !haskey(generators,i)
      lock(mutex)
      println("Creating random number generator for thread $i")
      generators[i] = MersenneTwister(rand(RandomDevice(),UInt8))
      unlock(mutex)
   end
   return rand(generators[i])
end

@time for i in 1:10000
    Threads.@threads for j in 1:1000
        test[i,j] = randomNumber()
    end
end

println(sum(test)/length(test))

```

prints

```julia
Creating random number generator for thread 1
Creating random number generator for thread 4
Creating random number generator for thread 2
Creating random number generator for thread 3
  1.071951 seconds (7.12 M allocations: 132.371 MiB, 17.82% gc time)
0.49999995453686974

```

However I don’t know if it can be improved by using a strongly typed dictionary instead of Dict{Int,Any}

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 4, 2019, 12:02pm UTC](https://discourse.julialang.org/t/multi-threading/22708/15 "2019-04-04T12:02:48Z")

</div>

> [@Aquaman](#):
>
> `in Julia Threads.@threads + for == parfor in Matlab` ???

No. It is kind of like `@parallel` with some automatic data transfers. It requires separate processes, so much that using `parfor` on a cluster requires a new license for each core! That is very different than threading.

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [April 4, 2019, 12:06pm UTC](https://discourse.julialang.org/t/multi-threading/22708/16 "2019-04-04T12:06:01Z")

</div>

Workers and threads are not the same. You can google “threads vs processes” or something to get an explanation. Workers are the same as processes.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [April 4, 2019, 12:23pm UTC](https://discourse.julialang.org/t/multi-threading/22708/17 "2019-04-04T12:23:08Z")

</div>

> [@Aquaman](#):
>
> `in Julia Threads.@threads + for == parfor in Matlab` ???

No, `parfor` spawns multiple processes.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [April 4, 2019, 12:55pm UTC](https://discourse.julialang.org/t/multi-threading/22708/18 "2019-04-04T12:55:41Z")

</div>

Wouldn’t it be `using Distributed; @distributed for` these days?

---

<div class="post-metadata">

**Author:** ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Post date:** [April 4, 2019, 3:28pm UTC](https://discourse.julialang.org/t/multi-threading/22708/19 "2019-04-04T15:28:28Z")

</div>

A few changes, namely concretely typing the `generators` dictionary, hoisting generator initialization out of the loop, and putting stuff into a function to avoid the global scope:

```julia
using Random
using BenchmarkTools

const generators = Dict{Int,MersenneTwister}()

function dothread(gen)
    for i in 1:Threads.nthreads()
        if !haskey(generators,i)
            println("Creating random number generator for thread $i")
            gen[i] = MersenneTwister(rand(RandomDevice(),UInt8))
        end
    end
    test = zeros(10000,1000)
    Threads.@threads for j in 1:1000
        tidx = Threads.threadid()
        for i in 1:10000
            test[i,j] = rand(gen[tidx])
        end
    end
    return test
end

@btime dothread($generators)

test = dothread(generators)
println(sum(test)/length(test))

```

```julia
JULIA_NUM_THREADS=4 julia threaded-rand.jl 
Creating random number generator for thread 1
Creating random number generator for thread 2
Creating random number generator for thread 3
Creating random number generator for thread 4
  62.128 ms (32 allocations: 76.30 MiB)
0.49998218701926284

```