How to add arrays using multithreading?

General Usage
1

I have a prototype program looking like

function calculate(i)
    # do something, maybe expensive
    return eye(2)*rand()
end

s  = zeros(2, 2)
Thread.@thread for i in 1:100 
    s += calculate(i)
end

I understand this program is not thread-safe since multiple threads may be accessing s at the same time. However, since the document of multithreading is very short. I don’t know how to fix the problem. Anyone offer some help?

2

Perhaps something like

function calculate(i)
    return sin(i)
end

s = zeros(Threads.nthreads())
Threads.@threads for i in 1:10^6
    s[Threads.threadid()] += calculate(i)
end
sum(s)

each thread write to their own part and then you do a serial reduction in the end.

4 Likes
3

Thank you. This definitely works. However, this solution is effectively storing everything calculate returns, which costs a lot of memory in my case. Is there anything like lock and unlock?

4

pretty sure it only stores as many variabels as there are threads, so negligible RAM usage.

2 Likes
5

Good catch! I misunderstood @kristoffer.carlsson. Thank you!

6

Using locks/unlocks:

function calculate(i)
    # do something, maybe expensive
    return eye(2)*rand()
end

s  = zeros(2, 2)
mutex = Threads.Mutex()
Thread.@thread for i in 1:100 
    Threads.lock(mutex)
    s += calculate(i)
    Threads.unlock(mutex)
end
1 Like
7

That would have terrible performance, since all threads are waiting and only one is working at any one time, so it’s a single-thread working disguised in multi-threading. If calculate is taken out of the lock and is really expensive then it will be better, but then the intermediate still needs to be stored, so you might as well avoid waiting altogether and use Kristoffer’s solution.

Edit: if it was just a demo of locks, then I may have over-reacted :slight_smile:

1 Like
8

it was :wink:

1 Like
9

Multithreading is new to me. A simple question: If I do

function calculate(i)
    # do something, maybe expensive
    return eye(2)*rand()
end

s  = zeros(2, 2)
mutex = Threads.Mutex()
Thread.@thread for i in 1:100 
    result = calculate(i)
    Threads.lock(mutex)
    s += result
    Threads.unlock(mutex)
end

Will there be any conflicts for name bindings of result?

10

No, all the variables that are created inside the @threads block will be private to each thread.

11

Also, due to https://github.com/JuliaLang/julia/issues/22581, do I need to avoid matrix mulplication in calculate?

12

Interesting, this is an issue I haven’t personally ran into. I’ve been doing excessive amounts of matrix multiplications inside loops, but probably not exactly in the way that the issue states. I guess just avoid this situation?

13

Maybe. Although I cannot identify the situation confidently and I need to do quite a lot of things inside calculate. Maybe that’s why multithreading is labeled as experimental in the document.

14

Best way to check these things, is to just run the calculation a couple of times and check that the results are always the same.

1 Like
15

There is another problem with this code as rand() is not thread safe. See end of section https://docs.julialang.org/en/latest/manual/parallel-computing/#Multi-Threading-(Experimental)-1 in the Julia Manual.

2 Likes