Multithreading

New to Julia
1

I have 3/several functions taking some parameter(s) and returning Bool.
Behind each of them is SQL Query.
Currently I am running them sychronously, one after another, and if one of them returns false, I don’t need to continue to the next one.
I want to execute them asynchronously in different threads and wait/be notified once they finished.
Simple solution would be to wait until they all finish and then evaluate their return values.
More effective solution would be to wait for them, but if some of them returns false, cancel/abandon the rest (one returned false means overall failure).
How could this be done in Julia?
Thank you.

2

Here is a comment (and the whole topic) which you might find useful: How to kill thread? - #8 by pbayer

In my opinion, the task approach is the way to go.

3

You could spawn your queries to several threads and if one fails, you can simply ignore the rest. You can realize that with channels. Let’s assume you have a probability p_fail of a query failing. Then you could do something like:

const p_fail = 0.1

function query(i, ch)
    result = rand()
    sleep(result)
    put!(ch, (i, result > p_fail, result))
end

function parallel_query(n)
    ch = Channel(n)
    foreach(i->(Threads.@spawn query(i, ch)), 1:n)
    for i in 1:n
        res = take!(ch)
        println(res)
        !res[2] && break
    end
end

You break the receiving loop if you get a false. Let’s test it:

julia> parallel_query(4)
(1, true, 0.1929237943634805)
(4, true, 0.4677059775526944)
(2, true, 0.495052606962584)
(3, true, 0.792777535854849)

julia> parallel_query(4)
(3, true, 0.1485337651216685)
(4, true, 0.6231878227831775)
(1, true, 0.9297823992436225)
(2, true, 0.9284756704085468)

julia> parallel_query(4)
(3, false, 0.0160732618267605)

julia>

PS: I do not know if SQL allows parallel access. But this is another question.

2 Likes
4

Thank you. I have implemented my solution using framework as suggested here.

I wanted to ask: how does these Threads functions compare to Task functions @sync @async ? What is faster or more efficient?

5

You can find answers to those questions easily in the Julia Manual or by typing ? and then a command in the Julia REPL:

help?> Threads.@spawn
  Threads.@spawn expr

  Create and run a Task on any available thread. ...

help?> @async
  @async

  Wrap an expression in a Task and add it to the local machine's scheduler
  queue. ...

help?> @sync
  @sync

  Wait until all lexically-enclosed uses of @async, @spawn, @spawnat and
  @distributed are complete. ...

The Julia documentation is quite good, it gives quick answers … No worry!

@async starts a task on the same thread you are on, @spawn starts it on another thread. In your case you could go also with the @threads macro if you want be sure the queries all run in parallel on different threads (This is not guaranteed by @spawn). My example above would become then:

function parallel_query(n)
    ch = Channel(n)
    Threads.@threads for i in 1:n
        @async query(i, ch)
    end
    for i in 1:n
        res = take!(ch)
        println(res)
        !res[2] && break
    end
end

I put an additional @async here to make the @threads for ... loop return immediately after the queries have started on all threads.

2 Likes