Do Channels work with multithreaded tasks in 1.3?

Specific Domains Julia at Scale
1

This is perhaps an overly simple question, but I haven’t been able to find a straightfoward answer for it. Now that Julia 1.3 supports spawning parallel tasks across multiple threads, can we do work stealing via those parallel tasks reading to/from shared (thread-safe) channels?

The dev docs currently say no, but the relevant page hasn’t been updated in while. If this is possible now, is it just a matter of replacing @async with @spawn when launching the tasks?

const jobs = Channel{Int}(32);
const results = Channel{Tuple{Int,Float64}}(32);

function do_work()
    for job_id in jobs
        result = sum(rand(i) for i in 1:1_000_000_000)
        put!(results, (job_id, result))
    end
end

function make_jobs(n)
    for i in 1:n
        put!(jobs, i)
    end
end

n = 12

@async make_jobs(n) # feed the jobs channel with "n" jobs

for i in 1:4 # start 4 tasks to process requests in parallel
    @spawn do_work() # not @async
end

while n > 0 # print out results
    job_id, result = take!(results)
    println("Job $job_id returned $result")
    global n = n - 1
end
1 Like
2

Channels are a fine way for the multi-threaded Tasks in v1.3+ to communicate; so please file an issue for the obsolete documentation. I’ve had good experience with unbuffered channels. (I’ve seen possible signs of resource leaks with buffered ones, but those might be programming errors on my part.) Note that you probably want (need?) to close the channels in the above example.

Remember that multi-threading is still “experimental”, but we really want the logic to be solid for v1.4, so please do experiment and report anomalies.

2 Likes