Parallelism? Async != Parallel?

New to Julia
,
1

Hi Guys,
for me the Documentation about Parallel Programming is kinda Confusing in Julia. For me its hard to distinguish the first two Groups of Parallel Programming.

https://docs.julialang.org/en/v1/manual/parallel-computing/

  1. Asynchronous “tasks”, or coroutines
  2. Multi-threading

Is the first Coroutine variant only for IO things, where I have to wait for hardware or how do #1 and #2 differ? Does #1 not run in parallel but is simply “paused” & “startable” and thus only waits for things running in the background and tries to optimize this waiting time by executing other calculations in the meantime?

I would expect that if #1 would run parallel, the following parallel example would be faster than sequential. but this is not the case, can I conclude that #1 is not running parallel? or is the example just bad?

function sum_of_sum_seq(a, b)
    l = sum(a)
    r = sum(b)
    return l + r
end  
function sum_of_sum_parallel(a, b)
    taskR = @async sum(b)
    l = sum(a)
    r = fetch(taskR)
    return l + r
end  
a = rand(Int, 2_000_000_000)
b = rand(Int, 2_000_000_000)  
time_seq = @elapsed result_seq = sum_of_sum_seq(a, b)
time_parallel = @elapsed result_parallel = sum_of_sum_parallel(a, b)
  
$ JULIA_NUM_THREADS=2 julia test.jl 
  Sequential: -2088123306310354629, Time =  66.3472296 ms
  Parallel:   -2088123306310354629, Time = 100.3715951 ms

How do #1 and #2 differ? When should you use #1 or #2? Can both be scaled accross threads?

Ty in advanced

1 Like
2

Coroutines are concurrent, which are not necessarily parallel. For now,

To aid compatibility, code will continue to run within a single thread by default. When tasks are launched using existing primitives ( schedule , @async ), they will run only within the thread that launches them.

that’s why you got no performance gains in the example.

1 Like
3

just to clearify
are couroutines (#1) never parallel or is julia deciding if its worth running the couroutine in parallel ?

if its not parallel in julia, whats the point of using it?

is it just usefull if i would be waiting for IO? like asking the OS for files etc?

4

Coroutines run on the same thread where you start them. You will get performance benefit from @async only when using blocking operations, like IO. You have to use Threads.@spawn instead of @async to utilize more threads.

3 Likes
5

your 2nd function is still on thread 1, try

using .Threads

function sum_of_sum_parallel(a, b)
    taskR = Threads.@spawn sum(b)
    l = sum(a)
    r = fetch(taskR)
    return l + r
end

and you will see parallel performance.

3 Likes
6

ah okay, so to summarize
#1 for blocking operation such as IO
#2/#3 for running the code on diffrent cores/threads

or is there any function which belongs to #1 which runs on diffrent threads?

7

still if you work on other threads (#2) you have to use tasks (#1). As the documentation says:

Multi-threading functionality builds on tasks by allowing them to run simultaneously on more than one thread or CPU core, sharing memory.

Finally distributed computing (#3) is yet another game.

2 Likes
8

i have read that. but my thought problem is that if #1 does not run parallel and #2 is based on #1, how can #2 run parallel?
and by parallel i just mean run on multiple threads

9

A Task is just a task. It can run asynchronously with @async on the same thread or parallel with @spawn on parallel threads. Therefore the task mechanism is for #1 and #2.

Perhaps this is easier to understand if you think of @async and @spawn doing two things:

  1. both create a Task, then
  2. #1 starts it on the thread you are on, #2 starts it on one of all available threads.
7 Likes
10

Rob Pike had a couple of video explaining concurrency vs parallelism.

2 Likes
11

thank you very much! i think i got it!

just to see if i understood it
#1, #2 both creat Tasks, depending on how you shedule/start the task its run on the same thread or on a new one
example:
Threads.spawn wraps the function into a Task and runs it on a diffrent thread (#2)
async also wraps the function into a Task but runs it on the same thread (#1)

#1 is concurrent
#2 is parallel

1 Like
12

sounds good, happy coding!

@spawn accidentally can start it on the same thread since it puts it on one of all available threads. But in my experience it chooses first other threads.

2 Likes