# Lightweight tasks, Julia vs Elixir/OTP

**URL:** <https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082>\
**Category:** Performance\
**Tags:** multithreading\
**Created:** [February 24, 2020, 6:30pm UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082 "2020-02-24T18:30:48Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![pbayer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbayer/32/11675_2.png) [@pbayer](https://discourse.julialang.org/u/pbayer)\
**Post date:** [February 24, 2020, 6:30pm UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/1 "2020-02-24T18:30:48Z")

</div>

I wanted to know, whether Julia can cope with Elixir/OTP lightweight tasks and made a comparison for chaining a lot of them following the [example](https://media.pragprog.com/titles/elixir16/code/spawn/chain.exs) from “Programming in Elixir”.

My Julia code is:

```julia
function counter(next::Channel{Int})
    return ch -> put!(next, take!(ch)+1)
end

function create_processes(start::Channel{Int}, n::Int; spawn=false)
    next = start
    for i in 1:n
        next = Channel{Int}(counter(next), spawn=spawn)
    end

    put!(next, 0)

    take!(start)
end

const start = Channel{Int}()

```

I create a counter, returning a closure on a channel. Then I create a chain of n channels with the closure as a task, listening to its channel. I put 0 into the last created channel and take back the accumulated value from the channel I started with. It is a strictly sequential operation over multiple threads.

then

```julia
julia> using BenchmarkTools

julia> @btime create_processes(start, 10, spawn=true)
  24.595 μs (318 allocations: 17.84 KiB)
10

```

… The results were as follows on my 2013 MacBook Pro with 4 cores:

![result](https://global.discourse-cdn.com/julialang/original/3X/e/f/efe4eef3964bd5264cddae42ad99111a19a7b07f.png)

Julia scales better. With fewer tasks it is much faster than Elixir. What I don’t understand that it takes **much** longer if I set `spawn=false`:

```julia
julia> @btime create_processes(start, 1000, spawn=true)
  1.955 ms (33319 allocations: 1.77 MiB)
1000

julia> @btime create_processes(start, 1000)
  36.067 ms (32980 allocations: 1.77 MiB)
1000

```

---

<div class="post-metadata">

**Author:** ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Post date:** [February 24, 2020, 8:40pm UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/2 "2020-02-24T20:40:00Z")

</div>

How sure are you that the Elixir code you posted is a reasonable comparison? It looks like the example returns an interpolated string. I also believe Elixir has a JIT compiler, so you’ll want to ensure that you aren’t timing JIT overhead as well.

---

<div class="post-metadata">

**Author:** ![Pbellive](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbellive/32/3604_2.png) [@Pbellive](https://discourse.julialang.org/u/Pbellive)\
**Post date:** [February 24, 2020, 8:41pm UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/3 "2020-02-24T20:41:06Z")

</div>

Re the spawn = true bit: interesting. I cannot reproduce (julia 1.3.1, linux). Running julia single threaded I get:

```julia
julia> @btime create_processes(start, 1000)
  2.777 ms (32980 allocations: 1.82 MiB)
1000

@btime create_processes(start, 1000, spawn=true)
  2.691 ms (31981 allocations: 1.80 MiB)

```

Then running julia with 10 threads (# of physical cores on my desktop) I get:

```julia
julia> @btime create_processes(start, 1000)
  2.749 ms (32980 allocations: 1.82 MiB)
1000

julia> @btime create_processes(start, 1000, spawn=true)
  2.098 ms (33768 allocations: 1.83 MiB)
1000

```

---

<div class="post-metadata">

**Author:** ![pbayer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbayer/32/11675_2.png) [@pbayer](https://discourse.julialang.org/u/pbayer)\
**Post date:** [February 24, 2020, 9:05pm UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/4 "2020-02-24T21:05:27Z")

</div>

> [@jpsamaroo](#):
>
> How sure are you that the Elixir code you posted is a reasonable comparison? It looks like the example returns an interpolated string. I also believe Elixir has a JIT compiler, so you’ll want to ensure that you aren’t timing JIT overhead as well.

The string is returned only at the end of the chain. Elixir generates byte code and this creates a reasonable startup time, which may be responsible for Elixir taking longer at the first two measurements. Then Elixir/OTP and Julia scale nearly parallel in spawning multiple tasks, sending and receiving message. It was a positive surprise for me that Julia keeps up so well at those scales with OTP.

> [@Pbellive](#):
>
> Re the spawn = true bit: interesting. I cannot reproduce (julia 1.3.1, linux).

Interesting, your results are more what I expected. Interesting though that communication across threads doesn’t show any overhead.

---

<div class="post-metadata">

**Author:** ![Ratingulate](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ratingulate/32/9242_2.png) [@Ratingulate](https://discourse.julialang.org/u/Ratingulate)\
**Post date:** [February 24, 2020, 9:18pm UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/5 "2020-02-24T21:18:27Z")

</div>

Very cool. This would make a nice blog post

---

<div class="post-metadata">

**Author:** ![pbayer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbayer/32/11675_2.png) [@pbayer](https://discourse.julialang.org/u/pbayer)\
**Post date:** [February 25, 2020, 8:38am UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/6 "2020-02-25T08:38:07Z")

</div>

> [@Pbellive](#):
>
> Re the spawn = true bit: interesting. I cannot reproduce (julia 1.3.1, linux).

Now an astounding result – almost for me.

I investigated, why on my machine (MacBook Pro 2013, 4 cores) it takes so much longer with `spawn=false`. My first guess was, that an application gone crazy (like Atom or Safari, sorry guys 😉 ) was keeping my first thread busy. So I shutdown my machine, but still the problem persisted with activity monitor showing nothing strange. So I had to assume that MacOS is keeping part of its kernel within the 1st thread cache. Then I wrote a function allowing me to start my chain on other threads as well:

```julia
function startonthread(id::Int, start::Channel{Int}, n::Int; spawn=false)
    t = Task(nothing)
    @threads for i in 1:nthreads()
        if i == id 
            t = @async create_processes(start, n, spawn=spawn)
        end
    end
    fetch(t)
end

```

and …

```julia
julia> using BenchmarkTools

julia> @btime startonthread(1, start, 1000)
  33.660 ms (33025 allocations: 1.77 MiB)
1000

julia> @btime startonthread(2, start, 1000)
  2.564 ms (33026 allocations: 1.77 MiB)
1000

julia> @btime startonthread(3, start, 1000)
  2.558 ms (33026 allocations: 1.77 MiB)
1000

julia> @btime startonthread(4, start, 1000)
  2.585 ms (33026 allocations: 1.77 MiB)
1000

```

Isn’t that stunning? Does this mean that scientific computing on such machines is much faster on others than the first thread?

edited: Please test this on other Macs as well and also on Windows machines! Thank you @Pbellive for showing that the problem doesn’t show up on Linux.

---

<div class="post-metadata">

**Author:** ![tisztamo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tisztamo/32/16200_2.png) [@tisztamo](https://discourse.julialang.org/u/tisztamo)\
**Post date:** [February 25, 2020, 9:40am UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/7 "2020-02-25T09:40:31Z")

</div>

Interesting, I was not able to reproduce this on my MacBook Pro (13" 2019, OSX 10.14.6 (18G3020), julia 1.3.1):

```julia
julia> @btime startonthread(1, start, 1000)
  3.863 ms (33025 allocations: 1.77 MiB)
1000

julia> @btime startonthread(2, start, 1000)
  2.331 ms (33026 allocations: 1.77 MiB)
1000

julia> @btime startonthread(3, start, 1000)
  2.372 ms (33025 allocations: 1.77 MiB)
1000

julia> @btime startonthread(4, start, 1000)
  2.293 ms (33023 allocations: 1.77 MiB)
1000

```

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [February 25, 2020, 2:54pm UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/8 "2020-02-25T14:54:24Z")

</div>

Thanks for the deep analysis.

> [@pbayer](#):
>
> Come on guys! Please test this on other Macs as well and also on Windows machines! Thank you @Pbellive for showing that the problem doesn’t show up on Linux.

You have no idea how many different systems Julia gets tested and benchmarked on. Please file an issue instead of taking the “shame on you for not testing on my system” attitude.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [February 25, 2020, 2:58pm UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/9 "2020-02-25T14:58:52Z")

</div>

> [@StefanKarpinski](#):
>
> Please file an issue instead of taking the “shame on you for not testing on my system” attitude.

I think this is more call to action to other users to test this too than a complaint, since that behaviour is indeed surprising. I’m currently on my phone, but would be happy test this on my windows machine later today.

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [February 25, 2020, 3:06pm UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/10 "2020-02-25T15:06:49Z")

</div>

This is on a Windows 10 Desktop (Intel(R) Core™ i5-6600 CPU @ 3.30GHz) with Julia 1.3.1:

```julia
julia> @btime startonthread(1, start, 1000)
  3.330 ms (33525 allocations: 3.49 MiB)
1000

julia> @btime startonthread(2, start, 1000)
  2.508 ms (33521 allocations: 3.49 MiB)
1000

```

---

<div class="post-metadata">

**Author:** ![pbayer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbayer/32/11675_2.png) [@pbayer](https://discourse.julialang.org/u/pbayer)\
**Post date:** [February 25, 2020, 3:27pm UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/11 "2020-02-25T15:27:28Z")

</div>

> [@StefanKarpinski](#):
>
> You have no idea how many different systems Julia gets tested and benchmarked on. Please file an issue instead of taking the “shame on you for not testing on my system” attitude.

😰, sorry, this did not come across as I intended and I didn’t even think that Julia was not tested enough. I only wanted – and still want – to see if this is a common problem.

> [@tisztamo](#):
>
> julia\> @btime startonthread(1, start, 1000) 3.863 ms (33025 allocations: 1.77 MiB) 1000

Look, @tisztamo’s result is way better than mine but still it takes `3.863/(2.331+2.372+2.293)*3 = 1.656` times longer on thread 1 than on his other ones. This seems significant enough to me. Maybe @tisztamo, you can repeat with `@btime startonthread(x, start, 100_000)` to see if this is no statistical fluke.

> [@StefanKarpinski](#):
>
> Please file an issue

Are we yet so far? I didn’t think yet. Where should the issue go?

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [February 25, 2020, 3:50pm UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/12 "2020-02-25T15:50:34Z")

</div>

> [@pbayer](#):
>
> Are we yet so far? I didn’t think yet. Where should the issue go?

Discourse is a good place to work out what’s happening but once enough digging is done, file an issue at [Sign in to GitHub · GitHub](https://github.com/JuliaLang/julia/issues/new) to make sure that those who develop and maintain the task/threading system know and can look at it.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [February 25, 2020, 4:01pm UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/13 "2020-02-25T16:01:50Z")

</div>

Win10 Entreprise, i7-8550U @1.8Ghz 1.99Ghz

```julia
@btime startonthread(1, start, 1000)
  4.544 ms (33556 allocations: 3.55 MiB)
@btime startonthread(2, start, 1000)
  2.460 ms (33556 allocations: 3.54 MiB)
@btime startonthread(3, start, 1000)
  2.539 ms (33556 allocations: 3.54 MiB)
@btime startonthread(4, start, 1000)
  2.441 ms (33556 allocations: 3.54 MiB)

```

---

<div class="post-metadata">

**Author:** ![pbayer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbayer/32/11675_2.png) [@pbayer](https://discourse.julialang.org/u/pbayer)\
**Post date:** [February 25, 2020, 7:28pm UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/14 "2020-02-25T19:28:02Z")

</div>

I opened [an issue](https://github.com/JuliaLang/julia/issues/34875) as you suggested. I cross linked this thread and [my repo](https://github.com/pbayer/ThreadingExperiments.jl/tree/master/experiments/chain) there. So I hope, it is enough to work on.

I think, it maybe helpful to see some further results, here or there under the issue.

Interesting stuff though! Thank you very much!

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [February 26, 2020, 7:24am UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/15 "2020-02-26T07:24:41Z")

</div>

```julia
julia> versioninfo()
Julia Version 1.3.1
Commit 2d5741174c (2019-12-30 21:36 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu) # This is a lie - I'm on windows using WSL with a locally compiled julia
  CPU: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 4

```

Took some time before I could test this, but the effect is really pronounced if my laptop is in power saving mode/not connected to a power outlet:

```julia
julia> @btime startonthread(1, start, 1000)
  34.281 ms (33029 allocations: 1.82 MiB)
1000

julia> @btime startonthread(2, start, 1000)
  5.699 ms (33030 allocations: 1.82 MiB)
1000

# to make sure the speedup was not just because of caching
julia> @btime startonthread(1, start, 1000)
  34.130 ms (33029 allocations: 1.82 MiB)
1000

julia> @btime startonthread(3, start, 1000)
  5.626 ms (33030 allocations: 1.82 MiB)
1000

julia> @btime startonthread(4, start, 1000)
  5.693 ms (33030 allocations: 1.82 MiB)
1000

```

In contrast, when in maximum-performance-give-me-all-you-got mode:

```julia
julia> @btime startonthread(1, start, 1000)
  14.043 ms (33029 allocations: 1.82 MiB)
1000

julia> @btime startonthread(2, start, 1000)
  3.437 ms (33030 allocations: 1.82 MiB)
1000

julia> @btime startonthread(3, start, 1000)
  3.993 ms (33030 allocations: 1.82 MiB)
1000

julia> @btime startonthread(4, start, 1000)
  3.451 ms (33030 allocations: 1.82 MiB)
1000

```

* * *

I don’t think this is a julia specific issue, though it could maybe be “solved” using Threadpools.jl [`@bthreads`](https://tro3.github.io/ThreadPools.jl/build/index.html#ThreadPools.@bthreads) to offload work onto anything but thread 1.

---

<div class="post-metadata">

**Author:** ![pbayer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbayer/32/11675_2.png) [@pbayer](https://discourse.julialang.org/u/pbayer)\
**Post date:** [February 26, 2020, 9:26am UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/16 "2020-02-26T09:26:38Z")

</div>

> [@Sukera](#):
>
> I don’t think this is a julia specific issue

I don’t think either. I ask myself how we can exploit it. Is there some free lunch possible by transferring work to less used threads? My first trials show that it is not so easy. Copying data from one thread to another is costly. And for simple mathematical calculations without involving much memory there is no difference between threads. e.g.:

```julia
function machin_series(n::Int)
    qpi = 0
    for i in 1:n
        qpi += (-1)^(i+1)/(2i-1)
    end
    qpi*4
end

function startonthread(id::Int, f::F) where {F<:Function}
    t = Task(nothing)
    @threads for i in 1:nthreads()
        if i == id
            t = @async f()
        end
    end
    fetch(t)
end

```

results:

```julia
julia> @btime startonthread(1, ()->machin_series(1_000_000))
  56.837 ms (43 allocations: 4.20 KiB)
3.1415916535897743

julia> @btime startonthread(2, ()->machin_series(1_000_000))
  56.511 ms (45 allocations: 4.23 KiB)
3.1415916535897743

julia> @btime startonthread(3, ()->machin_series(1_000_000))
  57.028 ms (45 allocations: 4.23 KiB)
3.1415916535897743

```

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [February 26, 2020, 10:04am UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/17 "2020-02-26T10:04:37Z")

</div>

This is an interesting / somewhat strange result.

However, do note that the migration from single threaded concurrency with `@async` to parallel concurrency with `Threads.@spawn` is still a work in progress, especially for the larger package ecosystem. Therefore, a lot of stuff (anything started with `@async`) is going to run on thread 1 and you might just be seeing competition with other tasks doing background work. You could try avoiding some of that by using `julia --startup-file=no do_benchmark.jl`, with all the benchmark code put into `do_benchmark.jl`?

BTW if it helps — I read your “Come on guys!” as a friendly call out to investigate; “hey look at this weird/interesting thing I found” 🙂

---

<div class="post-metadata">

**Author:** ![pbayer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbayer/32/11675_2.png) [@pbayer](https://discourse.julialang.org/u/pbayer)\
**Post date:** [February 26, 2020, 10:22am UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/18 "2020-02-26T10:22:15Z")

</div>

> [@c42f](#):
>
> I read your “Come on guys!” as a friendly call out to investigate; “hey look at this weird/interesting thing I found”

Thank you, it was meant like that. But I remove it, if it is misleading. Again, sorry, I’m no native English speaker.

I tried:

```julia
➜ chain (master) julia --startup-file=no do_benchmark.jl ✱
  32.348 ms (33025 allocations: 1.77 MiB)
  2.679 ms (33026 allocations: 1.77 MiB)
  2.700 ms (33020 allocations: 1.77 MiB)
  2.717 ms (33026 allocations: 1.77 MiB)

```

---

<div class="post-metadata">

**Author:** ![jtackm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jtackm/32/4784_2.png) [@jtackm](https://discourse.julialang.org/u/jtackm)\
**Post date:** [February 26, 2020, 10:38am UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/19 "2020-02-26T10:38:05Z")

</div>

Smaller but still notable effect on an Intel(R) Xeon(R) CPU E7- 4870:

```julia
julia> @btime startonthread(1, start, 1000)
  7.526 ms (33277 allocations: 1.85 MiB)
1000

julia> @btime startonthread(2, start, 1000)
  4.055 ms (33278 allocations: 1.85 MiB)
1000

julia> @btime startonthread(3, start, 1000)
  4.458 ms (33278 allocations: 1.85 MiB)
1000

julia> @btime startonthread(4, start, 1000)
  4.204 ms (33278 allocations: 1.85 MiB)
1000

```

---

<div class="post-metadata">

**Author:** ![tisztamo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tisztamo/32/16200_2.png) [@tisztamo](https://discourse.julialang.org/u/tisztamo)\
**Post date:** [February 26, 2020, 11:38am UTC](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082/20 "2020-02-26T11:38:26Z")

</div>

Sorry for my yesterday mistake about reproducibility! I had a misconception.

The difference is smaller at 100\_000, but still noticeable:

```julia
julia> @btime startonthread(1, start, 1000)
  4.121 ms (33025 allocations: 1.77 MiB)
1000

julia> @btime startonthread(2, start, 1000)
  2.332 ms (33026 allocations: 1.77 MiB)
1000

julia> @btime startonthread(1, start, 100_000)
  579.464 ms (3469044 allocations: 314.20 MiB)
100000

julia> @btime startonthread(2, start, 100_000)
  447.345 ms (3469046 allocations: 314.20 MiB)
100000

```

Not sure if related, but at smaller lengths `top` reports CPU usage up to 340% on the julia process during benchmarking, while at larger lengths it goes lower, at 100\_000 it is around 100%. It seems to be independent from the selected thread and I can see it also when running `startonthread` in a for cycle.

[Next page](https://discourse.julialang.org/t/lightweight-tasks-julia-vs-elixir-otp/35082.md?page=2)