# Julia 1.5.1 slows down when running multiple instances in parallel

**URL:** <https://discourse.julialang.org/t/julia-1-5-1-slows-down-when-running-multiple-instances-in-parallel/45934>\
**Category:** Performance\
**Tags:** question\
**Created:** [September 2, 2020, 7:09am UTC](https://discourse.julialang.org/t/julia-1-5-1-slows-down-when-running-multiple-instances-in-parallel/45934 "2020-09-02T07:09:13Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![stakaz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stakaz/32/4740_2.png) [@stakaz](https://discourse.julialang.org/u/stakaz)\
**Post date:** [September 2, 2020, 7:09am UTC](https://discourse.julialang.org/t/julia-1-5-1-slows-down-when-running-multiple-instances-in-parallel/45934/1 "2020-09-02T07:09:13Z")

</div>

Hello, I noticed a weird behavior in julia 1.5.1 performance (e. g. loading packages) compared to julia 1.1.0.

I have a batch script which starts various julia scripts simultaneously on a cluster.  
In a simplified version it looks like

```julia
...
#SBATCH --ntasks 16
...

for ((i=1; i<=16; i++))
do
    srun -n 1 -N 1 --mem=500 /path/to/julia_version-number/bin/julia -O3 -- script.jl $i > $i.log &
done
wait
...

```

As a testing script I use

```julia
# script.jl
using Dates 
println(now())
@time using DataFrames 

```

Now, if I use my old julia 1.1.0 installation I get the following times from each job in dependence of the number of tasks provided

```julia
1 task

2020-09-02T08:38:58.465
  1.940183 seconds (3.31 M allocations: 204.015 MiB, 5.53% gc time)

4 tasks

2020-09-02T08:39:49.865
  1.856393 seconds (3.31 M allocations: 204.015 MiB, 5.14% gc time)

2020-09-02T08:39:49.912
  2.126351 seconds (3.31 M allocations: 204.019 MiB, 5.49% gc time)

2020-09-02T08:39:49.987
  1.908475 seconds (3.31 M allocations: 204.019 MiB, 6.49% gc time)

2020-09-02T08:39:50.014
  2.124697 seconds (3.31 M allocations: 204.015 MiB, 5.55% gc time)

16 tasks 

2020-09-02T08:41:25.748
  2.578776 seconds (3.31 M allocations: 204.021 MiB, 5.83% gc time)

2020-09-02T08:41:25.819
  2.109553 seconds (3.31 M allocations: 204.021 MiB, 4.57% gc time)

2020-09-02T08:41:25.636
  2.213523 seconds (3.31 M allocations: 204.023 MiB, 5.78% gc time)

2020-09-02T08:41:25.422
  2.611317 seconds (3.31 M allocations: 204.021 MiB, 5.21% gc time)
...

```

As you can see the time remains identical for all cases and as expected all jobs start at the same time.

However, this are now times with julia 1.5.1:

```julia
1 task 

2020-09-02T08:32:12.112
  1.209848 seconds (1.43 M allocations: 90.376 MiB)

4 tasks 

2020-09-02T08:30:08.977
  5.184917 seconds (1.43 M allocations: 90.380 MiB)

2020-09-02T08:30:09.131
  5.163921 seconds (1.43 M allocations: 90.382 MiB)

2020-09-02T08:30:09.074
  5.063213 seconds (1.43 M allocations: 90.384 MiB)

2020-09-02T08:30:09.145
  5.118585 seconds (1.43 M allocations: 90.380 MiB)

16 tasks 

2020-09-02T08:33:07.054
 20.093713 seconds (1.43 M allocations: 90.380 MiB)

2020-09-02T08:33:04.616
 20.067529 seconds (1.43 M allocations: 90.382 MiB)

2020-09-02T08:33:06.904
 19.924340 seconds (1.43 M allocations: 90.378 MiB)

2020-09-02T08:33:05.201
 20.263399 seconds (1.43 M allocations: 90.376 MiB)
...

```

The jobs start simultaneously again but the times get much larger as I increase the number of tasks.

Does someone has an idea on what has changed in julia 1.5.1 so that is shows this weird behavior.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [September 2, 2020, 7:14am UTC](https://discourse.julialang.org/t/julia-1-5-1-slows-down-when-running-multiple-instances-in-parallel/45934/2 "2020-09-02T07:14:12Z")

</div>

Someone remind me please - there is a Julia package which deals with race conditions when packages are being compiled. this was referenced recently.

---

<div class="post-metadata">

**Author:** ![stakaz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stakaz/32/4740_2.png) [@stakaz](https://discourse.julialang.org/u/stakaz)\
**Post date:** [September 2, 2020, 7:16am UTC](https://discourse.julialang.org/t/julia-1-5-1-slows-down-when-running-multiple-instances-in-parallel/45934/3 "2020-09-02T07:16:10Z")

</div>

While someone can answer your question, do you suggest that I recompile everything on 1.5.1? Could that help?

---

<div class="post-metadata">

**Author:** ![stakaz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stakaz/32/4740_2.png) [@stakaz](https://discourse.julialang.org/u/stakaz)\
**Post date:** [September 2, 2020, 9:24am UTC](https://discourse.julialang.org/t/julia-1-5-1-slows-down-when-running-multiple-instances-in-parallel/45934/4 "2020-09-02T09:24:28Z")

</div>

Unfortunately it does affect every code, not only the `using SomePackege`, changing the script to

```julia
using Dates 
println(now())
@time A = rand(10_000_000)

```

gives the very same increase in time approximately proportional to the number of tasks.

| number of tasks | time in s |
| --- | --- |
| 40 | 1.663574 |
| 20 | 0.728958 |
| 10 | 0.378481 |
| 1 | 0.030064 |

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [September 2, 2020, 9:41am UTC](https://discourse.julialang.org/t/julia-1-5-1-slows-down-when-running-multiple-instances-in-parallel/45934/5 "2020-09-02T09:41:49Z")

</div>

I think you are refering to this:

> [@ANN: Parallel \`for\` loops in FLoops.jl with composable and extensible fold-based API](https://discourse.julialang.org/t/ann-parallel-for-loops-in-floops-jl-with-composable-and-extensible-fold-based-api/45390):
>
> A while ago I posted [FLoops.jl RFC](https://discourse.julialang.org/t/rfc-ann-floops-jl-fast-generic-for-loops-foldl-for-humans/37876) that demonstrates alternative for loop framework on top of [Transducers.jl](https://github.com/JuliaFolds/Transducers.jl). FLoops.jl lets users write fast for loops over complex data structures. Since it received quite positive reactions [\*1], it made me think that it might be nice to provide a nice syntax sugar that can give people imperative style feeling on top of functional style composable framework. In particular, I’ve been trying to expose the parallel computing part of Transducers.jl via FLoops.jl…

> **[GitHub - JuliaFolds/FLoops.jl: Fast sequential, threaded, and distributed...](https://github.com/JuliaFolds/FLoops.jl)**
>
> Fast sequential, threaded, and distributed for-loops for Julia—fold for humans™ - GitHub - JuliaFolds/FLoops.jl: Fast sequential, threaded, and distributed for-loops for Julia—fold for humans™

---

<div class="post-metadata">

**Author:** ![stakaz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stakaz/32/4740_2.png) [@stakaz](https://discourse.julialang.org/u/stakaz)\
**Post date:** [September 2, 2020, 9:46am UTC](https://discourse.julialang.org/t/julia-1-5-1-slows-down-when-running-multiple-instances-in-parallel/45934/6 "2020-09-02T09:46:45Z")

</div>

I have found the part which is responsible for the slowdown. It is the memory allocation (I accidentally did not included it in the batch script in this post, it is now edited). When I do not set the memory, the times are fine…

Edit: sorry, this does **not** solve the issue because if I do not provide the memory limit, the tasks start one after the other…

---

<div class="post-metadata">

**Author:** ![jipolanco](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jipolanco/32/12129_2.png) [@jipolanco](https://discourse.julialang.org/u/jipolanco)\
**Post date:** [September 2, 2020, 11:20am UTC](https://discourse.julialang.org/t/julia-1-5-1-slows-down-when-running-multiple-instances-in-parallel/45934/7 "2020-09-02T11:20:10Z")

</div>

Related to @johnh’s comment, the issue may be due to a [recent change](https://github.com/JuliaLang/julia/pull/36416) that makes the generation of the precompilation caches atomic, to avoid race conditions.

If that’s the issue, you can take a look at the [docs](https://docs.julialang.org/en/v1/manual/faq/#Computing-cluster) for possible workarounds. The easiest thing to try is to start Julia with `--compiled-modules=no`.

---

<div class="post-metadata">

**Author:** ![stakaz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stakaz/32/4740_2.png) [@stakaz](https://discourse.julialang.org/u/stakaz)\
**Post date:** [September 2, 2020, 11:27am UTC](https://discourse.julialang.org/t/julia-1-5-1-slows-down-when-running-multiple-instances-in-parallel/45934/8 "2020-09-02T11:27:57Z")

</div>

Hello, thanks for the explanation.

I was able to finally solve it by using

```julia
srun -n 1 --exclusive /path/to/julia_version-number/bin/julia -O3 -- script.jl $i > $i.log &

```

and including `#SBATCH --mem-per-cpu=500` in the batch script.

Could you imagine that this memory/CPU management somehow influences the julia 1.5.1 behavior in another way than it was the case in julia 1.1.0?