# @everywhere takes a very long time when using a cluster

**URL:** <https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724>\
**Category:** General Usage\
**Created:** [March 8, 2020, 5:27pm UTC](https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724 "2020-03-08T17:27:44Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [March 8, 2020, 5:27pm UTC](https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724/1 "2020-03-08T17:27:44Z")

</div>

I think I’ve found a bug with `@everywhere` on 1.3 but it’s hard to reproduce. Basically, it takes very long to load the package when using a cluster with multiple nodes (workload manager is Slurm). See the following results, particularly the output of `@time ...`

```julia
julia> using Distributed, ClusterManagers
julia> addprocs(SlurmManager(512), N=16) 
512-element Array{Int64,1}:t of 512
   2
   3
   4
   ⋮
 511
 512
 513

julia> @time @everywhere using mypackage
519.233305 seconds (17.21 M allocations: 878.619 MiB, 0.08% gc time)

```

It takes about 7-8 minutes!

At first I thought it was precompiling, but that’s not the case (I tested this by running `using mypackage` to trigger the compilation locally, and the compiled files are automatically copied to other nodes using Infiniband network). I don’t think precompilation is the problem.

I initially thought this was a cluster configuration problem, but running it on 1.0.5 is **much much quicker**. In a fresh launch of 1.0.5 and doing `addprocs(SlurmManager(512), N=16) ` as above, I get

```julia
julia> @time @everywhere using mypackage
  8.020800 seconds (5.05 M allocations: 245.780 MiB, 1.75% gc time)

```

(this further shows the precompilation is not a problem).

I don’t know what has happened to `@everywhere` between 1.0.5 to 1.3. The problem also dosn’t exist when just doing `addprocs(32)` (i.e. just using the local processors rather than distributed over multiple machines). Seems like a bug that `@everywhere` has regressed so much. Willing to give `ssh` access to a Julia contributor to debug if possible. I can also create a issue on github if this is indeed a bug.

---

<div class="post-metadata">

**Author:** ![anon94023334](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@anon94023334](https://discourse.julialang.org/u/anon94023334)\
**Post date:** [March 9, 2020, 4:17am UTC](https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724/2 "2020-03-09T04:17:28Z")

</div>

Just wondering: what happens if you change the topology to `:master_worker` instead of `:all_to_all`, which is the default?

---

<div class="post-metadata">

**Author:** ![pfarndt](https://avatars.discourse-cdn.com/v4/letter/p/8dc957/32.png) [@pfarndt](https://discourse.julialang.org/u/pfarndt)\
**Post date:** [March 9, 2020, 8:27am UTC](https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724/3 "2020-03-09T08:27:00Z")

</div>

I raised a similar (probably the same) issue here:

> [@Module loading on workers](https://discourse.julialang.org/t/module-loading-on-workers/31814):
>
> Through some trials and errors I realized that modules, which are loaded with using, are also loaded on the already existing workers. Therefore the following code runs without an error: using Distributed addprocs(2) using PyPlot pmap(workers()) do w w, myid(), PyPlot.version end I find this behavior quite inconvenient because in a situation with many workers (to generate some data) connected to one jupyter notebook kernel (to analyze this data) I rather prefer not to load a plotting packa…

I am still not satisfied with the current situation but now know how to avoid the greatest pitfalls, e.g. first load all the packages you need on the master, then add workers, then load packages you definitely need at workers.

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [March 9, 2020, 1:33pm UTC](https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724/5 "2020-03-09T13:33:15Z")

</div>

Is there a way to look at the code change in `@everywhere` from 1.0.5 to 1.3? That might give us a clue.

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [March 12, 2020, 8:17pm UTC](https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724/6 "2020-03-12T20:17:33Z")

</div>

Changing the topology doesn’t do anything. Its still about 8 minutes for `using mypackage`… I just can’t figure out what’s going on. I am not sure what the problem is, but it seems to be with `using` statements inside my package itself.

For example, removing `using DataFrames` reduces the time to about 3 minutes. A huge improvement over the 8 minutes. Again, to reiterate, the packages are already precompiled, so the slowdown must be coming from the ` __init__ ` functions or running the code for the global consts, if there are any.

But I am still not sure why it takes longer on 1.3 than 1.0.5. There is still code loading on both.

---

<div class="post-metadata">

**Author:** ![nantonel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nantonel/32/2889_2.png) [@nantonel](https://discourse.julialang.org/u/nantonel)\
**Post date:** [November 26, 2020, 4:55pm UTC](https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724/7 "2020-11-26T16:55:41Z")

</div>

I’m experiencing the same problem in Julia 1.5.2. Do you know if the issue has been solved? Or any tip for avoiding this? Thanks!

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [November 26, 2020, 7:46pm UTC](https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724/8 "2020-11-26T19:46:20Z")

</div>

I have not seen a solution to this yet, but 1.6 should help with faster pre-compilation speeds.

---

<div class="post-metadata">

**Author:** ![pfarndt](https://avatars.discourse-cdn.com/v4/letter/p/8dc957/32.png) [@pfarndt](https://discourse.julialang.org/u/pfarndt)\
**Post date:** [November 26, 2020, 8:50pm UTC](https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724/9 "2020-11-26T20:50:01Z")

</div>

ok - nowadays I usually take a `ElasticManager` to manage my workers and tweak the `Worker connect command` (see on the linked page) such that all the packages are all loaded before `ClusterManagers.elastic_worker(...)` is called. Depending on how you start your workers this is more gentle on the filesystem. I guess all this loading takes so long because all of a sudden several nodes (in my case 100+) want to load stuff (precompiled or not) from some files.

[https://juliahub.com/docs/ClusterManagers/2gedf/0.4.0/#Using--ElasticManager--dynamically-adding-workers-to-a-cluster](https://juliahub.com/docs/ClusterManagers/2gedf/0.4.0/#Using--ElasticManager--dynamically-adding-workers-to-a-cluster)

---

<div class="post-metadata">

**Author:** ![Lian\_Yunlong](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lian_yunlong/32/14437_2.png) [@Lian\_Yunlong](https://discourse.julialang.org/u/Lian_Yunlong)\
**Post date:** [January 17, 2021, 7:42am UTC](https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724/10 "2021-01-17T07:42:18Z")

</div>

same finding here.

The macro essentially run `remotecall_eval(Main, procs, ex)`, which is again slow after tests. Increasing nworkers makes the excution time (seen from master) to increase linearly. This indicates that it is essentially serialises the job …

---

<div class="post-metadata">

**Author:** ![Lian\_Yunlong](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lian_yunlong/32/14437_2.png) [@Lian\_Yunlong](https://discourse.julialang.org/u/Lian_Yunlong)\
**Post date:** [January 17, 2021, 7:50am UTC](https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724/11 "2021-01-17T07:50:24Z")

</div>

[https://github.com/JuliaLang/julia/issues/39291](https://github.com/JuliaLang/julia/issues/39291)

---

<div class="post-metadata">

**Author:** ![Sixzero](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sixzero/32/28737_2.png) [@Sixzero](https://discourse.julialang.org/u/Sixzero)\
**Post date:** [January 17, 2021, 10:26am UTC](https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724/12 "2021-01-17T10:26:07Z")

</div>

I checked the code inside `remotecall_eval(Main, procs, ex)` and it uses @sync\_add what is not defined for me as of julia 1.5.3.  
Or I don’t understand why I get @sync\_add not defined error messages when I try to use it in a @sync block.

---

<div class="post-metadata">

**Author:** ![Lian\_Yunlong](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lian_yunlong/32/14437_2.png) [@Lian\_Yunlong](https://discourse.julialang.org/u/Lian_Yunlong)\
**Post date:** [January 17, 2021, 10:50am UTC](https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724/13 "2021-01-17T10:50:41Z")

</div>

[https://github.com/JuliaLang/julia/blob/52883caab4333cc5d416125b2bc5ccc45e33496b/base/task.jl#L450](https://github.com/JuliaLang/julia/blob/52883caab4333cc5d416125b2bc5ccc45e33496b/base/task.jl#L450)

---

<div class="post-metadata">

**Author:** ![Lian\_Yunlong](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lian_yunlong/32/14437_2.png) [@Lian\_Yunlong](https://discourse.julialang.org/u/Lian_Yunlong)\
**Post date:** [January 17, 2021, 10:55am UTC](https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724/14 "2021-01-17T10:55:33Z")

</div>

Hi Sixzero,  
I have tested `@everywhere` and `pmap()` on HPC and the results are here:

> **[GitHub - algorithmx/nodeba](https://github.com/algorithmx/nodeba)**
>
> Contribute to algorithmx/nodeba development by creating an account on GitHub.

Basically I tested the parallelized version of `findpi` on HPC and put timestamp in between lines. Then I have pin-pointed the time-consuming part, which is the macro `@everywhere`. Did you have similar situations ? How did you walk around it ?

---

<div class="post-metadata">

**Author:** ![vancleve](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vancleve/32/4183_2.png) [@vancleve](https://discourse.julialang.org/u/vancleve)\
**Post date:** [September 7, 2021, 9:23pm UTC](https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724/15 "2021-09-07T21:23:43Z")

</div>

I have also seen very very very slow `@everywhere using pkgs` on slurm. Its quite bad. 5-10 minutes for 100s of cores even on Julia 1.6.2

---

<div class="post-metadata">

**Author:** ![moble](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/moble/32/23535_2.png) [@moble](https://discourse.julialang.org/u/moble)\
**Post date:** [September 7, 2021, 9:32pm UTC](https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724/16 "2021-09-07T21:32:05Z")

</div>

There’s a workaround described [here](https://discourse.julialang.org/t/why-does-julia-use-thousands-of-cpu-hours-to-compute-1-2/66041/3). Basically, you have to precompile the code that lets processes talk to each other.

---

<div class="post-metadata">

**Author:** ![bgroenks](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bgroenks/32/21784_2.png) [@bgroenks](https://discourse.julialang.org/u/bgroenks)\
**Post date:** [February 11, 2022, 2:20pm UTC](https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724/17 "2022-02-11T14:20:06Z")

</div>

@moble Thanks for the workaround! I am still seeing quite a lot of overhead even after applying your precompile/sysimage approach. The same block of code executed in `@everywhere` vs locally is something like 10x slower. Is this consistent with your experience?

---

<div class="post-metadata">

**Author:** ![moble](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/moble/32/23535_2.png) [@moble](https://discourse.julialang.org/u/moble)\
**Post date:** [February 11, 2022, 2:47pm UTC](https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724/18 "2022-02-11T14:47:32Z")

</div>

For my use case, at least, the scaling was almost perfect once I precompiled. I’m no expert here, but one question jumps out at me: Is that 10x consistent? Like, even after you run whatever computation that is, if you keep the subprocesses up and just ask to re-run it, is it still 10x slower?

If it is consistent, I would guess that you’re doing something that doesn’t actually benefit from parallelization. Is there lots of communication, for example? Is this a too-simple test?

If it’s not consistent, maybe you’re making some subtle mistake on the precompilation.

- Did you precompile the code that you actually need to run? In my gist, [I used `@everywhere 1+2`](https://gist.github.com/moble/709b25556ca229e9e340c6249e68010f#file-precompile-jl-L71), but you could run some short version of the computation you actually need there instead.
- Did you make sure to combine both the main process’s precompilation output _and_ the subprocess precompilation output?
- Did you look at the output from the compilation, to check for any warnings?
- Are you entirely sure you’re actually using the resulting sysimage?