I think I’ve found a bug with @everywhere on 1.3 but it’s hard to reproduce. Basically, it takes very long to load the package when using a cluster with multiple nodes (workload manager is Slurm). See the following results, particularly the output of @time ... julia> using Distributed, ClusterMana…

Just wondering: what happens if you change the topology to :master_worker instead of :all_to_all, which is the default?

I raised a similar (probably the same) issue here: [image] Module loading on workers General Usage Through some trials and errors I realized that modules, which are loaded with using, are also loaded on the already existing workers. Therefore the following code runs witho…

Is there a way to look at the code change in @everywhere from 1.0.5 to 1.3? That might give us a clue.

Changing the topology doesn’t do anything. Its still about 8 minutes for using mypackage… I just can’t figure out what’s going on. I am not sure what the problem is, but it seems to be with using statements inside my package itself. For example, removing using DataFrames reduces the time to about 3…

I’m experiencing the same problem in Julia 1.5.2. Do you know if the issue has been solved? Or any tip for avoiding this? Thanks!

I have not seen a solution to this yet, but 1.6 should help with faster pre-compilation speeds.

ok - nowadays I usually take a ElasticManager to manage my workers and tweak the Worker connect command (see on the linked page) such that all the packages are all loaded before ClusterManagers.elastic_worker(...) is called. Depending on how you start your workers this is more gentle on the filesyst…

same finding here. The macro essentially run remotecall_eval(Main, procs, ex), which is again slow after tests. Increasing nworkers makes the excution time (seen from master) to increase linearly. This indicates that it is essentially serialises the job …

https://github.com/JuliaLang/julia/issues/39291

I checked the code inside remotecall_eval(Main, procs, ex) and it uses @sync_add what is not defined for me as of julia 1.5.3. Or I don’t understand why I get @sync_add not defined error messages when I try to use it in a @sync block.

@everywhere takes a very long time when using a cluster

General Usage

moble September 7, 2021, 9:32pm 16

There’s a workaround described here. Basically, you have to precompile the code that lets processes talk to each other.

1 Like

Topic		Replies	Views
Why does julia use thousands of cpu-hours to compute 1+2? General Usage hpc , distributed	10	3416	August 16, 2021
@everywhere using MyLib fails when procs on remote node General Usage package , parallel , cluster	7	1296	June 29, 2017
"Failed to precompile" error on worker processes General Usage	17	4981	June 15, 2020
Unable to create parallel package General Usage question , package , distributed	1	155	April 19, 2024
Parallel computing, invalid checksum in cache file General Usage parallel	4	919	October 19, 2018

@everywhere takes a very long time when using a cluster

Related topics