# Module loading on workers

**URL:** <https://discourse.julialang.org/t/module-loading-on-workers/31814>\
**Category:** General Usage\
**Tags:** question, distributed\
**Created:** [December 3, 2019, 5:22pm UTC](https://discourse.julialang.org/t/module-loading-on-workers/31814 "2019-12-03T17:22:04Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![pfarndt](https://avatars.discourse-cdn.com/v4/letter/p/8dc957/32.png) [@pfarndt](https://discourse.julialang.org/u/pfarndt)\
**Post date:** [December 3, 2019, 5:22pm UTC](https://discourse.julialang.org/t/module-loading-on-workers/31814/1 "2019-12-03T17:22:04Z")

</div>

Through some trials and errors I realized that modules, which are loaded with `using`, are also loaded on the already existing workers. Therefore the following code runs without an error:

```julia
using Distributed
addprocs(2)

using PyPlot

pmap(workers()) do w
   w, myid(), PyPlot.version
end

```

I find this behavior quite inconvenient because in a situation with many workers (to generate some data) connected to one jupyter notebook kernel (to analyze this data) I rather prefer not to load a plotting package on all workers.

Somewhat inconsistently to this behavior I found out that a module which is already loaded in the main process will not be loaded on all workers, if it was include with `using` before the workers were created. Julia is very smart and seems to know that the module is already there in process 1, however does not load it into the workers.

Therefore this code gives errors:

```julia
using Distributed
using PyPlot

addprocs(2)

using PyPlot

pmap(workers()) do w
   w, myid(), PyPlot.version
end

```

which certainly can be resolved by dressing the second `using` with `@everywhere`:

```julia
using Distributed
using PyPlot

addprocs(2)

@everywhere using PyPlot

pmap(workers()) do w
   w, myid(), PyPlot.version
end

```

Is this intended the intended behavior? Is there a way to load a module only on the local process even if there are already workers running?

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [December 3, 2019, 6:51pm UTC](https://discourse.julialang.org/t/module-loading-on-workers/31814/2 "2019-12-03T18:51:13Z")

</div>

The intended behaviour is that `using` loads the packages on workers so that type definition are available and you can serialize and deserialize messages between the workers.

I consider the second behaviour where doing `using; addprocs; using` does not load the package on the recently added workers a bug, which i attempted to fix in [https://github.com/JuliaLang/julia/pull/28860](https://github.com/JuliaLang/julia/pull/28860)

---

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [December 3, 2019, 6:55pm UTC](https://discourse.julialang.org/t/module-loading-on-workers/31814/3 "2019-12-03T18:55:33Z")

</div>

> [@pfarndt](#):
>
> I find this behavior quite inconvenient because in a situation with many workers (to generate some data) connected to one jupyter notebook kernel (to analyze this data) I rather prefer not to load a plotting package on all workers.

I have the exact same usecase, although I’ve slowly trained myself to separate imports before/after `addprocs` depending on if you want them on the workers or not.

That said, I just played around with it a bit and the following seems like a way to remove the auto-loading feature:

```julia
using Distributed
addprocs()

filter!(!=(Distributed._require_callback), Base.package_callbacks)

using PyPlot # now only loaded on master process

pmap(workers()) do w
   w, myid(), PyPlot.version
end # will now error

```

and you can still `@everywhere using PyPlot` later and it will work. There may well be other things I haven’t thought of that this breaks though so I’d be careful messing with the internals like this.

---

<div class="post-metadata">

**Author:** ![pfarndt](https://avatars.discourse-cdn.com/v4/letter/p/8dc957/32.png) [@pfarndt](https://discourse.julialang.org/u/pfarndt)\
**Post date:** [December 3, 2019, 7:50pm UTC](https://discourse.julialang.org/t/module-loading-on-workers/31814/4 "2019-12-03T19:50:59Z")

</div>

> [@vchuravy](#):
>
> The intended behaviour is that `using` loads the packages on workers so that type definition are available and you can serialize and deserialize messages between the workers.

I still find this a bit inconsistent, since statements like `x=1` need an `@everywhere` to have an effect on workers - but `using XYZ` not.

And yes - if `using XYZ ` would finish fast and only make type definitions visible on the workers I would not care. But right now I use to have 200+ workers connected to my master process. Running `using Plots` would probably crash my hard drive and the network.

In the meantime I thought that probably `@everywhere [1] using XYZ` would do the job - but no - `using` overrules the specification of the worker processes `[1]`.

I would definitely vote for a some kind of `@onlyhere` macro.

---

<div class="post-metadata">

**Author:** ![dstarerstor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dstarerstor/32/8958_2.png) [@dstarerstor](https://discourse.julialang.org/u/dstarerstor)\
**Post date:** [December 5, 2019, 5:34pm UTC](https://discourse.julialang.org/t/module-loading-on-workers/31814/5 "2019-12-05T17:34:57Z")

</div>

Does `import` also load onto every worker? If not, you could use ImportAll.jl  
[https://github.com/NTimmons/ImportAll.jl](https://github.com/NTimmons/ImportAll.jl)  
There’s probably a better way, though (like the above, or just loading the package before adding workers)