# Unexpected behavior of package loading in workers using Distributed

**URL:** <https://discourse.julialang.org/t/unexpected-behavior-of-package-loading-in-workers-using-distributed/122814>\
**Category:** General Usage\
**Tags:** distributed\
**Created:** [November 19, 2024, 3:09pm UTC](https://discourse.julialang.org/t/unexpected-behavior-of-package-loading-in-workers-using-distributed/122814 "2024-11-19T15:09:11Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![HanD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hand/32/213908_2.png) [@HanD](https://discourse.julialang.org/u/HanD)\
**Post date:** [November 19, 2024, 3:09pm UTC](https://discourse.julialang.org/t/unexpected-behavior-of-package-loading-in-workers-using-distributed/122814/1 "2024-11-19T15:09:11Z")

</div>

The following phenomenon gave me quite a headache and took me a while to track down. Take the following example (using Julia 1.11.1 or 1.10.6, probably older ones as well):

```julia
julia> using Distributed, Statistics

julia> addprocs(1)
1-element Vector{Int64}:
 2

julia> remotecall_fetch(mean, 2, [1,2])
ERROR: On worker 2:
KeyError: key Base.PkgId(Base.UUID("10745b16-79ce-11e8-11f9-7d13ad32a3b2"), "Statistics") not found
...

```

This is fine and expected, the worker process hasn’t loaded the `Statistics` module. And now consider a slightly modified example:

```julia
julia> using Distributed

julia> addprocs(1)
1-element Vector{Int64}:
 2

julia> using Statistics

julia> remotecall_fetch(mean, 2, [1,2])
1.5

julia> remotecall_fetch(typeof, 2, Statistics)
Module

```

Notice that I _deliberately_ haven’t used `@everywhere using Statistics`. So how on earth does the worker process know about the `Statistics` module in this case? What makes `@everywhere` implicit? I tried to find anything at least hinting this kind of behavior in the manual, but failed.

Note that the same happens if I put the above lines in a source file and run Julia non-interactively. E.g., this throws an exception (as expected):

```julia
using Distributed, Statistics
addprocs(1)
println(remotecall_fetch(mean, 2, [1, 2]))

```

Whereas this prints 1.5:

```julia
using Distributed
addprocs(1)
using Statistics
println(remotecall_fetch(mean, 2, [1, 2]))

```

---

<div class="post-metadata">

**Author:** ![HanD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hand/32/213908_2.png) [@HanD](https://discourse.julialang.org/u/HanD)\
**Post date:** [November 19, 2024, 3:23pm UTC](https://discourse.julialang.org/t/unexpected-behavior-of-package-loading-in-workers-using-distributed/122814/2 "2024-11-19T15:23:43Z")

</div>

Two sides of the same coin:

```julia
pkg> activate --temp; add Results
  Activating new project at `/tmp/jl_0SG8Ja`
   Resolving package versions...
    Updating `/tmp/jl_0SG8Ja/Project.toml`
  [a6f57dee] + Results v0.1.2
    Updating `/tmp/jl_0SG8Ja/Manifest.toml`
  [a6f57dee] + Results v0.1.2

julia> using Distributed

julia> addprocs(1, exeflags=`--project=/dev/null`)
1-element Vector{Int64}:
 2

julia> using Results
┌ Error: Error during package callback
│ exception =
│ 1-element ExceptionStack:
│ On worker 2:
│ ArgumentError: Package Results [a6f57dee-de39-11ea-01ba-9f0f7c89923c] is required but does not seem to be installed:
│ - Run `Pkg.instantiate()` to install all recorded dependencies.
...

```

---

<div class="post-metadata">

**Author:** ![HanD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hand/32/213908_2.png) [@HanD](https://discourse.julialang.org/u/HanD)\
**Post date:** [November 19, 2024, 3:36pm UTC](https://discourse.julialang.org/t/unexpected-behavior-of-package-loading-in-workers-using-distributed/122814/3 "2024-11-19T15:36:56Z")

</div>

After further digging, I found the function `_require_callback()` in Distributed.jl, which is registered in `Base.package_callbacks` when the module is loaded. This is in turn invoked when a [new module is loaded](https://docs.julialang.org/en/v1/devdocs/require/#Module-loading-callbacks) via `Base.require` (which is behind `using`/`import`). And it is this function which notifies all workers of the loaded modules.

It is unfortunate that this behavior is not documented at all. It is also unfortunate, IMHO, that this is the default behavior, as it can be achieved with an explicit `@everywhere`, and now there is no way to load a module in only the main process, after (some) workers have been started (or is there?). Perhaps this is just a corner case, but still.

---

<div class="post-metadata">

**Author:** ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Post date:** [November 19, 2024, 4:28pm UTC](https://discourse.julialang.org/t/unexpected-behavior-of-package-loading-in-workers-using-distributed/122814/4 "2024-11-19T16:28:59Z")

</div>

This is definitely an uncomfortable edge case for Distributed users, and is something we’re aiming to fix in [DistributedNext.jl](https://github.com/JuliaParallel/DistributedNext.jl).

I would say that 99% of users generally want newly-added workers to inherit the same project and loaded packages as on worker 1, but we definitely don’t achieve that properly right now (unless everything is done in just the right order and the project specified manually, as you’ve shown).

Keep an eye on DistributedNext - this feature is definitely one of our highest priorities.

(And an important aside: Our goal is to merge any changes to DistributedNext, back into Distributed in the future, once they’ve been thoroughly tested and validated)

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [November 19, 2024, 5:27pm UTC](https://discourse.julialang.org/t/unexpected-behavior-of-package-loading-in-workers-using-distributed/122814/5 "2024-11-19T17:27:27Z")

</div>

> [@jpsamaroo](#):
>
> [DistributedNext.jl](https://github.com/JuliaParallel/DistributedNext.jl)

I had no idea - cool!

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [November 19, 2024, 5:48pm UTC](https://discourse.julialang.org/t/unexpected-behavior-of-package-loading-in-workers-using-distributed/122814/6 "2024-11-19T17:48:25Z")

</div>

> [@jpsamaroo](#):
>
> I would say that 99% of users generally want newly-added workers to inherit the same project and loaded packages as on worker 1

Are you assuming that all workers must run on the exact same platform? What about heterogeneous settings?

Consider a simple setup where node 1 has a GPU and node 2 has a CPU. I would like to dispatch different implementations on these nodes. The same environment can be loaded on all workers by default, but the code that gets compiled should be different. Is that what you have in mind too?

---

<div class="post-metadata">

**Author:** ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Post date:** [November 19, 2024, 7:35pm UTC](https://discourse.julialang.org/t/unexpected-behavior-of-package-loading-in-workers-using-distributed/122814/7 "2024-11-19T19:35:17Z")

</div>

> [@juliohm](#):
>
> Are you assuming that all workers must run on the exact same platform? What about heterogeneous settings?

Great point! We’re also planning for heterogeneous system support. The approach we’re thinking of taking is attaching project/package loading information to some kind of “config” object, which will store an ordered list of operations to be performed. There will be a default config object, used when one is not otherwise specified to `addprocs` (to meet that 99% case most easily), but it will also be possible to build and specify a custom config object to be applied to only certain workers. This would allow you to load only the code you need for each worker.

The more important thing I’m getting at is that, if you want more than one worker to share a given code configuration and environment, it’s very hard to do that right now. I want to add tools to make this automatic when users want that, but also make it just as easy to configure to suit your needs.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [November 27, 2024, 10:32am UTC](https://discourse.julialang.org/t/unexpected-behavior-of-package-loading-in-workers-using-distributed/122814/8 "2024-11-27T10:32:55Z")

</div>

> [@jpsamaroo](#):
>
> Great point! We’re also planning for heterogeneous system support. The approach we’re thinking of taking is attaching project/package loading information to some kind of “config” object, which will store an ordered list of operations to be performed. There will be a default config object, used when one is not otherwise specified to `addprocs` (to meet that 99% case most easily), but it will also be possible to build and specify a custom config object to be applied to only certain workers. This would allow you to load only the code you need for each worker.

We already did that in PlataformAware.jl programming. You can find the corresponding publication _Structured plataform-aware programming_ here:

[https://www.researchgate.net/publication/377248434\_Structured\_platform-aware\_programming](https://www.researchgate.net/publication/377248434_Structured_platform-aware_programming)

@decarvalhojuniorfh just registered his other package CloudClusters.jl, on top of PlataformAware.jl, for distributed execution with heterogeneous hardware:

> **[GitHub - PlatformAwareProgramming/CloudClusters.jl](https://github.com/PlatformAwareProgramming/CloudClusters.jl)**
>
> Contribute to PlatformAwareProgramming/CloudClusters.jl development by creating an account on GitHub.

It is still a prototype, but works already with AWS clusters and other cluster definitions. I suggest that you contact him to learn more about these ongoing efforts.