Problems parallelizing declaring used packages

Hi,
The issue here is that your worker processes aren’t running in the same environment as the main process. By default new workers will launch in the default environment. For some reason (on julia 1.1.1, linux)

@everywhere Pkg.activate(".")

doesn’t seem to activate the current directory’s project on the workers. I’m not sure if that’s a bug or the intended behaviour.
Edit: The fact that

@everywhere activate(".")

doesn’t work is by design. There is an open issue (Workers should inherit Pkg environment · Issue #28781 · JuliaLang/julia · GitHub) discussing why that is the case and how the situation can be improved. Seems like the workaround I describe below (which I came across on another discourse thread: Packages and workers - #9 by teored90) is still the way to go.

In any event, you can get the workers to launch in the environment defined in the current directory’s Project.toml file by passing the exeflags keyword argument to `addprocs, like so:

addprocs(2, exeflags="--project=.")

With the workers launched like that I’m able to load packages on the workers that are only installed in the current environment.

1 Like