Load modules in several workers

General Usage
1

Hi folks,
I’m just running into (another) problem, this time related to loading modules in several workers at the same time.
I have this simple module

module My_mod

export My_Type, f

mutable struct My_type
	x::Int
end

f(x) = x^2 + 1

println("Loaded !!")

end

which I want to load in several workers.

In julia 0.6.4 I could simply do the following

addprocs(3)
@everywhere using My_mod

and the whole module was loaded in all workers. Now in julia 1.0.2 I try

using Distributed
addprocs(3)
@everywhere using My_mod

I get the message

Loaded !!

On worker 2:
ArgumentError: Package My_mod not found in current path:
- Run `import Pkg; Pkg.add("My_mod")` to install the My_mod package.

require at ./loading.jl:823
eval at ./boot.jl:319
#116 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/process_messages.jl:276
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/process_messages.jl:56
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/process_messages.jl:65
#102 at ./task.jl:259
#remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:421
remotecall_wait(::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:412
#remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:433
remotecall_wait(::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:433
(::getfield(Distributed, Symbol("##163#165")){Module,Expr})() at ./task.jl:259

...and 3 more exception(s).


Stacktrace:
 [1] sync_end(::Array{Any,1}) at ./task.jl:226
 [2] remotecall_eval(::Module, ::Array{Int64,1}, ::Expr) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/macros.jl:207
 [3] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/macros.jl:190
 [4] top-level scope at In[5]:1

and,as you can see, it fails to load already in worker 2.
Now the message says I must do import Pkg etc… which does not seem to be the right thing here, as I do not want to install any package but to load a module.
I’m sure I’m probably misunderstanding something, so I could use a bit of help and get the correct way to load my (personal) modules into several workers at once…

Thx in advance,

Ferran.

1 Like
2

A bit more detail would help. In particular, how are you loading My_mod? Also see this comment.

3

Uh… I don’t know what else to say :frowning: I just wrote the whole code, which is just a test to check if functions and modules get loaded in every workout as it did doing exactly and only that in Julia 0.6.4
…but now it seems to fail.
So the question is: I have this module which I want to load in all workers I define: how shall I do that?

4

Is the module saved in its own file? Did you write it at the repl? Is there an associated Project.toml file?

5

Ah, now I understand your question, thanks. Yes the module is stored in its own file My_mod.jl, which is in the LUBRARY_PATH directory… You see the first worker finds and loads it, the rest doesn’t

6

…and I work in IJuoyter, and there is no project.toml thing. Actually and when everything is in place, I will move the code to a script .JL file that loads the module in its separate file.
I have been doing that in 0.6.4 without problems.

7

I don’t have any experience with Jupyter notebook, but the first thing I would try is to @everywhere include("My_mod.jl"), and see if that helps.

8

Aha! That seems to do the trck, thanks… Now when I do the

@everywhere include("My_mod.jl")

it spits

Loaded !!
      From worker 3:	Loaded !!
      From worker 4:	Loaded !!
      From worker 2:	Loaded !!

which I guess means we succeeded :slight_smile:

Now I’m safe, but keep wondering why the

@everywhere using My_mod

has changed to

@everywhere include("My_mod.jl")

but I guess I can live with the change :slight_smile:

Thanks for your help,

Ferran.

9

Here is how I load my modules on other workers

using Distributed
addprocs(4)
#mypath is the path the module file
@everywhere push!(LOAD_PATH,$mypath)
@everywhere using myModule
3 Likes
10

Ah! now I see what happens! It is just the @everywhere in front of

@everywhere push!(LOAD_PATH,$mypath)

that makes the trick. That was not needed before in 0.6.4, but now I see that things only work in later releases if you tell every worker of the LOAD_PATH variable.
Thanks a lot to you also, Christopher :slight_smile:

1 Like
11

Has anyone had the same issue with packages? I cannot get them to load on workers. As an example, the following script returns a file not found in current path error when it hits the @everywhere commands:

using Distributed
using SharedArrays
using Interpolations
addprocs(2)

@everywhere using Interpolations
@everywhere using SharedArrays

vector1 = collect(range(1.0,stop=200.0,length=20))
vector2 = collect(range(100.0,stop=2000.0,length=20))
test = SharedArray{Float64,1}((20))

@distributed for i=1:20
    interp_linear = LinearInterpolation((vector1),vector2, extrapolation_bc = -1e6)
    test[i] = interp_linear(2 .* i)
end

If I modify the script as below to activate packages on the workers, it will run but it doesn’t actually interpolate anything (the vector “test” remains all zeroes; if I delete the @distributed before the for loop, it interpolates correctly). Note that if I include the commented out lines, it returns the same error described above.

using Distributed
using SharedArrays
using Interpolations
addprocs(2)

@everywhere using Pkg
@everywhere Pkg.activate("Interpolations")
@everywhere Pkg.activate("SharedArrays")
#@everywhere using Interpolations
#@everywhere using SharedArrays

vector1 = collect(range(1.0,stop=200.0,length=20))
vector2 = collect(range(100.0,stop=2000.0,length=20))
test = SharedArray{Float64,1}((20))

@distributed for i=1:20
    interp_linear = LinearInterpolation((vector1),vector2, extrapolation_bc = -1e6)
    test[i] = interp_linear(2 .* i)
end

Finally, I can get distributed for loops to run so long as they do not need any packages (unfortunately this makes them pretty useless). Thanks for any ideas.

1 Like
12

This works for me (note the order):

using Distributed
addprocs(2)
@everywhere using SharedArrays
@everywhere using Interpolations
...

.

Are you waiting long enough for the result?

@distributed (without a reduction operator) runs asynchronously and returns immediately.
You can see the return value at the REPL (something like Task (queued) @0x00000000069559f0)

You can wait by calling fetch on the task, or by wrapping the loop with @sync :

@sync @distributed for i=1:20
    interp_linear = LinearInterpolation((vector1),vector2, extrapolation_bc = -1e6)
    test[i] = interp_linear(2 .* i)
end

See the manual: Parallel Computing · The Julia Language

13

Thanks but unfortunately it looks like that exact order still gives me the same error as when I had the using commands both before and after addprocs. Running…

using Distributed

addprocs(2)

@everywhere using Interpolations
@everywhere using SharedArrays

… I get the following error for the Interpolations package (running for SharedArrays just before Interpolations as in your note has the same behavior):

ERROR: LoadError: On worker 2:
ArgumentError: Package Interpolations not found in current path:
- Run `import Pkg; Pkg.add("Interpolations")` to install the Interpolations package.

As for the second point, I was waiting for the script to finish. I’m not sure why going the Pkg.activate route isn’t throwing an error but it doesn’t seem to be activating anything on the workers.

14

Hi, were you able to resolve the issue in the end? I’m running into the same issue, but couldn’t find anywhere a solution to this… Thanks

15

No I wasn’t sadly. It looks to me like it’s just a bug in Julia so I don’t know how to get around it. I switched back to MATLAB with C-MEX.

16

Well, this might be because of the location of the environment.
Have you tried something along

EDIT: Actually depending on your setup it might not be pwd() that you need to activate, but some other environment (maybe “C:\users\myname\.julia\environments\v1.0”)

current_dir=pwd()
@everywhere current_dir=$current_dir
@everywhere import Pkg
@everywhere Pkg.activate(current_dir)
@everywhere using Interpolations
17

Perhaps compare the output of load_path() on master vs worker processes (maybe there’s a path missing from the worker process).

This works for me on Windows, Julia v1.1.1 with a fresh REPL.

julia> using Distributed

julia> addprocs(2)
2-element Array{Int64,1}:
 2
 3

julia> @everywhere using SharedArrays

julia> @everywhere using Interpolations

julia> Base.load_path()
2-element Array{String,1}:
 "C:\\Users\\Greg\\.julia\\environments\\v1.1\\Project.toml"
 "C:\\Program Files\\Julia-1.1.1\\share\\julia\\stdlib\\v1.1"

julia> remotecall_fetch(Base.load_path, 2)
2-element Array{String,1}:
 "C:\\Users\\Greg\\.julia\\environments\\v1.1\\Project.toml"
 "C:\\Program Files\\Julia-1.1.1\\share\\julia\\stdlib\\v1.1"
18

The output of load_path is different in my case. I also tried with Julia v1.1.1, the same issue arises. How should I fix this?

julia> Base.load_path()
2-element Array{String,1}:
 "/Users/nk/.juliapro/JuliaPro_v1.0.4.1/environments/v1.0/Project.toml"                                        
 "/Applications/JuliaPro-1.0.4.1.app/Contents/Resources/julia/Contents/Resources/julia/share/julia/stdlib/v1.0"

julia> remotecall_fetch(Base.load_path, 2)
2-element Array{String,1}:
 "/Users/nk/.julia/environments/v1.0/Project.toml"                                                            
 "/Applications/JuliaPro-1.0.4.1.app/Contents/Resources/julia/Contents/Resources/julia/share/julia/stdlib/v1.0"

I tried adding the missing path to worker processes by running dir=Base.load_path() @everywhere push!(LOAD_PATH,$dir[1]). But I’m still getting the same error message…

LoadError: On worker 2:
ArgumentError: Package Interpolations [a98d9a8b-a2ab-59e6-89dd-64a1c18fca59] is required but does not seem to be installed:
19

Do you have both standard Julia and JuliaPro installed?

Similar problems have been reported before:

Clean installs seem to have resolved the issues.

2 Likes
20

I did at some point have both Julia and JuliaPro installed. I uninstalled them both and reinstalled JuliaPro 1.2 after I encountered some other issues.

I’m still getting these issues even though I thought I did have a “clean install.” I even tried uninstalling everything again and deleting all the odd files I could find. How do I get a “clean install” when it looks like everything was uninstalled?