Load modules in several workers


#1

Hi folks,
I’m just running into (another) problem, this time related to loading modules in several workers at the same time.
I have this simple module

module My_mod

export My_Type, f

mutable struct My_type
	x::Int
end

f(x) = x^2 + 1

println("Loaded !!")

end

which I want to load in several workers.

In julia 0.6.4 I could simply do the following

addprocs(3)
@everywhere using My_mod

and the whole module was loaded in all workers. Now in julia 1.0.2 I try

using Distributed
addprocs(3)
@everywhere using My_mod

I get the message

Loaded !!

On worker 2:
ArgumentError: Package My_mod not found in current path:
- Run `import Pkg; Pkg.add("My_mod")` to install the My_mod package.

require at ./loading.jl:823
eval at ./boot.jl:319
#116 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/process_messages.jl:276
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/process_messages.jl:56
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/process_messages.jl:65
#102 at ./task.jl:259
#remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:421
remotecall_wait(::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:412
#remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:433
remotecall_wait(::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:433
(::getfield(Distributed, Symbol("##163#165")){Module,Expr})() at ./task.jl:259

...and 3 more exception(s).


Stacktrace:
 [1] sync_end(::Array{Any,1}) at ./task.jl:226
 [2] remotecall_eval(::Module, ::Array{Int64,1}, ::Expr) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/macros.jl:207
 [3] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/macros.jl:190
 [4] top-level scope at In[5]:1

and,as you can see, it fails to load already in worker 2.
Now the message says I must do import Pkg etc… which does not seem to be the right thing here, as I do not want to install any package but to load a module.
I’m sure I’m probably misunderstanding something, so I could use a bit of help and get the correct way to load my (personal) modules into several workers at once…

Thx in advance,

Ferran.


#2

A bit more detail would help. In particular, how are you loading My_mod? Also see this comment.


#3

Uh… I don’t know what else to say :frowning: I just wrote the whole code, which is just a test to check if functions and modules get loaded in every workout as it did doing exactly and only that in Julia 0.6.4
…but now it seems to fail.
So the question is: I have this module which I want to load in all workers I define: how shall I do that?


#4

Is the module saved in its own file? Did you write it at the repl? Is there an associated Project.toml file?


#5

Ah, now I understand your question, thanks. Yes the module is stored in its own file My_mod.jl, which is in the LUBRARY_PATH directory… You see the first worker finds and loads it, the rest doesn’t


#6

…and I work in IJuoyter, and there is no project.toml thing. Actually and when everything is in place, I will move the code to a script .JL file that loads the module in its separate file.
I have been doing that in 0.6.4 without problems.


#7

I don’t have any experience with Jupyter notebook, but the first thing I would try is to @everywhere include("My_mod.jl"), and see if that helps.


#8

Aha! That seems to do the trck, thanks… Now when I do the

@everywhere include("My_mod.jl")

it spits

Loaded !!
      From worker 3:	Loaded !!
      From worker 4:	Loaded !!
      From worker 2:	Loaded !!

which I guess means we succeeded :slight_smile:

Now I’m safe, but keep wondering why the

@everywhere using My_mod

has changed to

@everywhere include("My_mod.jl")

but I guess I can live with the change :slight_smile:

Thanks for your help,

Ferran.


#9

Here is how I load my modules on other workers

using Distributed
addprocs(4)
#mypath is the path the module file
@everywhere push!(LOAD_PATH,$mypath)
@everywhere using myModule

#10

Ah! now I see what happens! It is just the @everywhere in front of

@everywhere push!(LOAD_PATH,$mypath)

that makes the trick. That was not needed before in 0.6.4, but now I see that things only work in later releases if you tell every worker of the LOAD_PATH variable.
Thanks a lot to you also, Christopher :slight_smile:


#11

Has anyone had the same issue with packages? I cannot get them to load on workers. As an example, the following script returns a file not found in current path error when it hits the @everywhere commands:

using Distributed
using SharedArrays
using Interpolations
addprocs(2)

@everywhere using Interpolations
@everywhere using SharedArrays

vector1 = collect(range(1.0,stop=200.0,length=20))
vector2 = collect(range(100.0,stop=2000.0,length=20))
test = SharedArray{Float64,1}((20))

@distributed for i=1:20
    interp_linear = LinearInterpolation((vector1),vector2, extrapolation_bc = -1e6)
    test[i] = interp_linear(2 .* i)
end

If I modify the script as below to activate packages on the workers, it will run but it doesn’t actually interpolate anything (the vector “test” remains all zeroes; if I delete the @distributed before the for loop, it interpolates correctly). Note that if I include the commented out lines, it returns the same error described above.

using Distributed
using SharedArrays
using Interpolations
addprocs(2)

@everywhere using Pkg
@everywhere Pkg.activate("Interpolations")
@everywhere Pkg.activate("SharedArrays")
#@everywhere using Interpolations
#@everywhere using SharedArrays

vector1 = collect(range(1.0,stop=200.0,length=20))
vector2 = collect(range(100.0,stop=2000.0,length=20))
test = SharedArray{Float64,1}((20))

@distributed for i=1:20
    interp_linear = LinearInterpolation((vector1),vector2, extrapolation_bc = -1e6)
    test[i] = interp_linear(2 .* i)
end

Finally, I can get distributed for loops to run so long as they do not need any packages (unfortunately this makes them pretty useless). Thanks for any ideas.


#12

This works for me (note the order):

using Distributed
addprocs(2)
@everywhere using SharedArrays
@everywhere using Interpolations
...

.

Are you waiting long enough for the result?

@distributed (without a reduction operator) runs asynchronously and returns immediately.
You can see the return value at the REPL (something like Task (queued) @0x00000000069559f0)

You can wait by calling fetch on the task, or by wrapping the loop with @sync :

@sync @distributed for i=1:20
    interp_linear = LinearInterpolation((vector1),vector2, extrapolation_bc = -1e6)
    test[i] = interp_linear(2 .* i)
end

See the manual: https://docs.julialang.org/en/v1/manual/parallel-computing/#Parallel-Map-and-Loops-1


#13

Thanks but unfortunately it looks like that exact order still gives me the same error as when I had the using commands both before and after addprocs. Running…

using Distributed

addprocs(2)

@everywhere using Interpolations
@everywhere using SharedArrays

… I get the following error for the Interpolations package (running for SharedArrays just before Interpolations as in your note has the same behavior):

ERROR: LoadError: On worker 2:
ArgumentError: Package Interpolations not found in current path:
- Run `import Pkg; Pkg.add("Interpolations")` to install the Interpolations package.

As for the second point, I was waiting for the script to finish. I’m not sure why going the Pkg.activate route isn’t throwing an error but it doesn’t seem to be activating anything on the workers.