Loading modules on remote workers in Julia 1.0/.7

General Usage
1

I’m having difficulty loading a module on remote workers. Suppose I have a file called myModule.jl in a directory assigned to variable called path. In the past, I simply added path to LOAD_PATH and entered @everywhere using myModule. This does not work anymore.

#myModule.jl

module myModule
    export myFunction
    myFunction(x) = x + 1
end

#top level code in run.jl

path = "/home/dfish/myFolder"
push!(LOAD_PATH,path)
using Distributed
addprocs(4)
@everywhere using myModule
pmap(myFunction,1:1000)

When I run the code, I receive the following error:

ERROR: LoadError: On worker 2:
ArgumentError: Package myModule not found in current path:
- Run `Pkg.add("myModule")` to install the myModule package.

require at ./loading.jl:817
eval at ./boot.jl:319 
...

The error message indicates that I need to add it as a package. However, I can load it into the primary processor without issue:

 using myModule

 myFunction(2)
  3

As you can see, my LOAD_PATH and file system are both properly configured:

 readdir(path)
  2-element Array{String,1}:
 "myModule.jl"
 "run.jl"

 LOAD_PATH
  5-element Array{String,1}:
 "@"
 "@v#.#"
 "@stdlib"
 "/home/dfish/Projects/ACT-R-Likelihood/FFT Models/FFT"
 "/home/dfish/myFolder"

I tried @everywhere include("myModule.jl") but the code failed at pmap.

2

Code loading was revamped during 0.7 development and I think loading code from node 1 is no longer supported.

My understanding is that each worker must have access to a (possibly shared) filesystem that contains the module, and loads code locally.

Perhaps the “old” behavior of loading from node 1 is a feature left to be implemented in package.

See some discussion here:
https://github.com/JuliaLang/julia/pull/19738
https://github.com/JuliaLang/julia/pull/19910
https://github.com/JuliaLang/julia/pull/20120

3

Thanks for your reply. Unfortunately, the discussion did not clarify how to load my own modules to other workers. Evidently in Julia 1/.7, using can load any module to the first node and can load a module that is a package to any node, as this shows:

using Distributed
addprocs(4)
@everywhere using Distributions
@everywhere f(x) = rand(Normal(0,1),1000)
pmap(f,1:10)

I find this behavior to be confusing in contrast to previous versions in which using loaded modules in all situations. Do you know how I can make my own loadEverywhere() function to replace this lost functionality?

1 Like
4

I think your example of using Distributions works for local workers because all the workers have access to the local package directory.

I suspect this would not work for remote workers if Distributions is not installed on those remote workers.

A workaround for loading your own modules on local workers might be something like:

using Distributed
addprocs()
module_dir = "/home/dfish/myFolder"
@everywhere push!(LOAD_PATH, $module_dir)
@everywhere using myModule
pmap(myFunction, 1:1000)

AFAICT, the easiest way to load code on remote workers is to provide shared access to the local machine’s packages and modules.

1 Like
5

Thanks. Adding @everywhere push!(LOAD_PATH, $module_dir) fixed the issue. I tried @everywhere push!(LOAD_PATH,path) at one point, but it looks like the $ is critical to its operation. Would you be willing to explain how I provide shared access? This is new to me.

1 Like
6

from ?addprocs you see that calling addprocs(4)

addprocs(4) will add 4 processes on the local machine.

so the issue here is not a shared file system. all processes on your machine have access to the same, unique, file system, i.e. your disk. the only thing that changed to previous julia versions is that you need to push stuff onto the LOAD_PATH on all workers, as indeed @greg_plowman said.

If you had 2 machines, say, you would have to worry about this. either, you maintain exact equal code bases on both machines (via git probably), or you would have a file share between them, such that "/home/dfish/myFolder" is the exact same file on both of them, for example. on ubuntu this is not too bad to do yourself: SettingUpNFSHowTo - Community Help Wiki.

7

Excellent. Thanks for your explanation. I will make Greg’s post as the answer.

8

I also forgot to add @everywhere to push!(LOAD_PATH, pwd()). Thanks.