I’m having difficulty loading a module on remote workers. Suppose I have a file called myModule.jl in a directory assigned to variable called path. In the past, I simply added path to
LOAD_PATH and entered
@everywhere using myModule. This does not work anymore.
myFunction(x) = x + 1
#top level code in run.jl
path = "/home/dfish/myFolder"
@everywhere using myModule
When I run the code, I receive the following error:
ERROR: LoadError: On worker 2:
ArgumentError: Package myModule not found in current path:
- Run `Pkg.add("myModule")` to install the myModule package.
require at ./loading.jl:817
eval at ./boot.jl:319
The error message indicates that I need to add it as a package. However, I can load it into the primary processor without issue:
As you can see, my LOAD_PATH and file system are both properly configured:
@everywhere include("myModule.jl") but the code failed at
Code loading was revamped during 0.7 development and I think loading code from node 1 is no longer supported.
My understanding is that each worker must have access to a (possibly shared) filesystem that contains the module, and loads code locally.
Perhaps the “old” behavior of loading from node 1 is a feature left to be implemented in package.
See some discussion here:
Thanks for your reply. Unfortunately, the discussion did not clarify how to load my own modules to other workers. Evidently in Julia 1/.7,
using can load any module to the first node and can load a module that is a package to any node, as this shows:
@everywhere using Distributions
@everywhere f(x) = rand(Normal(0,1),1000)
I find this behavior to be confusing in contrast to previous versions in which
using loaded modules in all situations. Do you know how I can make my own
loadEverywhere() function to replace this lost functionality?
I think your example of
using Distributions works for local workers because all the workers have access to the local package directory.
I suspect this would not work for remote workers if
Distributions is not installed on those remote workers.
A workaround for loading your own modules on local workers might be something like:
module_dir = "/home/dfish/myFolder"
@everywhere push!(LOAD_PATH, $module_dir)
@everywhere using myModule
AFAICT, the easiest way to load code on remote workers is to provide shared access to the local machine’s packages and modules.
@everywhere push!(LOAD_PATH, $module_dir) fixed the issue. I tried
@everywhere push!(LOAD_PATH,path) at one point, but it looks like the
$ is critical to its operation. Would you be willing to explain how I provide shared access? This is new to me.
?addprocs you see that calling
addprocs(4) will add 4 processes on the local machine.
so the issue here is not a shared file system. all processes on your machine have access to the same, unique, file system, i.e. your disk. the only thing that changed to previous julia versions is that you need to push stuff onto the LOAD_PATH on all workers, as indeed @greg_plowman said.
If you had 2 machines, say, you would have to worry about this. either, you maintain exact equal code bases on both machines (via git probably), or you would have a file share between them, such that
"/home/dfish/myFolder" is the exact same file on both of them, for example. on ubuntu this is not too bad to do yourself: https://help.ubuntu.com/community/SettingUpNFSHowTo.
Excellent. Thanks for your explanation. I will make Greg’s post as the answer.
I also forgot to add
push!(LOAD_PATH, pwd()). Thanks.