Basic question on distributing modules

Specific Domains Julia at Scale
1

I am using Julia 1.6.1 with Distributed.

Q: Must I first install a module on every node in a cluster to use it? Or is there a way to “push” a module to where it’s needed at runtime?
Apparently, if my reading is any guide, certain things change from one Julia version to another.

I’ve tried
@everywhere include()
but it seems that the remote(s) attempt to include from their respective local file systems.

I would like to write code, push/run it, then code some more, rinse and repeat.

Thanks

2

The usual assumption is that all of your nodes are using a shared network filesystem, so that the packages installed on the head node are available to all nodes. If you are on a network filesystem but it still seems like the workers don’t have the right packages, first check they have the correct environment activated by doing something like @everywhere (using Pkg; Pkg.status()).

If your workers don’t share a filesystem, you will need to install your relevant packages manually into each nodes’ local filesystem. The best way to do this would be to distribute a Manifest.toml to all the worker nodes, and then have them instantiate it. I believe there was talk of doing this automatically in an old thread on Github, but it doesn’t exist yet. Something along these lines should work (note, I haven’t actually tested this so it probably needs tweaks, but gives the general idea)

addprocs(...) # in whatever way you normally do

master_manifest = read("Manifest.toml", String)
@everywhere workers() begin
    write("Manifest.toml", $master_manifest)
    using Pkg
    Pkg.instantiate(".")
end
3

Thanks for clearing that up. Your answer is what I expected, but some discussion threads (old threads) seemed to indicate otherwise.

My situation is cloud based and everything must go through ssh–NFS? no way. And it is “big data” so it won’t fit into my puny, pipsqueak, pusillanimous little laptop.

My dream scenario is to code code code locally, then push/run remotely. But if we must jump through a few hoops along the way–well, we’ve been through that before. I am thinking to first scp my module to remote and install from there. We’ll see.

Thanks so much