Is there a way to execute a function on a remote worker which doesn't have the module?

General Usage
1

Consider the following code:

module Foo
using Distributed
remote_work(i) = Distributed.remotecall_fetch(()->1+i, 2)
end

using Distributed
addprocs(1)

Foo.remote_work(1) # errors

The last line errors with,

ERROR: On worker 2:
UndefVarError: Foo not defined
deserialize_module at /home/marius/src/julia-1.4/usr/share/julia/stdlib/v1.4/Serialization/src/Serialization.jl:915
...

presumably because module Foo is (intentionally) not defined on the worker. However, obviously nothing in ()->1+i actually depends on Foo, so it seems like it could run, its just that somewhere in that closure (maybe a line number or some other hidden metadata?) there’s a reference to Foo. So is there a way to get this to run on a remote worker anyway?

One obvious solution is to eval the closure in Main which works in stripping out any references to Foo which may have been there:

remote_work(i) = Distributed.remotecall_fetch(@eval(Main,()->1+$i), 2)

but in my real case it becomes too onerous to track down all the variables like $i that need to be interpolated, so I’m hoping there’s another (even if hacky) solution. Any ideas? Thanks.

2

I think it’s a little more complicated since you can have multiple remote_work from different modules. Someone can probably explain better than me.

That being said, why not just evaluate the module on the workers? i.e. @everywhere using Foo or are you just trying this as a theoretical exercise?

3

Yea, that’s a good point, I guess the fact that ()->1+i is a closure inside Foo.remote_work is kind of part of the “metadata” I was referring to above, and it does seem like that’s going to be hard to strip out…

Only because it doubles the loading times for packages (which are already ~1min for my modules). Specifically, its for my pacakge Py2Call which lets you call Python 2 & 3 in the same session by spawning a worker which is running a Python-2-built PyCall, which doesn’t really need to load any of the modules loaded in the main process.