Using module does not work with SSH remote workers

General Usage
1

From what I understand, using Module should use the module available on the master process, and send it to the workers. However, it is not working for me:

julia> addprocs([("admin@some.server",4)],exename="julia",dir="/home/admin",tunnel=true)
4-element Array{Int64,1}:
 2
 3
 4
 5

julia> using OrdinaryDiffEq
ERROR: On worker 2:
UndefVarError: ##694#696 not defined
#99 at ./event.jl:73
#remotecall_fetch#141(::Array{Any,1}, ::Function, ::Function, ::Base.Distributed.Worker) at .\distributed\remotecall.jl:354
remotecall_fetch(::Function, ::Base.Distributed.Worker) at .\distributed\remotecall.jl:346
#remotecall_fetch#144(::Array{Any,1}, ::Function, ::Function, ::Int64) at .\distributed\remotecall.jl:367
remotecall_fetch(::Function, ::Int64) at .\distributed\remotecall.jl:367
macro expansion at .\loading.jl:180 [inlined]
(::Base.##693#695)() at .\task.jl:335

...and 3 more exception(s).

Stacktrace:
 [1] sync_end() at .\task.jl:287
 [2] macro expansion at .\task.jl:303 [inlined]
 [3] _require_from_serialized(::Int64, ::Symbol, ::String, ::Bool) at .\loading.jl:177
 [4] _require_search_from_serialized(::Int64, ::Symbol, ::String, ::Bool) at .\loading.jl:236
 [5] _require(::Symbol) at .\loading.jl:441
 [6] require(::Symbol) at .\loading.jl:405

Am I missing anything obvious here or is it a Julia issue?

Thanks,
Jeremy

2

You will have to say @everywhere using OrdinaryDiffEq.

The manual says:

using DummyModule causes the module to be loaded on all processes; however, the module is brought into scope only on the one executing the statement.

https://docs.julialang.org/en/stable/manual/parallel-computing/#Code-Availability-and-Loading-Packages-1

3

That’s not the issue, like the passage you quoted says, using Module should be sufficient to load the code on the remote workers.

In any case, adding @everywhere produces the exact same error.

4

It will load the code, but it will not bring the code into scope.

But it it seems that the error you are seeing is coming from somewhere else…
On which Julia version are you? (versioninfo())?

5

I’m running 0.6.2 on the master and workers.

BTW, in this context, what does it mean for the code to not be in scope on the workers? I though that only meant that no functions were exported, but that the module name would still be available. Is that not correct?

6

The fully-qualified module name will work, but none of the methods exported by it will be implicitly available. So in most cases you do want to do an @everywhere using ....

Is this happening only for OrdinaryDiffEq or also for other modules?

7

It happens with other modules as well.

What is the fully qualified module name? Is it Main.Module?

8

And this is SSH worker specific and doesn’t occur with only local processes? Did you ever try this on 0.6.1?

This looks to me like a proper Julia bug and so I am trying to narrow down what is happening.

Yes Main.X.func is the fully-qualified name but X.func should work as well.

9

Local workers work properly. I have not tried on 0.6.1, and I also have not tried with non-tunneled workers since I can’t get them to work.

Interestingly, I tried a few more packages and found that using Compat works.

10

That actually seems to be because Compat is automatically loaded on the workers.

11

@jebej thank you for reporting this. If you have the time/chance to try yet another module that makes use of multiple workers, please try to follow this notebook that I know was working with Julia v0.6.1 on my laptop at least. Please ignore the warnings as it is not fully implemented: https://github.com/juliohm/GeoStats.jl/blob/master/examples/ParallelSimulation.ipynb

12

The problem is not with using the workers, it’s just with loading the code with using.

Loading the code with @everywhere f(x,y) = x+y works fine, and executes properly.