How to handle code availability in a parallel-computing-enabled package

General Usage
1

I am becoming increasingly confused about code availability in parallel computing. Basically, I want to write a package that will distribute tasks automatically on all processes and I don’t know how to understand code availability.

This is official documentation:

julia> function rand2(dims...)
           return 2*rand(dims...)
       end

julia> rand2(2,2)
2×2 Array{Float64,2}:
 0.153756  0.368514
 1.15119   0.918912

julia> fetch(@spawn rand2(2,2))
ERROR: RemoteException(2, CapturedException(UndefVarError(Symbol("#rand2"))
[...]

It shows rand2 has to be made available on all processes.

However, this is working:

# mytest.jl
push!(LOAD_PATH, "C:/Users/wangc/Downloads") # This is my path

using Para

println(mypara())

# Para.jl
module Para

export mypara

function rand2(dims...)
    return 2*rand(dims...)
end

function mypara()
	return fetch(@spawn rand2(2, 2))
end

end

Executing julia -p 3 mytest.jl gives precisely what I want although I never made rand2 available on other processes.

I have two questions:

  1. How to understand the behavior above?
  2. More generally, how to take care of code availability inside a package that performs parallel computing?
2

From the rest of the subsection in docs https://docs.julialang.org/en/stable/manual/parallel-computing:

Starting Julia with julia -p 2, you can use this to verify the following:

  • include("DummyModule.jl") loads the file on just a single process (whichever one executes the statement).
  • using DummyModule causes the module to be loaded on all processes; however, the module is brought into scope only on the one executing the statement.

An example of loading a module but not bringing it to scope on other processes (julia -p 3):

julia> using StaticArrays

julia> fetch(@spawnat 1 SVector((1,2,3)))
3-element SVector{3,Int64}:
 1
 2
 3

julia> fetch(@spawnat 2 SVector((1,2,3)))
ERROR: On worker 2:
UndefVarError: SVector not defined
#3 at .\distributed\macros.jl:25
#103 at .\distributed\process_messages.jl:264 [inlined]
run_work_thunk at .\distributed\process_messages.jl:56
run_work_thunk at .\distributed\process_messages.jl:65 [inlined]
#96 at .\event.jl:73
Stacktrace:
 [1] #remotecall_fetch#141(::Array{Any,1}, ::Function, ::Function, ::Base.Distributed.Worker, ::Base.Distributed.RRID, ::Vararg{Any,N} where N) at .\distributed\remotecall.jl:354
 [2] remotecall_fetch(::Function, ::Base.Distributed.Worker, ::Base.Distributed.RRID, ::Vararg{Any,N} where N) at .\distributed\remotecall.jl:346
 [3] #remotecall_fetch#144(::Array{Any,1}, ::Function, ::Function, ::Int64, ::Base.Distributed.RRID, ::Vararg{Any,N} where N) at .\distributed\remotecall.jl:367
 [4] call_on_owner(::Function, ::Future) at .\distributed\remotecall.jl:440
 [5] fetch(::Future) at .\distributed\remotecall.jl:460

julia> fetch(@spawnat 2 StaticArrays.SVector((1,2,3)))
3-element SVector{3,Int64}:
 1
 2
 3

Notice if you start julia without additional processes, @spawn will still work but it will spawn to itself. An example with julia (single process):

julia> using StaticArrays

julia> fetch(@spawn SVector((1,2,3)))
3-element SVector{3,Int64}:
 1
 2
 3

But this is not what’s happening in your example because you started with additional processes, so @spawn will only spawn to a proc of id > 1. In the above code, mypara is run by proc 1. But when spawning rand2, it spawns to proc 2 for example. So the only way this could just work, which it does, is if rand2 was actually called as Para.rand2 which makes it identifiable to proc 2. This means that Julia replaces any “internal” (non-exported) identifier by Module.identifier. This is verifiable from the following code:

julia> module Dummy
           f() = 2
           g() = f()
           export g
       end
Dummy

julia> using Dummy

julia> @code_lowered g()
CodeInfo(:(begin
        nothing
        return (Dummy.f)()
    end))
3
  1. It seems it also works even if rand2 is exported.
  2. Is this behavior guaranteed such that I can count on it in my program? Or it is just how julia is implemented such that it may change due to change of implimentation in future versions?
4

Yes it works, my bad. It seems I mis-defined the word “internal”, it is not equivalent to non-exported. It seems all functions called indirectly are lowered to their Module.func form:

julia> module Dummy
           f() = 2
           g() = f()
           export f, g
       end
Dummy

julia> using Dummy

julia> @code_lowered g()
CodeInfo(:(begin
        nothing
        return (Dummy.f)()
    end))

I think “indirectly called” is more correct than “internal”. The case is different with exported functions called indirectly from global functions as shown below:

julia> h() = g()
h (generic function with 1 method)

julia> @code_lowered h()
CodeInfo(:(begin
        nothing
        return (Main.g)()
    end))

I am not sure what is the best way to describe this behavior, but there you have it.

I am in no position to guarantee anything. But if something like this changes, it may break some code, so it may have to wait until 2.0.