Parallel computing: calling module failed

Specific Domains Julia at Scale
1

Background:

I wrote a module myself, where the function requires parallel computing.Here is my code:

module sds

    using Distributed
    @everywhere using SharedArrays

    export test

    function test(N)
        S = SharedMatrix{Int64}(N,N)
        @sync @distributed for i in 1:length(S)
            S[i] = i
        end
        return S
    end
    
end

Then I call the above module in another file:

using Distributed
addprocs(2)
using .sds

test(1000)

Error：

On worker 2:
UndefVarError: sds not defined
deserialize_module at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Serialization\src\Serialization.jl:893
handle_deserialize at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Serialization\src\Serialization.jl:795
deserialize at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Serialization\src\Serialization.jl:731
deserialize_datatype at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Serialization\src\Serialization.jl:1114
handle_deserialize at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Serialization\src\Serialization.jl:771
deserialize at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Serialization\src\Serialization.jl:731
deserialize_datatype at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Serialization\src\Serialization.jl:1139
handle_deserialize at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Serialization\src\Serialization.jl:771
deserialize at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Serialization\src\Serialization.jl:731
handle_deserialize at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Serialization\src\Serialization.jl:778
deserialize_msg at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Serialization\src\Serialization.jl:731
#invokelatest#1 at .\essentials.jl:742 [inlined]
invokelatest at .\essentials.jl:741 [inlined]
message_handler_loop at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Distributed\src\process_messages.jl:160
process_tcp_streams at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Distributed\src\process_messages.jl:117
#105 at .\task.jl:259
#remotecall_fetch#149(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker, ::Distributed.RRID, ::Vararg{Any,N} where N) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:379
remotecall_fetch(::Function, ::Distributed.Worker, ::Distributed.RRID, ::Vararg{Any,N} where N) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:371
#remotecall_fetch#152(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64, ::Distributed.RRID, ::Vararg{Any,N} where N) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:406
call_on_owner at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:406 [inlined]
wait(::Future) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:500
sync_end(::Array{Any,1}) at .\task.jl:213
(::getfield(Distributed, Symbol("##167#169")){getfield(Main.sds, Symbol("##1#2")){SharedArray{Int64,2}},UnitRange{Int64}})() at .\task.jl:245

Stacktrace:
 [1] sync_end(::Array{Any,1}) at .\task.jl:226
 [2] macro expansion at .\task.jl:245 [inlined]
 [3] test(::Int64) at .\In[1]:10
 [4] top-level scope at In[2]:4

Questions:

In fact, I am not sure how to call a module with parallel parts.
Everyone can help me to change code or provide information on this. Thank you very much.:smile:

2

Hi, the following should work. Basically @everywhere module sds loads the module in all workers

using Distributed
addprocs(2)

@everywhere module sds

using Distributed, SharedArrays

export test

function test(N)
    S = SharedMatrix{Int}(N,N)
    @sync @distributed for i in 1:length(S)
        S[i] = i
    end
    return S
end

end  # module

using .sds

test(1000)

I would advise you to go through the Parallel computing/Code availability section in the Julia docs. There they also describe what to do if your module is in a separate function.