Error, parallel computing in a module

General Usage
1

I am creating a package which use parallel computing. I encountered some errors and not sure which place to put addprocs() and implement the parallel code. Please give some suggestions. The code is as follows.
Repost: correct code errors caused by editing

./src/test_map.jl

module test_map
using Distributed
addprocs(1)             # there a one worker on the top-level namespace of TestPara,
@show nprocs()      # 2 processes

#include("mymath.jl")
include("do.jl")

function include_everywhere(filepath)
    fullpath = joinpath(@__DIR__, filepath)
    @sync for p in procs()
        @async remotecall_wait(include, p, fullpath)
    end
end

function check_procs()
    return nprocs()
end 

function add_worker(n)
    @show nprocs()     # 1 process here 
    addprocs(2)     
    @show nprocs()    # 3 process now
end

function addme_error1()
    @everywhere begin 
        function myadd1(n)
            return 0
        end
    end

    sqrts = pmap(sqrt, 1:10)   # it work

    pmap( myadd1, 1:10 )    # error here, `addme_test` not defined
end

function addme_error2()
    include_everywhere( "do.jl" )
    pmap( myadd2, 1:10 )
end
end

./src/do.jl

function myadd2(n::Int)
    return 0
end

The ./test/runtests.jl

using Test
import test_map 
using Distributed

@testset "Test procs number" begin
    @show test_map.check_procs() .    # 1 proc
    test_map.add_worker(2)   
    @show nprocs() 
    test_map.check_procs()    # 3 procs,  does these two workers only available in all TestPara's function ?
end

@testset "Test quick calculation" begin
    test_map.check_procs()        # 3 process
    @show nprocs(), workers()    # 3 process
    test_map.addme_error1()                 # error here, `UndefVarError: myadd1 not defined`
end

@testset "Test quick calculation seperate file" begin
    test_map.addme_error2()    # error here, `KeyError: key test_map`
end
end

Errors:

Error in `addme_error1()`
est quick calculation: Error During Test at /Users/zhangliye/julia_dev/test_map/test/runtests.jl:21
  Got exception outside of a @test
  UndefVarError: myadd1 not defined
  Stacktrace:
   [1] addme_error1() at /Users/zhangliye/julia_dev/test_map/src/test_map.jl:53
   [2] top-level scope at /Users/zhangliye/julia_dev/test_map/test/runtests.jl:25
   [3] top-level scope at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083
   [4] top-level scope at /Users/zhangliye/julia_dev/test_map/test/runtests.jl:22
   [5] include at ./boot.jl:326 [inlined]
   [6] include_relative(::Module, ::String) at ./loading.jl:1038
   [7] include(::Module, ::String) at ./sysimg.jl:29
   [8] include(::String) at ./client.jl:403
   [9] top-level scope at none:0
   [10] eval(::Module, ::Any) at ./boot.jl:328
   [11] exec_options(::Base.JLOptions) at ./client.jl:243
   [12] _start() at ./client.jl:436
Test Summary:          | Error  Total
Test quick calculation |     1      1
ERROR: LoadError: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.
in expression starting at /Users/zhangliye/julia_dev/test_map/test/runtests.jl:21
ERROR: Package test_map errored during testing

errors in addme_error2()

Test quick calculation seperate file: Error During Test at /Users/zhangliye/julia_dev/test_map/test/runtests.jl:28
  Got exception outside of a @test
  On worker 2:
  KeyError: key test_map [eb2e8f2e-cb12-11e9-2feb-1d5725766938] not found
  getindex at ./dict.jl:478 [inlined]
  root_module at ./loading.jl:898 [inlined]
  deserialize_module at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Serialization/src/Serialization.jl:890
  handle_deserialize at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Serialization/src/Serialization.jl:795
  deserialize at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Serialization/src/Serialization.jl:731
  deserialize_datatype at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Serialization/src/Serialization.jl:1114
  handle_deserialize at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Serialization/src/Serialization.jl:771
  deserialize at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Serialization/src/Serialization.jl:731
  handle_deserialize at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Serialization/src/Serialization.jl:778
  deserialize_msg at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Serialization/src/Serialization.jl:731
  #invokelatest#1 at ./essentials.jl:742 [inlined]
  invokelatest at ./essentials.jl:741 [inlined]
  message_handler_loop at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Distributed/src/process_messages.jl:160
  process_tcp_streams at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Distributed/src/process_messages.jl:117
  #105 at ./task.jl:259
  #remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker, ::String) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Distributed/src/remotecall.jl:421
  remotecall_wait(::Function, ::Distributed.Worker, ::String) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Distributed/src/remotecall.jl:412
  #remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64, ::String) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Distributed/src/remotecall.jl:433
  remotecall_wait(::Function, ::Int64, ::String) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Distributed/src/remotecall.jl:433
  (::getfield(test_map, Symbol("##1#2")){String})() at ./task.jl:259
  
  ...and 1 more exception(s).
  
  Stacktrace:
   [1] sync_end(::Array{Any,1}) at ./task.jl:226
   [2] include_everywhere(::String) at ./task.jl:245
   [3] addme_error2() at /Users/zhangliye/julia_dev/test_map/src/test_map.jl:57
   [4] top-level scope at /Users/zhangliye/julia_dev/test_map/test/runtests.jl:29
   [5] top-level scope at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083
   [6] top-level scope at /Users/zhangliye/julia_dev/test_map/test/runtests.jl:29
   [7] include at ./boot.jl:326 [inlined]
   [8] include_relative(::Module, ::String) at ./loading.jl:1038
   [9] include(::Module, ::String) at ./sysimg.jl:29
   [10] include(::String) at ./client.jl:403
   [11] top-level scope at none:0
   [12] eval(::Module, ::Any) at ./boot.jl:328
   [13] exec_options(::Base.JLOptions) at ./client.jl:243
   [14] _start() at ./client.jl:436
Test Summary:                        | Error  Total
Test quick calculation seperate file |     1      1
ERROR: LoadError: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.
in expression starting at /Users/zhangliye/julia_dev/test_map/test/runtests.jl:28
ERROR: Package test_map errored during testing
1 Like
2

If I put all the code outside a module, everything works fine. The smallest example is as follows.

    using Distributed

    function include_everywhere(filepath)
        fullpath = joinpath(@__DIR__, filepath)
        @sync for p in procs()
            @async remotecall_wait(include, p, fullpath)
        end
    end

    addprocs(2)
    @show nprocs(), workers()

    @everywhere function myadd1(n)
            println("Hello from: ")
            return 0
        end

    function test_para()
        @show nprocs()   
        sqrts = pmap(sqrt, 1:10)   # it work

        pmap(myadd1, 1:10)
    end

    test_para()

If I put the code within a module, there is error.

module TestPara 
    using Distributed

    function include_everywhere(filepath)
        fullpath = joinpath(@__DIR__, filepath)
        @sync for p in procs()
            @async remotecall_wait(include, p, fullpath)
        end
    end

    addprocs(2)
    @show nprocs(), workers()

    @everywhere function myadd1(n)
            println("Hello from: ")
            return 0
        end

    function test_para()
        @show nprocs()   
        sqrts = pmap(sqrt, 1:10)   # it work

        pmap(myadd1, 1:10)
    end

    # test_para()

end 

using .TestPara
TestPara.test_para()

Error info:

ERROR: LoadError: UndefVarError: myadd1 not defined
Stacktrace:
 [1] test_para() at /Users/zhangliye/julia_dev/TestPara/src/main.jl:25
 [2] top-level scope at none:0
 [3] include at ./boot.jl:326 [inlined]
 [4] include_relative(::Module, ::String) at ./loading.jl:1038
 [5] include(::Module, ::String) at ./sysimg.jl:29
 [6] exec_options(::Base.JLOptions) at ./client.jl:267
 [7] _start() at ./client.jl:436
in expression starting at /Users/zhangliye/julia_dev/TestPara/src/main.jl:33
1 Like
3

Firstly, it’s probably best to avoid hard-coding adding workers in the module itself, because you presumably want the flexibility of running the module in different configurations with different numbers of workers.

For your example, something like this should work:

using Distributed
addprocs(2)
@show nprocs(), workers()

@everywhere module TestPara
    using Distributed

    function myadd1(n)
        println("Hello from: ", myid())
        return 0
    end

    function test_para()
        sqrts = pmap(sqrt, 1:10)
        adds = pmap(myadd1, 1:10)
        return sqrts, adds
    end
end

using .TestPara
TestPara.test_para()

Note that @everywhere before module TestPara defines the code on all workers, and using .TestPara brings it into scope.

If the module is already defined, you just need to ensure that the path to the module is in LOAD_PATH on all the workers, and then @everywhere using TestPara should be sufficient.

2 Likes