`KeyError` when using `pmap`

I’m trying to use pmap form Distributed.jl for parallel computing in the runtests.jl file of a package (MyPkg) that is in development. The code is like:

# runtests.jl

using Distributed: @everywhere, addprocs, pmap
using MyPkg

myvect = [...] # The vector that is to be computed by `myfunc` parallelly.

addprocs(2)
@time result = pmap(myfunc, myvect); # `myfunc` is defined in `MyPkg.jl` (and other `.jl` files included by it)

If the last line is map(myfunc, myvect) the code can run normally. But when I change it to map(myfunc, myvect), it repots:

ERROR: On worker 2:
KeyError: key MyPkg[8f2d0ece-bb0f-4d0c-a9d4-8a8c66038d81] not found
Stacktrace:
  [1] getindex
    @ .\dict.jl:498 [inlined]
  [2] macro expansion
    @ .\lock.jl:267 [inlined]
  [3] root_module
    @ .\loading.jl:1878
  [4] deserialize_module
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Serialization\src\Serialization.jl:994
  [5] handle_deserialize
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Serialization\src\Serialization.jl:896
  [6] deserialize
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Serialization\src\Serialization.jl:814
  [7] deserialize_datatype
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Serialization\src\Serialization.jl:1398
  [8] handle_deserialize
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Serialization\src\Serialization.jl:867
  [9] deserialize
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Serialization\src\Serialization.jl:814
 [10] handle_deserialize
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Serialization\src\Serialization.jl:874
 [11] deserialize
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Serialization\src\Serialization.jl:814 [inlined]
 [12] deserialize_global_from_main
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Distributed\src\clusterserialize.jl:160
 [13] #5
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Distributed\src\clusterserialize.jl:72 [inlined]
 [14] foreach
    @ .\abstractarray.jl:3097
 [15] deserialize
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Distributed\src\clusterserialize.jl:72
 [16] handle_deserialize
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Serialization\src\Serialization.jl:960
 [17] deserialize
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Serialization\src\Serialization.jl:814
 [18] handle_deserialize
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Serialization\src\Serialization.jl:871
 [19] deserialize
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Serialization\src\Serialization.jl:814
 [20] handle_deserialize
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Serialization\src\Serialization.jl:874
 [21] deserialize
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Serialization\src\Serialization.jl:814 [inlined]
 [22] deserialize_msg
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Distributed\src\messages.jl:87
 [23] #invokelatest#2
    @ .\essentials.jl:892 [inlined]
 [24] invokelatest
    @ .\essentials.jl:889 [inlined]
 [25] message_handler_loop
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Distributed\src\process_messages.jl:176
 [26] process_tcp_streams
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Distributed\src\process_messages.jl:133
 [27] #103
    @ D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Distributed\src\process_messages.jl:121
Stacktrace:
  [1] (::Base.var"#1033#1035")(x::Task)
    @ Base .\asyncmap.jl:171
  [2] foreach(f::Base.var"#1033#1035", itr::Vector{Any})
    @ Base .\abstractarray.jl:3097
  [3] maptwice(wrapped_f::Function, chnl::Channel{Any}, worker_tasks::Vector{Any}, c::Vector{Any})
    @ Base .\asyncmap.jl:171
  [4] wrap_n_exec_twice
    @ .\asyncmap.jl:147 [inlined]
  [5] #async_usemap#1018
    @ .\asyncmap.jl:97 [inlined]
  [6] async_usemap
    @ .\asyncmap.jl:78 [inlined]
  [7] #asyncmap#1017
    @ .\asyncmap.jl:75 [inlined]
  [8] asyncmap
    @ .\asyncmap.jl:74 [inlined]
  [9] pmap(f::Function, p::Distributed.WorkerPool, c::Vector{…}; distributed::Bool, batch_size::Int64, on_error::Nothing, retry_delays::Vector{…}, retry_check::Nothing)
    @ Distributed D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Distributed\src\pmap.jl:126
 [10] pmap(f::Function, p::Distributed.WorkerPool, c::Vector{Any})
    @ Distributed D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Distributed\src\pmap.jl:99
 [11] pmap(f::Function, c::Vector{Any}; kwargs::@Kwargs{})
    @ Distributed D:\Software\julia-1.10.3\share\julia\stdlib\v1.10\Distributed\src\pmap.jl:156
 [12] macro expansion
    @ .\timing.jl:279 [inlined]
 [13] top-level scope
    @ d:\name\MyPkg\test\runtests.jl:35

You likely need @everywhere using MyPkg

1 Like

Oh, I see! All the packaged and variables used in runtests.jl should be added @everywhere. Is it?

Sorry, I tried again and it report the same KeyError even I have added @everywhere :woozy_face: