Scoping is too confusing with custom package and modules to use `pmap`

General Usage
,
1

I saw the previous post about pmap.
However, the answers do not work for me.

Here’s my example. What am I missing?
Note that MyPackage is generated by PkgTemplates.

  • src/MyPackage.jl
module MyPackage

using Reexport


include("ModuleA.jl")
@reexport using .ModuleA


end
  • src/ModuleA.jl
module ModuleA


export myfunc

function myfunc(x)
    return x
end


end
  • test/runtests.jl
using Distributed
addprocs()


thisDir = dirname(@__FILE__())
any(path -> path == thisDir, LOAD_PATH) || push!(LOAD_PATH, thisDir)

using MyPackage

result = pmap(myfunc, 1:10)
@show result
  • error message
julia> include("test/runtests.jl")
ERROR: LoadError: On worker 2:
KeyError: key MyPackage [f8991370-b1bb-43b0-a6d9-a6aa70bc52be] not found
getindex at ./dict.jl:467 [inlined]
root_module at ./loading.jl:968 [inlined]
deserialize_module at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:953
handle_deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:855
deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:773
deserialize_datatype at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:1251
handle_deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:826
deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:773
handle_deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:833
deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:773 [inlined]
deserialize_msg at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/messages.jl:99
#invokelatest#1 at ./essentials.jl:710 [inlined]
invokelatest at ./essentials.jl:709 [inlined]
message_handler_loop at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/process_messages.jl:185
process_tcp_streams at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/process_messages.jl:142
#99 at ./task.jl:356
Stacktrace:
 [1] (::Base.var"#770#772")(::Task) at ./asyncmap.jl:178
 [2] foreach(::Base.var"#770#772", ::Array{Any,1}) at ./abstractarray.jl:2009
 [3] maptwice(::Function, ::Channel{Any}, ::Array{Any,1}, ::UnitRange{Int64}) at ./asyncmap.jl:178
 [4] wrap_n_exec_twice(::Channel{Any}, ::Array{Any,1}, ::Distributed.var"#206#209"{WorkerPool}, ::Function, ::UnitRange{Int64}) at ./asyncmap.jl:154
 [5] async_usemap(::Distributed.var"#190#192"{Distributed.var"#190#191#193"{WorkerPool,typeof(myfunc)}}, ::UnitRange{Int64}; ntasks::Function, batch_size::Nothing) at ./asyncmap.jl:103
 [6] #asyncmap#754 at ./asyncmap.jl:81 [inlined]
 [7] pmap(::Function, ::WorkerPool, ::UnitRange{Int64}; distributed::Bool, batch_size::Int64, on_error::Nothing, retry_delays::Array{Any,1}, retry_check::Nothing) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/pmap.jl:126
 [8] pmap(::Function, ::WorkerPool, ::UnitRange{Int64}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/pmap.jl:101
 [9] pmap(::Function, ::UnitRange{Int64}; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/pmap.jl:156
 [10] pmap(::Function, ::UnitRange{Int64}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/pmap.jl:156
 [11] top-level scope at /home/jinrae/.julia/dev/MyPackage/test/runtests.jl:11
 [12] include(::String) at ./client.jl:457
 [13] top-level scope at REPL[3]:1
in expression starting at /home/jinrae/.julia/dev/MyPackage/test/runtests.jl:11
2

I do not fully understand but this way works for me.

More precisely, change test/runtests.jl to this:

using Distributed
addprocs()


@everywhere push!(LOAD_PATH, ".")
@everywhere using MyPackage

result = pmap(myfunc, 1:10)
@show result
  • result
julia> include("test/runtests.jl")
result = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
10-element Array{Int64,1}:
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
3

You should not be using push!(LOAD_PATH, ".") but instead have your package deved in the environment you set the REPL before doing the include.

In fact, if you are running test/runtests.jl you probably should be doing:

cd MyPackage.jl
julia
> import Pkg
> Pkg.activate(".")
> Pkg.instantiate(".") # Only necessary if you changed dependencies since last call.
> Pkg.test()

If you want to use include directly in the REPL, it may be necessary to replace @everywhere push!(LOAD_PATH, ".") by

@everywhere import Pkg
@everywhere Pkg.activate(".")

inside your test script, and this be executed from a folder that has a Manifest.toml where you deved you package.

1 Like
4

Thanks. For convenience, I changed my startup.jl like this:

% ...
using Distributed
@everywhere using Pkg
if isfile("Project.toml") && isfile("Manifest.toml")
    @everywhere Pkg.activate(".")
end
1 Like