# Scoping is too confusing with custom package and modules to use \`pmap\`

**URL:** <https://discourse.julialang.org/t/scoping-is-too-confusing-with-custom-package-and-modules-to-use-pmap/53383>\
**Category:** General Usage\
**Tags:** parallel, pmap\
**Created:** [January 15, 2021, 1:04pm UTC](https://discourse.julialang.org/t/scoping-is-too-confusing-with-custom-package-and-modules-to-use-pmap/53383 "2021-01-15T13:04:57Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![iHany](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ihany/32/18151_2.png) [@iHany](https://discourse.julialang.org/u/iHany)\
**Post date:** [January 15, 2021, 1:04pm UTC](https://discourse.julialang.org/t/scoping-is-too-confusing-with-custom-package-and-modules-to-use-pmap/53383/1 "2021-01-15T13:04:57Z")

</div>

I saw [the previous post](https://discourse.julialang.org/t/how-am-i-supposed-to-use-pmap/3943/12) about `pmap`.  
However, the answers do not work for me.

Here’s my example. What am I missing?  
Note that `MyPackage` is generated by [PkgTemplates](https://github.com/invenia/PkgTemplates.jl).

- `src/MyPackage.jl`

```julia
module MyPackage

using Reexport

include("ModuleA.jl")
@reexport using .ModuleA

end

```

- `src/ModuleA.jl`

```julia
module ModuleA

export myfunc

function myfunc(x)
    return x
end

end

```

- `test/runtests.jl`

```julia
using Distributed
addprocs()

thisDir = dirname(@ __FILE__ ())
any(path -> path == thisDir, LOAD_PATH) || push!(LOAD_PATH, thisDir)

using MyPackage

result = pmap(myfunc, 1:10)
@show result

```

- error message

```nohighlight
julia> include("test/runtests.jl")
ERROR: LoadError: On worker 2:
KeyError: key MyPackage [f8991370-b1bb-43b0-a6d9-a6aa70bc52be] not found
getindex at ./dict.jl:467 [inlined]
root_module at ./loading.jl:968 [inlined]
deserialize_module at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:953
handle_deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:855
deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:773
deserialize_datatype at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:1251
handle_deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:826
deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:773
handle_deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:833
deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:773 [inlined]
deserialize_msg at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/messages.jl:99
#invokelatest#1 at ./essentials.jl:710 [inlined]
invokelatest at ./essentials.jl:709 [inlined]
message_handler_loop at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/process_messages.jl:185
process_tcp_streams at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/process_messages.jl:142
#99 at ./task.jl:356
Stacktrace:
 [1] (::Base.var"#770#772")(::Task) at ./asyncmap.jl:178
 [2] foreach(::Base.var"#770#772", ::Array{Any,1}) at ./abstractarray.jl:2009
 [3] maptwice(::Function, ::Channel{Any}, ::Array{Any,1}, ::UnitRange{Int64}) at ./asyncmap.jl:178
 [4] wrap_n_exec_twice(::Channel{Any}, ::Array{Any,1}, ::Distributed.var"#206#209"{WorkerPool}, ::Function, ::UnitRange{Int64}) at ./asyncmap.jl:154
 [5] async_usemap(::Distributed.var"#190#192"{Distributed.var"#190#191#193"{WorkerPool,typeof(myfunc)}}, ::UnitRange{Int64}; ntasks::Function, batch_size::Nothing) at ./asyncmap.jl:103
 [6] #asyncmap#754 at ./asyncmap.jl:81 [inlined]
 [7] pmap(::Function, ::WorkerPool, ::UnitRange{Int64}; distributed::Bool, batch_size::Int64, on_error::Nothing, retry_delays::Array{Any,1}, retry_check::Nothing) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/pmap.jl:126
 [8] pmap(::Function, ::WorkerPool, ::UnitRange{Int64}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/pmap.jl:101
 [9] pmap(::Function, ::UnitRange{Int64}; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/pmap.jl:156
 [10] pmap(::Function, ::UnitRange{Int64}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/pmap.jl:156
 [11] top-level scope at /home/jinrae/.julia/dev/MyPackage/test/runtests.jl:11
 [12] include(::String) at ./client.jl:457
 [13] top-level scope at REPL[3]:1
in expression starting at /home/jinrae/.julia/dev/MyPackage/test/runtests.jl:11

```

---

<div class="post-metadata">

**Author:** ![iHany](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ihany/32/18151_2.png) [@iHany](https://discourse.julialang.org/u/iHany)\
**Post date:** [January 15, 2021, 1:14pm UTC](https://discourse.julialang.org/t/scoping-is-too-confusing-with-custom-package-and-modules-to-use-pmap/53383/2 "2021-01-15T13:14:26Z")

</div>

I do not fully understand but [this way](https://discourse.julialang.org/t/custom-modules-with-distributed/14321/2) works for me.

More precisely, change `test/runtests.jl` to this:

```julia
using Distributed
addprocs()

@everywhere push!(LOAD_PATH, ".")
@everywhere using MyPackage

result = pmap(myfunc, 1:10)
@show result

```

- result

```nohighlight
julia> include("test/runtests.jl")
result = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
10-element Array{Int64,1}:
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10

```

---

<div class="post-metadata">

**Author:** ![Henrique\_Becker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrique_becker/32/15443_2.png) [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)\
**Post date:** [January 15, 2021, 3:22pm UTC](https://discourse.julialang.org/t/scoping-is-too-confusing-with-custom-package-and-modules-to-use-pmap/53383/3 "2021-01-15T15:22:28Z")

</div>

You should not be using `push!(LOAD_PATH, ".")` but instead have your package `dev`ed in the environment you set the REPL before doing the include.

In fact, if you are running `test/runtests.jl` you probably should be doing:

```julia
cd MyPackage.jl
julia
> import Pkg
> Pkg.activate(".")
> Pkg.instantiate(".") # Only necessary if you changed dependencies since last call.
> Pkg.test()

```

If you want to use `include` directly in the REPL, it _may_ be necessary to replace `@everywhere push!(LOAD_PATH, ".")` by

```julia
@everywhere import Pkg
@everywhere Pkg.activate(".")

```

inside your test script, and this be executed from a folder that has a `Manifest.toml` where you `dev`ed you package.

---

<div class="post-metadata">

**Author:** ![iHany](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ihany/32/18151_2.png) [@iHany](https://discourse.julialang.org/u/iHany)\
**Post date:** [January 16, 2021, 12:37pm UTC](https://discourse.julialang.org/t/scoping-is-too-confusing-with-custom-package-and-modules-to-use-pmap/53383/4 "2021-01-16T12:37:18Z")

</div>

Thanks. For convenience, I changed my `startup.jl` like this:

```julia
% ...
using Distributed
@everywhere using Pkg
if isfile("Project.toml") && isfile("Manifest.toml")
    @everywhere Pkg.activate(".")
end

```