# Weird behaviour with packages and multiple processes

**URL:** <https://discourse.julialang.org/t/weird-behaviour-with-packages-and-multiple-processes/37651>\
**Category:** Performance\
**Tags:** distributed\
**Created:** [April 15, 2020, 3:46pm UTC](https://discourse.julialang.org/t/weird-behaviour-with-packages-and-multiple-processes/37651 "2020-04-15T15:46:15Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ward9250](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ward9250/32/42768_2.png) [@Ward9250](https://discourse.julialang.org/u/Ward9250)\
**Post date:** [April 15, 2020, 3:46pm UTC](https://discourse.julialang.org/t/weird-behaviour-with-packages-and-multiple-processes/37651/1 "2020-04-15T15:46:16Z")

</div>

Hi, I’m trying to develop a package which has some functionality that uses Distributed, I created a project to test and run my function but I’m getting some weirdness:

So I created a project with the package I wanted to test:

```julia
(mercounttest) pkg> st
    Status `~/mercounttest/Project.toml`
  [6e4b80f9] BenchmarkTools v0.5.0
  [7e6ae17a] BioSequences v2.0.1
  [a20136b7] MerCounting v0.1.0 #master (https://github.com/BioJulia/MerCounting.jl.git)
  [70a005b8] ReadDatastores v0.2.1

```

The package I want to test is installed from github master as it has no release or version yet.

Ok there’s a function I want to test called `MerCounting.calculate_minimizer_distribution` which basically just launches a worker function on however many processes are available and collects the results.

So this is what I see after doing `julia --project`:

```julia
N82106:mercounttest bward$ julia --project
               _
   _ _ _(_)_ | Documentation: https://docs.julialang.org
  (_) | (_) (_) |
   _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` | |
  | | |_| | | | (_| | | Version 1.3.0 (2019-11-26)
 _/ |\ __'_|_|_|\__'_| |  
|__/ |

julia> using BioSequences, ReadDatastores, MerCounting, BenchmarkTools

julia> r = MerCounting.calculate_minimizer_distribution(MerCounting.MinimizerTable{31,7}, PairedReads{DNAAlphabet{4}}, "ecoli-test-paired.prseq", MerCounting.CANONICAL, 100)
1-element Array{MerCounting.MinimizerTable{31,7},1}:
 MerCounting.MinimizerTable{31,7}(0x00000064, UInt32[0x000e15c4, 0x000fb9dc, 0x0010f51f, 0x000ed441, 0x000abd05, 0x000bf0c4, 0x000a6376, 0x000bcc40, 0x000d74b4, 0x000897d4 … 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000], UInt32[])

```

Awesome, that’s what I expect to see. It’s a whopping 5 seconds, which is not great but you know I hope that gets faster as I add processes:

```julia
N82106:mercounttest bward$ julia --project -p 3
               _
   _ _ _(_)_ | Documentation: https://docs.julialang.org
  (_) | (_) (_) |
   _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` | |
  | | |_| | | | (_| | | Version 1.3.0 (2019-11-26)
 _/ |\ __'_|_|_|\__'_| |  
|__/ |

julia> using BioSequences, ReadDatastores, MerCounting, BenchmarkTools
ERROR: On worker 2:
ArgumentError: Package MerCounting [a20136b7-8e32-4c10-91d3-7060c0bd8ec7] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

_require at ./loading.jl:993
require at ./loading.jl:922
#1 at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/Distributed.jl:78
#105 at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:290
run_work_thunk at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:79
run_work_thunk at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:88
#98 at ./task.jl:333

...and 2 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at ./task.jl:300
 [2] _require_callback(::Base.PkgId) at ./task.jl:319
 [3] #invokelatest#1 at ./essentials.jl:709 [inlined]
 [4] invokelatest at ./essentials.jl:708 [inlined]
 [5] require(::Base.PkgId) at ./loading.jl:925
 [6] require(::Module, ::Symbol) at ./loading.jl:917

(mercounttest) pkg> instantiate

julia> using BioSequences, ReadDatastores, MerCounting, BenchmarkTools

julia> r = MerCounting.calculate_minimizer_distribution(MerCounting.MinimizerTable{31,7}, PairedReads{DNAAlphabet{4}}, "ecoli-test-paired.prseq", MerCounting.CANONICAL, 100)
ERROR: TaskFailedException:
On worker 2:
KeyError: key MerCounting [a20136b7-8e32-4c10-91d3-7060c0bd8ec7] not found
getindex at ./dict.jl:477 [inlined]
root_module at ./loading.jl:962 [inlined]
deserialize_module at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:894
handle_deserialize at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:799
deserialize at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:722
deserialize_datatype at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:1192
handle_deserialize at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:775
deserialize at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:722
handle_deserialize at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:782
deserialize at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:722 [inlined]
deserialize_msg at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/messages.jl:99
#invokelatest#1 at ./essentials.jl:709 [inlined]
invokelatest at ./essentials.jl:708 [inlined]
message_handler_loop at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:185
process_tcp_streams at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:142
#101 at ./task.jl:333
Stacktrace:
 [1] #remotecall_fetch#145(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_fetch), ::Function, ::Distributed.Worker, ::Type, ::Vararg{Any,N} where N) at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:390
 [2] remotecall_fetch(::Function, ::Distributed.Worker, ::Type, ::Vararg{Any,N} where N) at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:382
 [3] #remotecall_fetch#148(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_fetch), ::Function, ::Int64, ::Type, ::Vararg{Any,N} where N) at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:417
 [4] remotecall_fetch at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:417 [inlined]
 [5] (::MerCounting.var"#1#2"{PairedReads{DNAAlphabet{4}},MerCounting.MinimizerTable{31,7},String,Canonical,Int64,Array{MerCounting.MinimizerTable{31,7},1},Int64})() at ./task.jl:333

...and 2 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at ./task.jl:300
 [2] calculate_minimizer_distribution(::Type{MerCounting.MinimizerTable{31,7}}, ::Type{PairedReads{DNAAlphabet{4}}}, ::String, ::Canonical, ::Int64) at ./task.jl:319
 [3] top-level scope at REPL[4]:1

julia> @everywhere using BioSequences, ReadDatastores, MerCounting, BenchmarkTools
ERROR: On worker 2:
ArgumentError: Package ReadDatastores not found in current path:
- Run `import Pkg; Pkg.add("ReadDatastores")` to install the ReadDatastores package.

require at ./loading.jl:887
eval at ./boot.jl:330
#105 at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:290
run_work_thunk at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:79
run_work_thunk at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:88
#98 at ./task.jl:333

...and 2 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at ./task.jl:300
 [2] macro expansion at ./task.jl:319 [inlined]
 [3] remotecall_eval(::Module, ::Array{Int64,1}, ::Expr) at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/macros.jl:217
 [4] top-level scope at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/macros.jl:201

(mercounttest) pkg> st
    Status `~/mercounttest/Project.toml`
  [6e4b80f9] BenchmarkTools v0.5.0
  [7e6ae17a] BioSequences v2.0.1
  [a20136b7] MerCounting v0.1.0 #master (https://github.com/BioJulia/MerCounting.jl.git)
  [70a005b8] ReadDatastores v0.2.1

```

So I’m not sure what to do about this. It first says MerTools is not instantiated. So I call instantiate to see if that helps, it does not seem to. I figure trying the @everwhere macro with my using statement, but that doesn’t help either, I get a message about ReadDatastores, even though it is installed.

Can anyone see what it is I’m missing to just load my in dev package and test the function?

Thanks

---

<div class="post-metadata">

**Author:** ![Pbellive](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbellive/32/3604_2.png) [@Pbellive](https://discourse.julialang.org/u/Pbellive)\
**Post date:** [April 15, 2020, 6:23pm UTC](https://discourse.julialang.org/t/weird-behaviour-with-packages-and-multiple-processes/37651/2 "2020-04-15T18:23:11Z")

</div>

The problem is that when you launch julia with the `-p` flag, the worker processes are launched in the default environment, regardless of the value passed to the `--project` argument. The only way I’ve gotten packages that are not in my default environment to load on remote workers is to use the `exeflags` keyword argument to addprocs. Here’s an example, where the `StatsBase` package is included in the environment defined by the Project.toml file in the working directory but is not in my default environment:

```julia
(base) ➜ test-proj julia --project=.
               _
   _ _ _(_)_ | Documentation: https://docs.julialang.org
  (_) | (_) (_) |
   _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` | |
  | | |_| | | | (_| | | Version 1.3.1 (2019-12-30)
 _/ |\ __'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |

julia> using Distributed

julia> addprocs(4,exeflags="--project=.")
4-element Array{Int64,1}:
 2
 3
 4
 5

julia> @everywhere using StatsBase

```

That works, whereas the following fails:

```julia
(base) ➜ test-proj julia --project=. -p 4
               _
   _ _ _(_)_ | Documentation: https://docs.julialang.org
  (_) | (_) (_) |
   _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` | |
  | | |_| | | | (_| | | Version 1.3.1 (2019-12-30)
 _/ |\ __'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |

julia> @everywhere using StatsBase
ERROR: On worker 2:
ArgumentError: Package StatsBase not found in current path:
- Run `import Pkg; Pkg.add("StatsBase")` to install the StatsBase package.

require at ./loading.jl:887
eval at ./boot.jl:330
#105 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:290
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:79
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:88
#98 at ./task.jl:333

...and 3 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at ./task.jl:300
 [2] macro expansion at ./task.jl:319 [inlined]
 [3] remotecall_eval(::Module, ::Array{Int64,1}, ::Expr) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/macros.jl:217
 [4] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/macros.jl:201

```

There may be a more elegant way to handle this problem but if so, I don’t know what that is.

---

<div class="post-metadata">

**Author:** ![Ward9250](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ward9250/32/42768_2.png) [@Ward9250](https://discourse.julialang.org/u/Ward9250)\
**Post date:** [April 15, 2020, 6:25pm UTC](https://discourse.julialang.org/t/weird-behaviour-with-packages-and-multiple-processes/37651/3 "2020-04-15T18:25:57Z")

</div>

Thanks @Pbellive, that works perfectly!