Weird behaviour with packages and multiple processes

General Usage Performance
1

Hi, I’m trying to develop a package which has some functionality that uses Distributed, I created a project to test and run my function but I’m getting some weirdness:

So I created a project with the package I wanted to test:

(mercounttest) pkg> st
    Status `~/mercounttest/Project.toml`
  [6e4b80f9] BenchmarkTools v0.5.0
  [7e6ae17a] BioSequences v2.0.1
  [a20136b7] MerCounting v0.1.0 #master (https://github.com/BioJulia/MerCounting.jl.git)
  [70a005b8] ReadDatastores v0.2.1

The package I want to test is installed from github master as it has no release or version yet.

Ok there’s a function I want to test called MerCounting.calculate_minimizer_distribution which basically just launches a worker function on however many processes are available and collects the results.

So this is what I see after doing julia --project:

N82106:mercounttest bward$ julia --project
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.3.0 (2019-11-26)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |

julia> using BioSequences, ReadDatastores, MerCounting, BenchmarkTools

julia> r = MerCounting.calculate_minimizer_distribution(MerCounting.MinimizerTable{31,7}, PairedReads{DNAAlphabet{4}}, "ecoli-test-paired.prseq", MerCounting.CANONICAL, 100)
1-element Array{MerCounting.MinimizerTable{31,7},1}:
 MerCounting.MinimizerTable{31,7}(0x00000064, UInt32[0x000e15c4, 0x000fb9dc, 0x0010f51f, 0x000ed441, 0x000abd05, 0x000bf0c4, 0x000a6376, 0x000bcc40, 0x000d74b4, 0x000897d4  …  0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000], UInt32[])

Awesome, that’s what I expect to see. It’s a whopping 5 seconds, which is not great but you know I hope that gets faster as I add processes:

N82106:mercounttest bward$ julia --project -p 3
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.3.0 (2019-11-26)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |

julia> using BioSequences, ReadDatastores, MerCounting, BenchmarkTools
ERROR: On worker 2:
ArgumentError: Package MerCounting [a20136b7-8e32-4c10-91d3-7060c0bd8ec7] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

_require at ./loading.jl:993
require at ./loading.jl:922
#1 at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/Distributed.jl:78
#105 at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:290
run_work_thunk at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:79
run_work_thunk at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:88
#98 at ./task.jl:333

...and 2 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at ./task.jl:300
 [2] _require_callback(::Base.PkgId) at ./task.jl:319
 [3] #invokelatest#1 at ./essentials.jl:709 [inlined]
 [4] invokelatest at ./essentials.jl:708 [inlined]
 [5] require(::Base.PkgId) at ./loading.jl:925
 [6] require(::Module, ::Symbol) at ./loading.jl:917

(mercounttest) pkg> instantiate

julia> using BioSequences, ReadDatastores, MerCounting, BenchmarkTools

julia> r = MerCounting.calculate_minimizer_distribution(MerCounting.MinimizerTable{31,7}, PairedReads{DNAAlphabet{4}}, "ecoli-test-paired.prseq", MerCounting.CANONICAL, 100)
ERROR: TaskFailedException:
On worker 2:
KeyError: key MerCounting [a20136b7-8e32-4c10-91d3-7060c0bd8ec7] not found
getindex at ./dict.jl:477 [inlined]
root_module at ./loading.jl:962 [inlined]
deserialize_module at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:894
handle_deserialize at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:799
deserialize at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:722
deserialize_datatype at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:1192
handle_deserialize at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:775
deserialize at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:722
handle_deserialize at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:782
deserialize at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Serialization/src/Serialization.jl:722 [inlined]
deserialize_msg at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/messages.jl:99
#invokelatest#1 at ./essentials.jl:709 [inlined]
invokelatest at ./essentials.jl:708 [inlined]
message_handler_loop at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:185
process_tcp_streams at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:142
#101 at ./task.jl:333
Stacktrace:
 [1] #remotecall_fetch#145(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_fetch), ::Function, ::Distributed.Worker, ::Type, ::Vararg{Any,N} where N) at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:390
 [2] remotecall_fetch(::Function, ::Distributed.Worker, ::Type, ::Vararg{Any,N} where N) at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:382
 [3] #remotecall_fetch#148(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_fetch), ::Function, ::Int64, ::Type, ::Vararg{Any,N} where N) at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:417
 [4] remotecall_fetch at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:417 [inlined]
 [5] (::MerCounting.var"#1#2"{PairedReads{DNAAlphabet{4}},MerCounting.MinimizerTable{31,7},String,Canonical,Int64,Array{MerCounting.MinimizerTable{31,7},1},Int64})() at ./task.jl:333

...and 2 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at ./task.jl:300
 [2] calculate_minimizer_distribution(::Type{MerCounting.MinimizerTable{31,7}}, ::Type{PairedReads{DNAAlphabet{4}}}, ::String, ::Canonical, ::Int64) at ./task.jl:319
 [3] top-level scope at REPL[4]:1

julia> @everywhere using BioSequences, ReadDatastores, MerCounting, BenchmarkTools
ERROR: On worker 2:
ArgumentError: Package ReadDatastores not found in current path:
- Run `import Pkg; Pkg.add("ReadDatastores")` to install the ReadDatastores package.

require at ./loading.jl:887
eval at ./boot.jl:330
#105 at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:290
run_work_thunk at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:79
run_work_thunk at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:88
#98 at ./task.jl:333

...and 2 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at ./task.jl:300
 [2] macro expansion at ./task.jl:319 [inlined]
 [3] remotecall_eval(::Module, ::Array{Int64,1}, ::Expr) at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/macros.jl:217
 [4] top-level scope at /Users/bward/repos/julia13/usr/share/julia/stdlib/v1.3/Distributed/src/macros.jl:201

(mercounttest) pkg> st
    Status `~/mercounttest/Project.toml`
  [6e4b80f9] BenchmarkTools v0.5.0
  [7e6ae17a] BioSequences v2.0.1
  [a20136b7] MerCounting v0.1.0 #master (https://github.com/BioJulia/MerCounting.jl.git)
  [70a005b8] ReadDatastores v0.2.1

So I’m not sure what to do about this. It first says MerTools is not instantiated. So I call instantiate to see if that helps, it does not seem to. I figure trying the @everwhere macro with my using statement, but that doesn’t help either, I get a message about ReadDatastores, even though it is installed.

Can anyone see what it is I’m missing to just load my in dev package and test the function?

Thanks

2

The problem is that when you launch julia with the -p flag, the worker processes are launched in the default environment, regardless of the value passed to the --project argument. The only way I’ve gotten packages that are not in my default environment to load on remote workers is to use the exeflags keyword argument to addprocs. Here’s an example, where the StatsBase package is included in the environment defined by the Project.toml file in the working directory but is not in my default environment:

(base) ➜  test-proj julia --project=.
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.3.1 (2019-12-30)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Distributed

julia> addprocs(4,exeflags="--project=.")
4-element Array{Int64,1}:
 2
 3
 4
 5

julia> @everywhere using StatsBase

That works, whereas the following fails:

(base) ➜  test-proj julia --project=. -p 4
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.3.1 (2019-12-30)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> @everywhere using StatsBase
ERROR: On worker 2:
ArgumentError: Package StatsBase not found in current path:
- Run `import Pkg; Pkg.add("StatsBase")` to install the StatsBase package.

require at ./loading.jl:887
eval at ./boot.jl:330
#105 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:290
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:79
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:88
#98 at ./task.jl:333

...and 3 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at ./task.jl:300
 [2] macro expansion at ./task.jl:319 [inlined]
 [3] remotecall_eval(::Module, ::Array{Int64,1}, ::Expr) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/macros.jl:217
 [4] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/macros.jl:201

There may be a more elegant way to handle this problem but if so, I don’t know what that is.

2 Likes
3

Thanks @Pbellive, that works perfectly!