Error sending packages to other workers (parallelization)

General Usage
1

Hi,
In previous versions of julia I used to run

using Distributed
addprocs(3)
@everywhere using Distributions

where Distributions is just one example of a package. Now I get this error

On worker 2:
ArgumentError: Package Distributions not found in current path:

  • Run import Pkg; Pkg.add("Distributions") to install the Distributions package.

How can I send packages to other workers in Juia 1.?

Thank you

2

Can’t reproduce:

julia> using Distributed

julia> addprocs(3)
3-element Array{Int64,1}:
 2
 3
 4

julia> @everywhere using Distributions

julia> remotecall_fetch(rand, 2, Normal(5,5))
8.680947219352166

julia> versioninfo()
Julia Version 1.0.3
Commit 099e826241 (2018-12-18 01:34 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-4200U CPU @ 1.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, haswell)

What does using Distributions return if you try without @everywhere?

3

Distributions works for me, but I have the same problem with OnlineStats, with and without the @everywhere macro.

If I don’t use addprocs(), I can load OnlineStats fine:

julia> using Pkg; Pkg.activate("C:\\Users\\op\\Documents\\Julia Projects\\StrategyGames")
"C:\\Users\\op\\Documents\\Julia Projects\\StrategyGames\\Project.toml"

julia> using Distributed

julia> using OnlineStats

Starting a fresh Julia session and using addprocs(), I get the same error as juliauser.

julia> using Pkg; Pkg.activate("C:\\Users\\op\\Documents\\Julia Projects\\StrategyGames")
"C:\\Users\\op\\Documents\\Julia Projects\\StrategyGames\\Project.toml"

julia> using Distributed; addprocs(2)
2-element Array{Int64,1}:
 2
 3

julia> using OnlineStats
ERROR: On worker 2:
ArgumentError: Package OnlineStats [a15396b6-48d5-5d58-9928-6d29437db91e] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

_require at .\loading.jl:929
require at .\loading.jl:858
#2 at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\Distributed.jl:77
#116 at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\process_messages.jl:276
run_work_thunk at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\process_messages.jl:56
run_work_thunk at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\process_messages.jl:65
#102 at .\task.jl:259
#remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\remotecall.jl:421
remotecall_wait(::Function, ::Distributed.Worker) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\remotecall.jl:412
#remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\remotecall.jl:433
remotecall_wait(::Function, ::Int64) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\remotecall.jl:433
(::getfield(Distributed, Symbol("##1#3")){Base.PkgId})() at .\task.jl:259

...and 1 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at .\task.jl:226
 [2] macro expansion at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\Distributed.jl:75 [inlined]
 [3] macro expansion at .\task.jl:244 [inlined]
 [4] _require_callback(::Base.PkgId) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\Distributed.jl:74
 [5] #invokelatest#1 at .\essentials.jl:697 [inlined]
 [6] invokelatest at .\essentials.jl:696 [inlined]
 [7] require(::Base.PkgId) at .\loading.jl:861
 [8] macro expansion at .\logging.jl:309 [inlined]
 [9] require(::Module, ::Symbol) at .\loading.jl:840

Here’s my version info:

julia> versioninfo()
Julia Version 1.0.2
Commit d789231e99 (2018-11-08 20:11 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-6300U CPU @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 1

As far as I can tell I’m following the documentation from the manual. Should I submit an issue to Distributed and/or OnlineStats?

4

[quote=“ajkeith, post:3, topic:19851”]
Pkg.activate(“C:\Users\op\Documents\Julia Projects\StrategyGames”) [/quote]

What if you add

@everywhere using Pkg
@everywhere Pkg.activate("C:\\Users\\op\\Documents\\Julia Projects\\StrategyGames")

after using Distributed; addprocs(2) ?

I think the workers do not have to right project activated.

5

I’ll try in a bit, but I’m pretty sure that Distributions is in the same project and not in the StdLib

6

Thanks @anon43133060! That fixed it.

Earlier I had tried using

@everywhere using Pkg; Pkg.activate("C:\\Users\\op\\Documents\\Julia Projects\\StrategyGames")

But clearly, in hindsight, that means I wasn’t applying the @everywhere macro to the Pkg.activate command after the semicolon.

7

Relevant issue with other workarounds here:
https://github.com/JuliaLang/julia/issues/28781