Add Package on every processor?

Specific Domains Julia at Scale
1

I’m trying to use DistributedArrays (per the example), but it seems like the package needs to be added to every processor. Is this right?

When I first start Julia, and run

using Distributed
addprocs()
@everywhere using DistributedArrays

it tells me

On worker 2:
ArgumentError: Package DistributedArrays [aaf54ef3-cdf8-58ed-94cc-d582ad619b94] is required but does not seem to be installed:

  • Run Pkg.instantiate() to install all recorded dependencies.

_require at .\loading.jl:982
require at .\loading.jl:911
#2 at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\Distributed.jl:78
#116 at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\process_messages.jl:307
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\process_messages.jl:79
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\process_messages.jl:88
#102 at .\task.jl:268
#remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_wait), ::Function, ::Distributed.Worker) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\remotecall.jl:421
remotecall_wait(::Function, ::Distributed.Worker) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\remotecall.jl:412
#remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_wait), ::Function, ::Int64) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\remotecall.jl:433
remotecall_wait at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\remotecall.jl:433 [inlined]
(::getfield(Distributed, Symbol(“##1#3”)){Base.PkgId,Int64})() at .\task.jl:268

…and 3 more exception(s).

sync_end(::Array{Any,1}) at task.jl:235
_require_callback(::Base.PkgId) at task.jl:254
#invokelatest#1 at essentials.jl:790 [inlined]
invokelatest at essentials.jl:789 [inlined]
require(::Base.PkgId) at loading.jl:914
require(::Module, ::Symbol) at loading.jl:906
top-level scope at macros.jl:181

If I run the the last line

@everywhere using DistributedArrays

a second time (even without calling Pkg.instantiate()) it then tells me

On worker 2:
ArgumentError: Package DistributedArrays not found in current path:

  • Run import Pkg; Pkg.add("DistributedArrays") to install the DistributedArrays package.

require at .\loading.jl:876
eval at .\boot.jl:330
#116 at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\process_messages.jl:307
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\process_messages.jl:79
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\process_messages.jl:88
#102 at .\task.jl:268
#remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_wait), ::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\remotecall.jl:421
remotecall_wait(::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\remotecall.jl:412
#remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_wait), ::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\remotecall.jl:433
remotecall_wait at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\remotecall.jl:433 [inlined]
(::getfield(Distributed, Symbol(“##161#163”)){Module,Expr,Int64})() at .\task.jl:268

…and 7 more exception(s).

sync_end(::Array{Any,1}) at task.jl:235
macro expansion at task.jl:254 [inlined]
remotecall_eval(::Module, ::Array{Int64,1}, ::Expr) at macros.jl:199
top-level scope at macros.jl:183

EDIT: Okay. So if I don’t include addprocs(4), then it works??

using Distributed
using DistributedArrays
@everywhere using DistributedArrays

Doesn’t produce any errors. Why is this?

1 Like
2

No, you shouldn’t add packages on remote worker processes. You’re likely getting the error you’re getting because the workers are being launched with a different environment to your master process. For example, if you launch julia with the flag --project <project-file> and then add worker processes on the same machine with addprocs(n) then the workers will launch in the machine’s default environment. So if you launch the master process with a project file that includes DistributedArrays and your default environment does not have DistributedArrays installed, then you’ll see the error you’re seeing. You can pass the exeflags keyword argument to addprocs to insure your workers are using the same project file as your master process.

If you’re using Juno or JuliaPro it could be that these tools are doing something to your environment that’s causing this. That’s total speculation though. I don’t use julia pro or Juno.

Also, from your stacktrace it looks like you’re on Windows. Maybe this is Windows specific? On linux, launching julia from the command line in my default environment, I can do the following successfully:

using Distributed
addprocs()
@everywhere using DistributedArrays
3

Yes, I’m using JuliaPro on Windows. When I open the julia command application and do what you have, this is what happens.

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.2.0 (2019-08-20)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Distributed

julia> addprocs()
8-element Array{Int64,1}:
 2
 3
 4
 5
 6
 7
 8
 9

julia> @everywhere using DistributedArrays
ERROR: On worker 2:
ArgumentError: Package DistributedArrays [aaf54ef3-cdf8-58ed-94cc-d582ad619b94] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

_require at .\loading.jl:982
require at .\loading.jl:911
#2 at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\Distributed.jl:78
#116 at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\process_messages.jl:307
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\process_messages.jl:79
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\process_messages.jl:88
#102 at .\task.jl:268
#remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_wait), ::Function, ::Distributed.Worker) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\remotecall.jl:421
remotecall_wait(::Function, ::Distributed.Worker) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\remotecall.jl:412
#remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_wait), ::Function, ::Int64) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\remotecall.jl:433
remotecall_wait at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\remotecall.jl:433 [inlined]
(::getfield(Distributed, Symbol("##1#3")){Base.PkgId,Int64})() at .\task.jl:268

...and 7 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at .\task.jl:235
 [2] _require_callback(::Base.PkgId) at .\task.jl:254
 [3] #invokelatest#1 at .\essentials.jl:790 [inlined]
 [4] invokelatest at .\essentials.jl:789 [inlined]
 [5] require(::Base.PkgId) at .\loading.jl:914
 [6] require(::Module, ::Symbol) at .\loading.jl:906
 [7] top-level scope at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\macros.jl:181

julia> @everywhere using DistributedArrays
ERROR: On worker 2:
ArgumentError: Package DistributedArrays not found in current path:
- Run `import Pkg; Pkg.add("DistributedArrays")` to install the DistributedArrays package.

require at .\loading.jl:876
eval at .\boot.jl:330
#116 at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\process_messages.jl:307
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\process_messages.jl:79
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\process_messages.jl:88
#102 at .\task.jl:268
#remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_wait), ::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\remotecall.jl:421
remotecall_wait(::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\remotecall.jl:412
#remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_wait), ::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\remotecall.jl:433
remotecall_wait at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\remotecall.jl:433 [inlined]
(::getfield(Distributed, Symbol("##161#163")){Module,Expr,Int64})() at .\task.jl:268

...and 7 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at .\task.jl:235
 [2] macro expansion at .\task.jl:254 [inlined]
 [3] remotecall_eval(::Module, ::Array{Int64,1}, ::Expr) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\macros.jl:199
 [4] top-level scope at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Distributed\src\macros.jl:183

julia> Pkg.instantiate()
  Updating registry at `C:\Users\User\.juliapro\JuliaPro_v1.2.0-1\registries\JuliaPro`
  Updating git-repo `https://pkg.juliacomputing.com//registry/JuliaPro`
fatal: ArgumentException encountered.
   Value cannot be null.
Parameter name: Username
Parameter name: operationArguments

julia>

The last error looks like they’re trying to log into juliacomputing.com. I log into that and it downloads a token.toml file for me. But this file tells me

• token.toml , this file is required only if you’re using a Linux machine
that does not have GUI, instructions to download this file can be found in
section Using JuliaPro in a non-GUI environment

But regardless, I follow the instructions to add the token file, but when I replace the one I already have, it still gives me the same error if I try to instantiate it again.

4

How do you use julia?

5

Ok. I imagine this is JuliaPro related. I’m not sure how to fix it but I recommend taking a look at the links listed in this thread. Seems like others have had issues loading packages on remote worker processes when using JuliaPro.

To answer your other question:

I download the binaries from julialang.org/download. Every so often I’ll also clone the current master branch of the julia git repository and build it from source to see what’s happening at the bleeding edge, maybe poke around Base and the standard libs a bit.

As for running code, I tend to use jupyter notebooks for more exploratory stuff. Otherwise I just launch julia from the command line and run scripts using include. I use vscode with the julia extension for editing.

1 Like
6

Thanks for the link. It seems to be the same issue I have, but removing julia and atom reinstalling juliapro didn’t fix it.

I did however find something that gets around this error on JuliaBox. I found a tutorial that included the extra line

@sync @everywhere workers() include("/opt/julia-1.0/etc/julia/startup.jl") # Needed just for JuliaBox

and I tweaked it to add

@sync @everywhere workers() include("C:\\Users\\User\\AppData\\Local\\JuliaPro-1.2.0-1\\Julia-1.2.0\\etc\\julia\\startup.jl")

to my file in JuliaPro and it seems to have worked.

7

Had the same problem on 1.3.1, starting in a local environment.
@Pbellive seems to have the clue: with plain addproc(n), the workers start in the default environment and don’t see the added package.

Two fixes:

a) add the package also to the default environment

b) addproc( [(“localhost”,n)], exename=“julia --project=…” dir=“…”)

Both worked for me.