# Problems parallelizing declaring used packages
**URL:** https://discourse.julialang.org/t/problems-parallelizing-declaring-used-packages/27728
**Category:** New to Julia
**Tags:** package, pkg, parallel, distributed
**Created:** [August 19, 2019, 7:41pm UTC](https://discourse.julialang.org/t/problems-parallelizing-declaring-used-packages/27728 "2019-08-19T19:41:36Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@jmcastro2109](https://discourse.julialang.org/u/jmcastro2109)
#### Post date: [August 19, 2019, 7:41pm UTC](https://discourse.julialang.org/t/problems-parallelizing-declaring-used-packages/27728/1 "2019-08-19T19:41:36Z")
Hi! I am having big trouble setting up a code for parallelization. When I try to “declare” the packages that I am using in the workers by running the following code:
```julia
#--------------------------------#
# Initialization #
#--------------------------------#
# Packages
#using Distributed
using QuantEcon, Optim, Distributions, DelimitedFiles, ExcelReaders
using ProgressMeter, BenchmarkTools, DataFrames, Combinatorics
using LinearAlgebra, Statistics, Random, StatsBase
using BlackBoxOptim, Distributed
# Number of cores/workers
addprocs(2)
@everywhere begin
using Pkg; Pkg.activate(".") # required
using QuantEcon, Optim, Distributions, DelimitedFiles, ExcelReaders
using ProgressMeter, BenchmarkTools, DataFrames, Combinatorics
using LinearAlgebra, Statistics, Random, StatsBase
using BlackBoxOptim, Distributed
end
# Set Directory
cd("XXX\\julia_codes")
```
I get the following error:
```julia
On worker 2:
ArgumentError: Package QuantEcon not found in current path:
- Run `import Pkg; Pkg.add("QuantEcon")` to install the QuantEcon package.
require at .\loading.jl:823
top-level scope at XXX\julia_codes\4-Simulations.jl:26
eval at .\boot.jl:328
#116 at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\process_messages.jl:276
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\process_messages.jl:56
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\process_messages.jl:65
#102 at .\task.jl:259
#remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:421
remotecall_wait(::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:412
#remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:433
remotecall_wait(::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:433
(::getfield(Distributed, Symbol("##161#163")){Module,Expr})() at .\task.jl:259
...and 7 more exception(s).
in top-level scope at [stdlib\v1.1\Distributed\src\macros.jl:183](#)
in remotecall_eval at [stdlib\v1.1\Distributed\src\macros.jl:199](#)
in macro expansion at [base\task.jl:245](#)
in sync_end at [base\task.jl:226](#)
```
---
### Author:  [@Pbellive](https://discourse.julialang.org/u/Pbellive)
#### Post date: [August 19, 2019, 9:21pm UTC](https://discourse.julialang.org/t/problems-parallelizing-declaring-used-packages/27728/2 "2019-08-19T21:21:06Z")
Hi,
The issue here is that your worker processes aren’t running in the same environment as the main process. By default new workers will launch in the default environment. For some reason (on julia 1.1.1, linux)
```julia
@everywhere Pkg.activate(".")
```
doesn’t seem to activate the current directory’s project on the workers. ~~I’m not sure if that’s a bug or the intended behaviour.~~
Edit: The fact that
```julia
@everywhere activate(".")
```
doesn’t work is by design. There is an open issue ([Workers should inherit Pkg environment · Issue #28781 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/28781)) discussing why that is the case and how the situation can be improved. Seems like the workaround I describe below (which I came across on another discourse thread: [Packages and workers - #9 by teored90](https://discourse.julialang.org/t/packages-and-workers/14072/9)) is still the way to go.
In any event, you can get the workers to launch in the environment defined in the current directory’s Project.toml file by passing the `exeflags` keyword argument to `addprocs, like so:
```julia
addprocs(2, exeflags="--project=.")
```
With the workers launched like that I’m able to load packages on the workers that are only installed in the current environment.
---
### Author:  [@jmcastro2109](https://discourse.julialang.org/u/jmcastro2109)
#### Post date: [August 20, 2019, 1:57pm UTC](https://discourse.julialang.org/t/problems-parallelizing-declaring-used-packages/27728/3 "2019-08-20T13:57:31Z")
Thanks for the reply. It still does not work form me. I am only replacing my addprocs line for the one you suggested and deleting the Pkg.activate(). Is that correct?
---
### Author:  [@Pbellive](https://discourse.julialang.org/u/Pbellive)
#### Post date: [August 20, 2019, 9:36pm UTC](https://discourse.julialang.org/t/problems-parallelizing-declaring-used-packages/27728/4 "2019-08-20T21:36:05Z")
Hmmm. Are you also activating the right environment on the master process? You’ll either have to start julia with the `--project=.` flag. E.g. from the shell run
```julia
$ julia --project=.
```
or start julia as normal and run
```julia
using Pkg; Pkg.activate(".")
```
on the master process before loading your packages.
---
### Author:  [@jmcastro2109](https://discourse.julialang.org/u/jmcastro2109)
#### Post date: [August 21, 2019, 6:28pm UTC](https://discourse.julialang.org/t/problems-parallelizing-declaring-used-packages/27728/5 "2019-08-21T18:28:44Z")
I am exactly doing that and it does not work. I am so puzzled! This is my code
```julia
using Pkg; Pkg.activate(".")
using Distributed
using QuantEcon, Optim, Distributions, DelimitedFiles, ExcelReaders
using ProgressMeter, BenchmarkTools, DataFrames, Combinatorics
using LinearAlgebra, Statistics, Random, StatsBase
using BlackBoxOptim, BitOperations
#Number of cores/workers
addprocs(2, exeflags="--project=.")
#Initializing packaes in workers
@everywhere begin
using QuantEcon, Optim, Distributions, DelimitedFiles, ExcelReaders
using ProgressMeter, BenchmarkTools, DataFrames, Combinatorics
using LinearAlgebra, Statistics, Random, StatsBase
using BlackBoxOptim, BitOperations
end
```
and the error message I get
**On worker 2:**
```julia
ArgumentError: Package QuantEcon not found in current path:
- Run `import Pkg; Pkg.add("QuantEcon")` to install the QuantEcon package.
require at .\loading.jl:823
top-level scope at C:\XXX\julia_codes\6-Simulations_par.jl:30
eval at .\boot.jl:328
#116 at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\process_messages.jl:276
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\process_messages.jl:56
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\process_messages.jl:65
#102 at .\task.jl:259
#remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:421
remotecall_wait(::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:412
#remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:433
remotecall_wait(::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:433
(::getfield(Distributed, Symbol("##161#163")){Module,Expr})() at .\task.jl:259
...and 1 more exception(s).
in top-level scope at [stdlib\v1.1\Distributed\src\macros.jl:183](#)
in remotecall_eval at [stdlib\v1.1\Distributed\src\macros.jl:199](#)
in macro expansion at [base\task.jl:245](#)
in sync_end at [base\task.jl:226](#)
```
---
### Author:  [@Pbellive](https://discourse.julialang.org/u/Pbellive)
#### Post date: [August 21, 2019, 6:43pm UTC](https://discourse.julialang.org/t/problems-parallelizing-declaring-used-packages/27728/6 "2019-08-21T18:43:05Z")
Henh. That is very strange. I’m sorry, I’m out of ideas. Not sure what’s happening on your machine.
---
### Author:  [@jmcastro2109](https://discourse.julialang.org/u/jmcastro2109)
#### Post date: [August 21, 2019, 6:49pm UTC](https://discourse.julialang.org/t/problems-parallelizing-declaring-used-packages/27728/7 "2019-08-21T18:49:25Z")
Thanks a lot anyways!
---
### Author:  [@jmcastro2109](https://discourse.julialang.org/u/jmcastro2109)
#### Post date: [August 21, 2019, 10:17pm UTC](https://discourse.julialang.org/t/problems-parallelizing-declaring-used-packages/27728/8 "2019-08-21T22:17:46Z")
Does someone have an idea about this issue? @ksmcreynolds
---
### Author:  [@jmcastro2109](https://discourse.julialang.org/u/jmcastro2109)
#### Post date: [August 22, 2019, 1:50pm UTC](https://discourse.julialang.org/t/problems-parallelizing-declaring-used-packages/27728/9 "2019-08-22T13:50:52Z")
@Pbellive can you please tell me in which order would you recommend running the above code?
---
### Author:  [@00vareladavid](https://discourse.julialang.org/u/00vareladavid)
#### Post date: [August 22, 2019, 11:05pm UTC](https://discourse.julialang.org/t/problems-parallelizing-declaring-used-packages/27728/10 "2019-08-22T23:05:49Z")
Have you tried something like this:
```julia
@everywhere begin
using Pkg; Pkg.activate(path_to_env)
....
end
```
Edit: Also, in case it is not obvious, `Pkg.activate(".")` depends on the working directory, so it will not activate the same environment in every case.
---
### Author:  [@greg\_plowman](https://discourse.julialang.org/u/greg_plowman)
#### Post date: [August 23, 2019, 3:16am UTC](https://discourse.julialang.org/t/problems-parallelizing-declaring-used-packages/27728/11 "2019-08-23T03:16:51Z")
To help diagnose your issue, you might compare the output of Base.load\_path() on master vs worker:
```julia
Base.load_path()
remotecall_fetch(Base.load_path, 2)
```
---
### Author:  [@jmcastro2109](https://discourse.julialang.org/u/jmcastro2109)
#### Post date: [August 23, 2019, 1:51pm UTC](https://discourse.julialang.org/t/problems-parallelizing-declaring-used-packages/27728/12 "2019-08-23T13:51:13Z")
Thanks for the reply:
I am running
```julia
using Pkg; Pkg.activate(".")
using Distributed
addprocs(2, exeflags="--project=.")
```
when I compare the outputs of Base.load\_path() I get
```julia
Base.load_path()
"C:\\Users\\jmcastro\\Project.toml"
"C:\\Users\\jmcastro\\.juliapro\\JuliaPro_v1.1.1.1\\environments\\v1.1\\Project.toml"
"C:\\Users\\jmcastro\\AppData\\Local\\JuliaPro-1.1.1.1\\Julia-1.1.1\\share\\julia\\stdlib\\v1.1"
remotecall_fetch(Base.load_path, 2)
"C:\\Users\\jmcastro\\Project.toml"
"C:\\Users\\jmcastro\\.julia\\environments\\v1.1\\Project.toml"
"C:\\Users\\jmcastro\\AppData\\Local\\JuliaPro-1.1.1.1\\Julia-1.1.1\\share\\julia\\stdlib\\v1.1"
```
then, what differs is the second path in each case.
Finally, when I’ll do
```julia
using QuantEcon, Optim, Distributions, DelimitedFiles, ExcelReaders
using ProgressMeter, BenchmarkTools, DataFrames, Combinatorics
using LinearAlgebra, Statistics, Random, StatsBase
using BlackBoxOptim, BitOperations
```
I get the error
```julia
On worker 2:
ArgumentError: Package ExcelReaders [c04bee98-12a5-510c-87df-2a230cb6e075] is required but does not seem to be installed:
- Run `Pkg.instantiate()` to install all recorded dependencies.
_require at .\loading.jl:929
require at .\loading.jl:858
#2 at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\Distributed.jl:77
#116 at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\process_messages.jl:276
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\process_messages.jl:56
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\process_messages.jl:65
#102 at .\task.jl:259
#remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:421
remotecall_wait(::Function, ::Distributed.Worker) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:412
#remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:433
remotecall_wait(::Function, ::Int64) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:433
(::getfield(Distributed, Symbol("##1#3")){Base.PkgId})() at .\task.jl:259
```
---
### Author:  [@jmcastro2109](https://discourse.julialang.org/u/jmcastro2109)
#### Post date: [August 23, 2019, 7:18pm UTC](https://discourse.julialang.org/t/problems-parallelizing-declaring-used-packages/27728/13 "2019-08-23T19:18:59Z")
Is it possible that this issue is related somehow to JuliaPro?
---
### Author:  [@bernhard](https://discourse.julialang.org/u/bernhard)
#### Post date: [August 23, 2019, 8:25pm UTC](https://discourse.julialang.org/t/problems-parallelizing-declaring-used-packages/27728/14 "2019-08-23T20:25:44Z")
It is quite strange that you get an error from worker 2 when it seems you have simply executed using statements (without everywhere).
Is that so?
I tend to run all using processes on the main proc (as this triggers precomp).
Then add ONE worker.
Then do „everywhere“ using.
Have you tried that?
Also, did you run Pkg.instantiate on the main process? (After pkg.activate)
---
### Author:  [@greg\_plowman](https://discourse.julialang.org/u/greg_plowman)
#### Post date: [August 24, 2019, 12:32am UTC](https://discourse.julialang.org/t/problems-parallelizing-declaring-used-packages/27728/15 "2019-08-24T00:32:55Z")
> [@jmcastro2109](#):
>
> Is it possible that this issue is related somehow to JuliaPro?
Most likely, especially since `load_path()` on workers is different from master process.
See [Load modules in several workers - #19 by greg\_plowman](https://discourse.julialang.org/t/load-modules-in-several-workers/17521/19)