# Distributed parallelism within packages/applications

**URL:** <https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320>\
**Category:** Julia at Scale\
**Tags:** question\
**Created:** [August 30, 2018, 2:19pm UTC](https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320 "2018-08-30T14:19:37Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![SebastianM-C](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sebastianm-c/32/2480_2.png) [@SebastianM-C](https://discourse.julialang.org/u/SebastianM-C)\
**Post date:** [August 30, 2018, 2:19pm UTC](https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320/1 "2018-08-30T14:19:38Z")

</div>

How can I use parallelism (on multiple processes) in a julia application/package?  
I have an application (in the Pkg3 sense that is not a package) in which I use `pmap`.  
I tried to load the package with `using ParallelTest` after doing

```julia
using Distributed
addprocs(2)

```

but I get an error saying that the package is not installed.

Steps to reproduce:

```julia
(v0.7) pkg> generate ParallelTest
Generating project ParallelTest:
    ParallelTest\Project.toml
    ParallelTest/src/ParallelTest.jl

shell> cd ParallelTest
C:\Users\memo\Documents\julia\ParallelTest

(v0.7) pkg> activate .

julia> using Distributed

julia> addprocs(2)
2-element Array{Int64,1}:
 2
 3

julia> using ParallelTest
[Info: Precompiling ParallelTest [0e712700-ac5e-11e8-3696-ef47f8f04c4e]
ERROR: On worker 2:
ArgumentError: Package ParallelTest [0e712700-ac5e-11e8-3696-ef47f8f04c4e] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

_require at .\loading.jl:923
require at .\loading.jl:852
#2 at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v0.7\Distributed\src\Distributed.jl:77
#116 at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v0.7\Distributed\src\process_messages.jl:276
run_work_thunk at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v0.7\Distributed\src\process_messages.jl:56
run_work_thunk at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v0.7\Distributed\src\process_messages.jl:65
#102 at .\task.jl:262
#remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v0.7\Distributed\src\remotecall.jl:407
remotecall_wait(::Function, ::Distributed.Worker) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v0.7\Distributed\src\remotecall.jl:398
#remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v0.7\Distributed\src\remotecall.jl:419
remotecall_wait(::Function, ::Int64) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v0.7\Distributed\src\remotecall.jl:419
(::getfield(Distributed, Symbol("##1#3")){Base.PkgId})() at .\task.jl:262

...and 1 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at .\task.jl:229
 [2] macro expansion at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v0.7\Distributed\src\Distributed.jl:75 [inlined]
 [3] macro expansion at .\task.jl:247 [inlined]
 [4] _require_callback(::Base.PkgId) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v0.7\Distributed\src\Distributed.jl:74
 [5] #invokelatest#1 at .\essentials.jl:691 [inlined]
 [6] invokelatest at .\essentials.jl:690 [inlined]
 [7] require(::Base.PkgId) at .\loading.jl:855
 [8] macro expansion at .\logging.jl:311 [inlined]
 [9] require(::Module, ::Symbol) at .\loading.jl:834

```

---

<div class="post-metadata">

**Author:** ![Rudi79](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rudi79/32/3884_2.png) [@Rudi79](https://discourse.julialang.org/u/Rudi79)\
**Post date:** [August 30, 2018, 2:53pm UTC](https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320/2 "2018-08-30T14:53:27Z")

</div>

```julia
using Distributed
addprocs(2)
@everywhere using Pkg
@everywhere Pkg.activate(".")
@everywhere using ParallelTest

```

should do the trick. But I don’t know whether this is the best way to solve your Problem.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [August 30, 2018, 2:55pm UTC](https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320/3 "2018-08-30T14:55:01Z")

</div>

If all you want is use a method from Paralleltest containing `pmap` on your main process, you have to do `using Paralleltest` before adding processes. The second process doesn’t need `Paralleltest` per se, but it needs the function given to it:

```julia
help?> pmap
Search: pmap promote_shape typemax PermutedDimsArray process_messages

  pmap(f, [::AbstractWorkerPool], c...; distributed=true, batch_size=1, on_error=nothing, retry_delays=[], retry_check=nothing) -> collection

  Transform collection c by applying f to each element using available workers and tasks.

  For multiple collection arguments, apply f elementwise.

  Note that f must be made available to all worker processes; see Code Availability and Loading Packages for details.

#<snip>

```

---

<div class="post-metadata">

**Author:** ![SebastianM-C](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sebastianm-c/32/2480_2.png) [@SebastianM-C](https://discourse.julialang.org/u/SebastianM-C)\
**Post date:** [August 30, 2018, 4:04pm UTC](https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320/4 "2018-08-30T16:04:26Z")

</div>

It’s a bit more complicated. I have a function that uses `pmap`, but it depends on a package, so the package should also be available on the workers.

Is there a way to load only some of the files of the package on all the workers?

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [August 30, 2018, 4:30pm UTC](https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320/5 "2018-08-30T16:30:34Z")

</div>

Did you try @Rudi79 code? It should do the trick. There is an open issue on GitHub that should fix this, pending some discussion.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [August 30, 2018, 6:22pm UTC](https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320/6 "2018-08-30T18:22:06Z")

</div>

Not that I know of - you can however load the other packages on the other workers too using `@everywhere using AnotherPackage`.

---

<div class="post-metadata">

**Author:** ![SebastianM-C](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sebastianm-c/32/2480_2.png) [@SebastianM-C](https://discourse.julialang.org/u/SebastianM-C)\
**Post date:** [August 31, 2018, 9:31am UTC](https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320/7 "2018-08-31T09:31:57Z")

</div>

Thanks for the answers!  
I am trying to load some files on all workers, but I am not sure how to do it properly.  
Considering the example in the first post, I changed the contents of `ParallelTest.jl` to:

```julia
__precompile__ (false)

module ParallelTest

using Distributed
@everywhere using Pkg
@everywhere Pkg.activate(".")

@everywhere include("$(@ __DIR__ )/parallel.jl")

@everywhere using .PMod

greet() = print("Hello World!")

end # module

```

and added another file `parallel.jl` in the `src` directory containting:

```julia
module PMod

println("loaded")

end # module PMod

```

I managed to load the module successfully, but I get redefinition warnings for the module loaded in parallel.

```julia
(v0.7) pkg> activate .

julia> using Distributed

julia> addprocs(2)
2-element Array{Int64,1}:
 2
 3

julia> using ParallelTest
[Info: Precompiling ParallelTest [0e712700-ac5e-11e8-3696-ef47f8f04c4e]
loaded
      From worker 3: loaded
      From worker 2: loaded
      From worker 2: WARNING: replacing module PMod.
WARNING: replacing module PMod.
      From worker 2: loadedWARNING: replacing module PMod.

      From worker 3: WARNING: replacing module PMod.
loaded
loaded
      From worker 2: WARNING: replacing module PMod.
      From worker 2: loaded
      From worker 3: loaded
      From worker 3: WARNING: replacing module PMod.
      From worker 3: loaded

```

I also tried to use `using .PMod` instead of `@everywhere using .PMod`, but I get a `LoadError` saying that `PMod` is not defined. I think that the problem is that on the workers it’s in the global scope, while on the master process is inside a module.

---

<div class="post-metadata">

**Author:** ![SebastianM-C](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sebastianm-c/32/2480_2.png) [@SebastianM-C](https://discourse.julialang.org/u/SebastianM-C)\
**Post date:** [August 31, 2018, 9:50am UTC](https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320/8 "2018-08-31T09:50:14Z")

</div>

Can you give a link to the issue so that I can follow?

---

<div class="post-metadata">

**Author:** ![Christian\_Dengler](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/christian_dengler/32/20969_2.png) [@Christian\_Dengler](https://discourse.julialang.org/u/Christian_Dengler)\
**Post date:** [October 2, 2018, 1:03pm UTC](https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320/9 "2018-10-02T13:03:38Z")

</div>

Was there any update on this? I’m failing to load my custom package on the workers ☹ I also tried the thingy recommended

```julia
@everywhere begin
 using Pkg
 Pkg.activate(".")
 using MyPackage
 end

```

without luck. This was definitely easier in 0.6.

---

<div class="post-metadata">

**Author:** ![SebastianM-C](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sebastianm-c/32/2480_2.png) [@SebastianM-C](https://discourse.julialang.org/u/SebastianM-C)\
**Post date:** [October 4, 2018, 3:52pm UTC](https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320/10 "2018-10-04T15:52:59Z")

</div>

I think `using MyPackage` should not be in the `@everywhere` block. What errors do you get?

---

<div class="post-metadata">

**Author:** ![Christian\_Dengler](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/christian_dengler/32/20969_2.png) [@Christian\_Dengler](https://discourse.julialang.org/u/Christian_Dengler)\
**Post date:** [October 5, 2018, 8:14am UTC](https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320/11 "2018-10-05T08:14:53Z")

</div>

I had the same error as stated above:

```julia
ArgumentError: Package ... is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.
e.t.c.

```

but I found the reason just now. The “startup.jl” file is only executed on the main process, not on the workers, so my “LOAD\_PATH” did not include the required directories. On 0.6.4 the “juliarc.jl” was always executed on the workers also. Doing `@everywhere println(LOAD_PATH)` shows this. So now I’ll just need to find out how to get this LOAD\_PATH to all my workers and I should be fine. No need for the `Pkg.activate` and all that stuff.

---

<div class="post-metadata">

**Author:** ![gasagna](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gasagna/32/1275_2.png) [@gasagna](https://discourse.julialang.org/u/gasagna)\
**Post date:** [January 21, 2019, 11:09pm UTC](https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320/12 "2019-01-21T23:09:01Z")

</div>

Bumping this thread, as there does not seem to be a working solution for everyone on this thread (nor for me).

---

<div class="post-metadata">

**Author:** ![egcjm](https://avatars.discourse-cdn.com/v4/letter/e/5fc32e/32.png) [@egcjm](https://discourse.julialang.org/u/egcjm)\
**Post date:** [July 17, 2019, 12:30am UTC](https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320/13 "2019-07-17T00:30:21Z")

</div>

Have you been able to resolve this? I am struggling with the same issue for packages SpecialFunctions and Match… I have tried all suggested remedies (Pks.instantiate etc) to no avail.

---

<div class="post-metadata">

**Author:** ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Post date:** [July 17, 2019, 1:27am UTC](https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320/14 "2019-07-17T01:27:15Z")

</div>

This should work if you split the `using MyPackage` into its own `@everywhere` block (because the macro moves all import statements to the top of the block, breaking your original attempt).

---

<div class="post-metadata">

**Author:** ![egcjm](https://avatars.discourse-cdn.com/v4/letter/e/5fc32e/32.png) [@egcjm](https://discourse.julialang.org/u/egcjm)\
**Post date:** [July 17, 2019, 9:14pm UTC](https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320/15 "2019-07-17T21:14:46Z")

</div>

Thanks for suggestion. Still having problems though, even just getting standard packages like ParallelDataTransfer, Interpolations…Specialfunctions to work – I’m not even attempting my own packages for now.

\<\<left out loading of all packages on master processor – this works without fail\>\>

## 

using Distributed  
addprocs(2) # verified that I now have 3 procs

@everywhere begin  
using Pkg  
Pkg.activate(“.”)  
end

@everywhere using ParallelDataTransfer # FAILS SEE OUTPUT BELOW  
@everywhere using FFTW # OK!  
@everywhere using LinearAlgebra # OK!  
@everywhere using SpecialFunctions # FAILS  
@everywhere using LowRankApprox # FAILS  
@everywhere using Interpolations # FAILS  
@everywhere using Match # FAILS

The message generated for ParallelDataTransfer is

**On worker 2:**  
**ArgumentError: Package ParallelDataTransfer not found in current path:**

 
 - **Run `import Pkg; Pkg.add("ParallelDataTransfer")` to install the ParallelDataTransfer package.  
require at .\loading.jl:823  
eval at .\boot.jl:328  
 #116 at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\process\_messages.jl:276  
run\_work\_thunk at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\process\_messages.jl:56  
run\_work\_thunk at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\process\_messages.jl:65  
 #102 at .\task.jl:259  
 #remotecall\_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:421  
remotecall\_wait(::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:412  
 #remotecall\_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:433  
remotecall\_wait(::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:433  
(::getfield(Distributed, Symbol(“##161#163”)){Module,Expr})() at .\task.jl:259  
…and 1 more exception(s).**  
in top-level scope at stdlib\v1.1\Distributed\src\macros.jl:183  
in remotecall\_eval at stdlib\v1.1\Distributed\src\macros.jl:199  
in macro expansion at base\task.jl:245  
in sync\_end at base\task.jl:226

---

<div class="post-metadata">

**Author:** ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Post date:** [October 4, 2019, 2:30pm UTC](https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320/16 "2019-10-04T14:30:59Z")

</div>

Have you confirmed that `Pkg.activate(".")` actually does what you think? `"."` assumes that your workers started in the directory you expect; maybe try `@everywhere println(pwd())` before trying to load packages to confirm you are where you think you are.