# Parallel computing, sharing data with all workers

**URL:** <https://discourse.julialang.org/t/parallel-computing-sharing-data-with-all-workers/23907>\
**Category:** General Usage\
**Created:** [May 6, 2019, 11:36am UTC](https://discourse.julialang.org/t/parallel-computing-sharing-data-with-all-workers/23907 "2019-05-06T11:36:07Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![bernhard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bernhard/32/2619_2.png) [@bernhard](https://discourse.julialang.org/u/bernhard)\
**Post date:** [May 6, 2019, 11:36am UTC](https://discourse.julialang.org/t/parallel-computing-sharing-data-with-all-workers/23907/1 "2019-05-06T11:36:07Z")

</div>

How can I make some data (e.g. an array or a dataframe) available to all data?  
I tried using ParallelDatatrasfer.jl but that is not working (see my issue here [https://github.com/ChrisRackauckas/ParallelDataTransfer.jl/issues/16](https://github.com/ChrisRackauckas/ParallelDataTransfer.jl/issues/16) )

In the example below, assume I want to perform an expensive calculation on vvvec (on serveral workers in parallel).

```julia
using Distributed
using ParallelDataTransfer
addprocs(3)
@everywhere using Random

vvvec=[2,3,1]
sendto(workers(),vvvec=vvvec)
@everywhere vvvec.^2

```

---

<div class="post-metadata">

**Author:** ![bernhard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bernhard/32/2619_2.png) [@bernhard](https://discourse.julialang.org/u/bernhard)\
**Post date:** [May 10, 2019, 5:59am UTC](https://discourse.julialang.org/t/parallel-computing-sharing-data-with-all-workers/23907/2 "2019-05-10T05:59:16Z")

</div>

bump.  
Is that a difficult thing to do, or is my question not well formulated?  
In my view it is a pretty common use case, e.g. to work on a large piece of data (maybe read from a CSV, or generated by some piece of code) with different parameter settings (maybe model parameters). Thus I often want to share data created on the main worker with all other workers.

---

<div class="post-metadata">

**Author:** ![jonathanBieler](https://avatars.discourse-cdn.com/v4/letter/j/82dd89/32.png) [@jonathanBieler](https://discourse.julialang.org/u/jonathanBieler)\
**Post date:** [May 10, 2019, 8:24am UTC](https://discourse.julialang.org/t/parallel-computing-sharing-data-with-all-workers/23907/3 "2019-05-10T08:24:12Z")

</div>

I think you are looking for `DistributedArrays`:

[https://juliaparallel.github.io/DistributedArrays.jl/latest/](https://juliaparallel.github.io/DistributedArrays.jl/latest/)

```julia
using Distributed
addprocs(3)
@everywhere using DistributedArrays, Distributed

@everywhere f(x) = x * myid()
data = distribute([1,1,1,1])

julia> f.(data)
4-element DArray{Int64,1,Array{Int64,1}}:
 2
 2
 3
 4

```

As you can see the first two elements are processed on worker 2, and the 3 and 4th on worker 3 and 4.

Otherwise there’s also `SharedArrays`.

---

<div class="post-metadata">

**Author:** ![platawiec](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/platawiec/32/31914_2.png) [@platawiec](https://discourse.julialang.org/u/platawiec)\
**Post date:** [May 10, 2019, 1:40pm UTC](https://discourse.julialang.org/t/parallel-computing-sharing-data-with-all-workers/23907/4 "2019-05-10T13:40:18Z")

</div>

I have found the interpolation syntax useful for this:  
[https://docs.julialang.org/en/v1/stdlib/Distributed/index.html#Distributed.@everywhere](https://docs.julialang.org/en/v1/stdlib/Distributed/index.html#Distributed.@everywhere)

```julia
foo = 1
@everywhere bar = $foo

```

so `foo` can point to data loaded on your main worker, maybe even unique to its filesystem, and then get loaded everywhere else under `bar`