# Alternative to SharedArrays for multi-node cluster?

**URL:** <https://discourse.julialang.org/t/alternative-to-sharedarrays-for-multi-node-cluster/37794>\
**Category:** General Usage\
**Tags:** question\
**Created:** [April 17, 2020, 11:35pm UTC](https://discourse.julialang.org/t/alternative-to-sharedarrays-for-multi-node-cluster/37794 "2020-04-17T23:35:09Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nadou407](https://avatars.discourse-cdn.com/v4/letter/n/f07891/32.png) [@Nadou407](https://discourse.julialang.org/u/Nadou407)\
**Post date:** [April 17, 2020, 11:35pm UTC](https://discourse.julialang.org/t/alternative-to-sharedarrays-for-multi-node-cluster/37794/1 "2020-04-17T23:35:09Z")

</div>

Hello!

I’m moving my codes from my own computer to a multi-node cluster and realized SharedArrays won’t work on multiple nodes without shared memory.

The old code using SharedArrays looks like below:

```julia
using Distributed, ClusterManagers, SharedArrays
Nworker = 100
addprocs(SlurmManager(Nworker), t="20:00:00")

@everywhere Data = CSV.read("Data.csv"; header=false)
@everywhere N = 1_000_000

Result = SharedArray{Float64}(N, 4)

@sync @distributed for i = 1:N
    Result[i,:] = ones(4,1)*Data[i] # for illustration only
    # however, each iteration will be short, 
    # so -pmap- may not be good option here.
end

```

My questions is what will be good alternatives to SharedArrays in this case? If possible, I’d like to keep the @distributed for loop structure to minimize the work of code-rewriting.

I saw DistributedArrays but really didn’t figure out how to use it in my setting. Specifically, how to combine it with @distributed for loop since I don’t control for which part of the iteration goes to which processor. Any suggestions will be greatly appreciated!

---

<div class="post-metadata">

**Author:** ![jishnub](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jishnub/32/33620_2.png) [@jishnub](https://discourse.julialang.org/u/jishnub)\
**Post date:** [April 19, 2020, 3:49am UTC](https://discourse.julialang.org/t/alternative-to-sharedarrays-for-multi-node-cluster/37794/2 "2020-04-19T03:49:21Z")

</div>

You may use the `DistributedArray` constructor to get around these issues, it’s not very much more complicated.

```julia
julia> nworkers()
3

julia> @everywhere using DistributedArrays, OffsetArrays

julia> N = 6 # for demonstration
6

julia> Data = distribute(rand(N)); # using the default distribution at this point

julia> Result = DArray((N,4)) do inds # local indices on each processor
           arr = zeros(inds) # We create an OffsetArray with the correct local indices
           data_local = OffsetArray(localpart(Data),localindices(Data)) # get the local part of data on each processor and assign the proper indices
           for i in inds[1]
               arr[i,:] .= ones(4)*data_local[i]
           end
           parent(arr) # strip the OffsetArray wrapper
       end
6×4 DArray{Float64,2,Array{Float64,2}}:
 0.54454 0.54454 0.54454 0.54454 
 0.285744 0.285744 0.285744 0.285744
 0.678867 0.678867 0.678867 0.678867
 0.62554 0.62554 0.62554 0.62554 
 0.862751 0.862751 0.862751 0.862751
 0.103856 0.103856 0.103856 0.103856
```

---

<div class="post-metadata">

**Author:** ![henry2004y](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henry2004y/32/9284_2.png) [@henry2004y](https://discourse.julialang.org/u/henry2004y)\
**Post date:** [April 19, 2020, 3:39pm UTC](https://discourse.julialang.org/t/alternative-to-sharedarrays-for-multi-node-cluster/37794/3 "2020-04-19T15:39:29Z")

</div>

In my personal experience, `DistributedArrays` works, but the performance is much worse than `SharedArrays`.

---

<div class="post-metadata">

**Author:** ![Nadou407](https://avatars.discourse-cdn.com/v4/letter/n/f07891/32.png) [@Nadou407](https://discourse.julialang.org/u/Nadou407)\
**Post date:** [April 19, 2020, 7:38pm UTC](https://discourse.julialang.org/t/alternative-to-sharedarrays-for-multi-node-cluster/37794/5 "2020-04-19T19:38:01Z")

</div>

> [@jishnub](#):
>
> for i in inds[1]  
> arr[i,:] .= ones(4)\*data\_local[i]  
> end

Thank you so much! Maybe it’s a dumb question: does this for-loop work in parallel over workers or sequential?

---

<div class="post-metadata">

**Author:** ![Nadou407](https://avatars.discourse-cdn.com/v4/letter/n/f07891/32.png) [@Nadou407](https://discourse.julialang.org/u/Nadou407)\
**Post date:** [April 20, 2020, 12:09am UTC](https://discourse.julialang.org/t/alternative-to-sharedarrays-for-multi-node-cluster/37794/6 "2020-04-20T00:09:21Z")

</div>

Thanks! I just adapted the codes provided by @jishnub and found basically the same performance. Maybe it differs from case to case.

---

<div class="post-metadata">

**Author:** ![Nadou407](https://avatars.discourse-cdn.com/v4/letter/n/f07891/32.png) [@Nadou407](https://discourse.julialang.org/u/Nadou407)\
**Post date:** [April 20, 2020, 12:10am UTC](https://discourse.julialang.org/t/alternative-to-sharedarrays-for-multi-node-cluster/37794/7 "2020-04-20T00:10:37Z")

</div>

I actually figured out. It’s sequential for the block within each processor.

Thank you so much for the help. I now adapted it to my codes and it works perfect!

---

<div class="post-metadata">

**Author:** ![tk3369](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tk3369/32/2824_2.png) [@tk3369](https://discourse.julialang.org/u/tk3369)\
**Post date:** [April 20, 2020, 1:40am UTC](https://discourse.julialang.org/t/alternative-to-sharedarrays-for-multi-node-cluster/37794/8 "2020-04-20T01:40:31Z")

</div>

That is expected though because data needs to be transferred to the remote worker nodes for processing, whereas SharedArray is shared by all worker nodes that reside in the same node.

---

<div class="post-metadata">

**Author:** ![henry2004y](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henry2004y/32/9284_2.png) [@henry2004y](https://discourse.julialang.org/u/henry2004y)\
**Post date:** [April 20, 2020, 5:21am UTC](https://discourse.julialang.org/t/alternative-to-sharedarrays-for-multi-node-cluster/37794/9 "2020-04-20T05:21:28Z")

</div>

Despite of the reason you mentioned, I think still the performance is nowhere closer to, say, MPI implementation. I don’t know the details behind though.

---

<div class="post-metadata">

**Author:** ![jishnub](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jishnub/32/33620_2.png) [@jishnub](https://discourse.julialang.org/u/jishnub)\
**Post date:** [April 20, 2020, 5:25am UTC](https://discourse.julialang.org/t/alternative-to-sharedarrays-for-multi-node-cluster/37794/10 "2020-04-20T05:25:15Z")

</div>

If by MPI you mean repeated data transfer between cores then yes, Julia does have a performance bottleneck in this, as the arrays are not pre-allocated unlike MPI. This case is different as there’s not much transfer involved. I believe the data-transfer lags might be mitigated to some extent with [ArrayChannels.jl](https://github.com/rohanmclure/ArrayChannels.jl), but I haven’t used it myself.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [April 20, 2020, 8:02am UTC](https://discourse.julialang.org/t/alternative-to-sharedarrays-for-multi-node-cluster/37794/11 "2020-04-20T08:02:55Z")

</div>

I Admit my ignorance here… how are the arrays copied between the severs?  
I know - read the source!  
I would start to use the term RDMA here - which woudl put me int he firing line for investigating if this is feasible for DistributedArrays so I will not!