# How to efficiently handle parallelism on a DistributedArray
**URL:** https://discourse.julialang.org/t/how-to-efficiently-handle-parallelism-on-a-distributedarray/81098
**Category:** Performance
**Tags:** performance, parallel, distributed, loops
**Created:** [May 15, 2022, 1:27pm UTC](https://discourse.julialang.org/t/how-to-efficiently-handle-parallelism-on-a-distributedarray/81098 "2022-05-15T13:27:32Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@ultrapoci](https://discourse.julialang.org/u/ultrapoci)
#### Post date: [May 15, 2022, 1:27pm UTC](https://discourse.julialang.org/t/how-to-efficiently-handle-parallelism-on-a-distributedarray/81098/1 "2022-05-15T13:27:32Z")
I’m working with a `DArray` from DistributedArrays.jl. I want each process to work on its own part of the array independently, and access the rest of the array only when it needs a nearest neighbor which doesn’t reside in its local part of the array.
The way I’m currently handling this is with this function:
```julia
with_workers(f, args...; procs = workers()) = @sync for w in procs
@spawnat w f(args...)
end
```
The idea is that I combine `with_workers` with the `do end` statement:
```julia
with_workers() do
#=
Do stuff with the local array.
Here, a worker may access a cell of the DArray
which resides on another process
=#
end
```
My question is: is this an efficient way of handling parallelism? I cannot use `@distributed` or `pmap`, because I explicitly need each worker to work in parallel on its own part of the array, and these two methods would simply call a random worker to run a function. Also, the algorithm basically updates the array, so each worker needs that every other worker has finished its job to be able to work on the next update.
I’m asking this because my algorithm seems to be a bit slow, and I’m not sure if I’m doing something wrong with parallelism and DistributedArrays. [Here](https://github.com/ultrapoci/Tesi) is my Github repo in case you want to check out the entire project: [Lattice.jl](https://github.com/ultrapoci/Tesi/blob/main/src/DistributedQCD/src/Lattice.jl) deals with the construction of the lattice, and [CabibboMarinari.jl](https://github.com/ultrapoci/Tesi/blob/main/src/DistributedQCD/src/CabibboMarinari.jl) deals with the updating algorithm itself.
---
### Author:  [@deltaeecs](https://discourse.julialang.org/u/deltaeecs)
#### Post date: [May 20, 2022, 1:57pm UTC](https://discourse.julialang.org/t/how-to-efficiently-handle-parallelism-on-a-distributedarray/81098/2 "2022-05-20T13:57:09Z")
I have the same problem and I’m looking for ways to solve it,too. `@distributed` and `pmap` are insuitable for our needs, actually in my case them donot speedup at all.
---
### Author:  [@ultrapoci](https://discourse.julialang.org/u/ultrapoci)
#### Post date: [May 20, 2022, 2:51pm UTC](https://discourse.julialang.org/t/how-to-efficiently-handle-parallelism-on-a-distributedarray/81098/3 "2022-05-20T14:51:15Z")
I also have the feeling that Distributed.jl is too slow, but I don’t know if I’m doing something wrong or it’s just not useful unless you have really big arrays.
---
### Author:  [@deltaeecs](https://discourse.julialang.org/u/deltaeecs)
#### Post date: [May 23, 2022, 6:54am UTC](https://discourse.julialang.org/t/how-to-efficiently-handle-parallelism-on-a-distributedarray/81098/4 "2022-05-23T06:54:20Z")
Actually I find that there are 2 most time consuming parts in my work:
- spawning the tasks, because args passed to function are large in my task;
- passing the parts of array SubDArray, which is inefficient as mentioned in the source code of DistributedArray [DistributedArrays.jl/darray.jl at master · JuliaParallel/DistributedArrays.jl · GitHub](https://github.com/JuliaParallel/DistributedArrays.jl/blob/master/src/darray.jl) at line 819.
---
### Author:  [@ultrapoci](https://discourse.julialang.org/u/ultrapoci)
#### Post date: [May 23, 2022, 9:00am UTC](https://discourse.julialang.org/t/how-to-efficiently-handle-parallelism-on-a-distributedarray/81098/5 "2022-05-23T09:00:23Z")
So there’s really no solution at the moment, right? Spawning tasks and setting an array element are necessary operations, and I’m not sure there’s a more efficient way to the them. Maybe allocating a local array and then assigning it to the local portion of the DArray is more efficient than accessing the DArray everytime to set a single element?
---
### Author:  [@kfrb](https://discourse.julialang.org/u/kfrb)
#### Post date: [May 23, 2022, 10:47am UTC](https://discourse.julialang.org/t/how-to-efficiently-handle-parallelism-on-a-distributedarray/81098/6 "2022-05-23T10:47:26Z")
@ultrapoci and @deltaeecs, I had a similar problem and solved it using multithreading. See the old thread for an MWE:
> [@Performance issues with parallel Julia code](https://discourse.julialang.org/t/performance-issues-with-parallel-julia-code/68919/3):
>
> Thank you very much @goerch for your detailed comments and helpful information, which helped me to improve the performance significantly! However, after further research, I decided to also give multithreading a try, as it seems to be better suited for this use case. A slightly better simulation-adapted MWE looks like the following.
>
> > **MWE Multithreading**
> >
> > using Base.Threads, LinearAlgebra, Test, BenchmarkTools abstract type AbstractModel end struct Model \<: AbstractModel idx::Vector{Tuple{I…
---
### Author:  [@deltaeecs](https://discourse.julialang.org/u/deltaeecs)
#### Post date: [May 23, 2022, 11:06am UTC](https://discourse.julialang.org/t/how-to-efficiently-handle-parallelism-on-a-distributedarray/81098/7 "2022-05-23T11:06:48Z")
Yes, multi-threading is wonderful and I have a multi-threading version of my program. But my program will need serval T Bytes memory and thousands cores in application, which must be run on clusters. Therefore, blabla…, anyway thx! 😃
---
### Author:  [@deltaeecs](https://discourse.julialang.org/u/deltaeecs)
#### Post date: [May 24, 2022, 2:13am UTC](https://discourse.julialang.org/t/how-to-efficiently-handle-parallelism-on-a-distributedarray/81098/8 "2022-05-24T02:13:15Z")
I tried to **make all the large args of function DArray** and in this way spawning the tasks is ultra fast. Although this wiil **increase the executation time** of functions because remote fetching of data in function. But it is still better than spawning tasks with large args.
I wonder if there are some functions in distributed like `sactterv` in mpi… really confused now.
---
### Author:  [@ultrapoci](https://discourse.julialang.org/u/ultrapoci)
#### Post date: [May 24, 2022, 10:03am UTC](https://discourse.julialang.org/t/how-to-efficiently-handle-parallelism-on-a-distributedarray/81098/9 "2022-05-24T10:03:38Z")
I’ve done a benchmark a while ago measuring whether it is better to spawn a function passing arguments, or to have it capture the environment as an anonymous function. Strangely enough, it appears as tho passing the argument to the function is a bit slower than having it capture the environment. Not sure if I’ve done something wrong.