# Distributed generation of a DArray

**URL:** <https://discourse.julialang.org/t/distributed-generation-of-a-darray/4472>\
**Category:** General Usage\
**Tags:** parallel\
**Created:** [June 26, 2017, 4:29pm UTC](https://discourse.julialang.org/t/distributed-generation-of-a-darray/4472 "2017-06-26T16:29:34Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [June 26, 2017, 4:29pm UTC](https://discourse.julialang.org/t/distributed-generation-of-a-darray/4472/1 "2017-06-26T16:29:34Z")

</div>

I have generated arrays on separate processes, and cannot collect them into a single array on the master process because it cannot fit into memory. Is there a way to directly generate the `DArray` without having to put it on the master process?

---

<div class="post-metadata">

**Author:** ![andreasnoack](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andreasnoack/32/27_2.png) [@andreasnoack](https://discourse.julialang.org/u/andreasnoack)\
**Post date:** [June 26, 2017, 4:59pm UTC](https://discourse.julialang.org/t/distributed-generation-of-a-darray/4472/2 "2017-06-26T16:59:06Z")

</div>

It is described in [https://github.com/JuliaParallel/DistributedArrays.jl#constructing-distributed-arrays](https://github.com/JuliaParallel/DistributedArrays.jl#constructing-distributed-arrays). How did you construct the arrays on the workers?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [June 26, 2017, 5:09pm UTC](https://discourse.julialang.org/t/distributed-generation-of-a-darray/4472/3 "2017-06-26T17:09:40Z")

</div>

Minimal example of what I am trying, given my current understanding of the README:

```julia
addprocs()
N = 10
@everywhere begin
  srand(myid())
  r = rand(N)
end
using DistributedArrays
dfill(r,((N for i in 1:nprocs())...))

```

But the `dfill` doesn’t grab the part from each process. And what are you supposed to do if the array is not the same name on every process?

---

<div class="post-metadata">

**Author:** ![andreasnoack](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andreasnoack/32/27_2.png) [@andreasnoack](https://discourse.julialang.org/u/andreasnoack)\
**Post date:** [June 26, 2017, 5:28pm UTC](https://discourse.julialang.org/t/distributed-generation-of-a-darray/4472/4 "2017-06-26T17:28:28Z")

</div>

You should try to avoid defining variables on the workers with `@everywhere`. The best approach is to define the local parts with a closure, i.e. something like

```julia
DArray((N*nworkers(),)) do I
    srand(myid())
    rand(N) # or rand(length.(I))
end

```

Alternatively, you can create the local parts and keep a reference to them with a `Future` and then construct a `DArray` like

```julia
rAs = [@spawnat p rand(N) for p in workers()]
4-element Array{Future,1}:
 Future(2, 1, 382, #NULL)
 Future(3, 1, 383, #NULL)
 Future(4, 1, 384, #NULL)
 Future(5, 1, 385, #NULL)

DArray(rAs)

```

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [June 26, 2017, 5:35pm UTC](https://discourse.julialang.org/t/distributed-generation-of-a-darray/4472/5 "2017-06-26T17:35:05Z")

</div>

Thanks. Those are definitely a better way of handling it.

Is the `DArray` constructor able to handle the same stuff as `pmap`? For example, does it need to use a `CachingPool` for large anonymous functions? Is it essentially a `batch_size = N/nprocs()` kind of `pmap`?

---

<div class="post-metadata">

**Author:** ![andreasnoack](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andreasnoack/32/27_2.png) [@andreasnoack](https://discourse.julialang.org/u/andreasnoack)\
**Post date:** [June 26, 2017, 5:40pm UTC](https://discourse.julialang.org/t/distributed-generation-of-a-darray/4472/6 "2017-06-26T17:40:28Z")

</div>

> Is it essentially a batch\_size = N/nprocs() kind of pmap?

Yes, I think that is the right way to think of it. The closure is executed once on each of the workers specified (all if none are specified) so in that sense, the batch size is `N/nworkers()`. Notice that `nprocs() > 1 ? nworkers() == nprocs() - 1 : nworkers() == 1`.