# Bug in DistributedArrays?

**URL:** <https://discourse.julialang.org/t/bug-in-distributedarrays/12507>\
**Category:** Julia at Scale\
**Created:** [July 19, 2018, 5:25pm UTC](https://discourse.julialang.org/t/bug-in-distributedarrays/12507 "2018-07-19T17:25:14Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![raminammour](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raminammour/32/13572_2.png) [@raminammour](https://discourse.julialang.org/u/raminammour)\
**Post date:** [July 19, 2018, 5:25pm UTC](https://discourse.julialang.org/t/bug-in-distributedarrays/12507/1 "2018-07-19T17:25:14Z")

</div>

Hello all,

I encountered this in an application and it took forever to debug. I think this is a bug in `DistributedArrays` on `v0.6.4`. MWE:

```julia
julia> addprocs(2)
2-element Array{Int64,1}:
 2
 3

julia> using DistributedArrays
julia> a=[rand(2) for i=1:2];

julia> b,c=copy(a),deepcopy(a);

julia> da=@DArray [rand(2) for i=1:2];

julia> db,dc=copy(da),deepcopy(da);

julia> da.pids==db.pids==dc.pids
true

julia> da.indexes==db.indexes==dc.indexes
true

julia> @everywhere f(a,b)=pointer(a),pointer(b),pointer(a)==pointer(b)

julia> f.(a,b) #should be true
2-element Array{Tuple{Ptr{Float64},Ptr{Float64},Bool},1}:
 (Ptr{Float64} @0x00002b397b97b1d0, Ptr{Float64} @0x00002b397b97b1d0, true)
 (Ptr{Float64} @0x00002b397b97b2f0, Ptr{Float64} @0x00002b397b97b2f0, true)

julia> f.(a,c) #should be false
2-element Array{Tuple{Ptr{Float64},Ptr{Float64},Bool},1}:
 (Ptr{Float64} @0x00002b397b97b1d0, Ptr{Float64} @0x00002b397b989f70, false)
 (Ptr{Float64} @0x00002b397b97b2f0, Ptr{Float64} @0x00002b397b989fd0, false)

julia> @spawnat 2 @show pointer(da[:L][1]),pointer(db[:L][1]),pointer(dc[:L][1])
Future(2, 1, 116, Nullable{Any}())

julia> From worker 2: (pointer((da[:L])[1]), pointer((db[:L])[1]), pointer((dc[:L])[1])) = (Ptr{Float64} @0x00002b5fb31db410, Ptr{Float64} @0x00002b5fb31db410, Ptr{Float64} @0x00002b5fb31db410)
julia> 

julia> @spawnat 2 @show pointer(da[:L][1])==pointer(db[:L][1])==pointer(dc[:L][1])
Future(2, 1, 115, Nullable{Any}())

julia> From worker 2: pointer((da[:L])[1]) == pointer((db[:L])[1]) == pointer((dc[:L])[1]) = true

```

The last statement should be wrong but it is true. Is this a bug or expected behavior? Is it worth filing an issue?

Cheers!

---

<div class="post-metadata">

**Author:** ![traktofon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/traktofon/32/591_2.png) [@traktofon](https://discourse.julialang.org/u/traktofon)\
**Post date:** [July 19, 2018, 6:20pm UTC](https://discourse.julialang.org/t/bug-in-distributedarrays/12507/2 "2018-07-19T18:20:04Z")

</div>

The returned pointers seem to be all zero, even on the workers:

```julia
julia> pointer.(da)
2-element DistributedArrays.DArray{Ptr{Float64},1,Array{Ptr{Float64},1}}:
 Ptr{Float64} @0x0000000000000000
 Ptr{Float64} @0x0000000000000000

julia> fetch(@spawnat 2 pointer.(da))
2-element DistributedArrays.DArray{Ptr{Float64},1,Array{Ptr{Float64},1}}:
 Ptr{Float64} @0x0000000000000000
 Ptr{Float64} @0x0000000000000000

```

Actually, what should `pointer` return for a `DArray`? The data is not necessarily stored on the process that invokes `pointer`.

---

<div class="post-metadata">

**Author:** ![raminammour](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raminammour/32/13572_2.png) [@raminammour](https://discourse.julialang.org/u/raminammour)\
**Post date:** [July 19, 2018, 6:54pm UTC](https://discourse.julialang.org/t/bug-in-distributedarrays/12507/3 "2018-07-19T18:54:49Z")

</div>

Sorry those were the wrong examples to show, I had tested differently, this is correct and the issue still holds:

```julia
julia> @spawnat 2 @show pointer(da[:L][1]),pointer(db[:L][1]),pointer(dc[:L][1])

From worker 2: (pointer((da[:L])[1]), pointer((db[:L])[1]), pointer((dc[:L])[1])) = (Ptr{Float64} @0x00002b5fb31db410, Ptr{Float64} @0x00002b5fb31db410, Ptr{Float64} @0x00002b5fb31db410)

julia> @spawnat 2 @show pointer(da[:L][1])==pointer(db[:L][1])==pointer(dc[:L][1])
Future(2, 1, 115, Nullable{Any}())

julia> From worker 2: pointer((da[:L])[1]) == pointer((db[:L])[1]) == pointer((dc[:L])[1]) = true

```

Will update the original post.

Your two examples are because `pointer.(da)` is executed on master and worker two does not “own” `da`.

I am not sure why broadcasting didn’t work correctly for the distributed arrays though, I thought it would.

---

<div class="post-metadata">

**Author:** ![traktofon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/traktofon/32/591_2.png) [@traktofon](https://discourse.julialang.org/u/traktofon)\
**Post date:** [July 20, 2018, 12:49pm UTC](https://discourse.julialang.org/t/bug-in-distributedarrays/12507/4 "2018-07-20T12:49:43Z")

</div>

If I understand correctly, the issue is that `deepcopy(::DArray)` doesn’t create a deep but a shallow copy. But `copy(::DArray)` works as intended, correct?

At least when I try the following (with DistributedArrays master, julia 0.6.4):

```julia
julia> da=@DArray [rand(2) for i=1:2];
julia> db=copy(da); dc=deepcopy(da);
julia> fetch(@spawnat 2 da[:L][1] .= 0);
julia> db
2-element DistributedArrays.DArray{Array{Float64,1},1,Array{Array{Float64,1},1}}:
 [0.0, 0.0]          
 [0.224562, 0.483087]
julia> dc
2-element DistributedArrays.DArray{Array{Float64,1},1,Array{Array{Float64,1},1}}:
 [0.0, 0.0]          
 [0.224562, 0.483087]

```

the outcome for the `copy` looks correct (it is a shallow copy so shares data with the source) but the one for `deepcopy` looks wrong (it should be independent of the source). So it’s worth filing an issue.

I have a [fork of DistributedArrays](https://github.com/traktofon/DistributedArrays.jl/tree/for07) with updates for julia 0.7. It currently passes all tests without deprecation warnings and I’ll open a PR soon. The issue you found was also present there, and I have added an explicit method for `deepcopy(::DArray)` which fixes the issue (also added tests for it). It should also be possible to backport that fix to the current upstream master.

---

<div class="post-metadata">

**Author:** ![raminammour](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raminammour/32/13572_2.png) [@raminammour](https://discourse.julialang.org/u/raminammour)\
**Post date:** [July 20, 2018, 2:50pm UTC](https://discourse.julialang.org/t/bug-in-distributedarrays/12507/5 "2018-07-20T14:50:07Z")

</div>

Correct, that is the issue. Thank you for working on it, it is great!

It may also be worth understanding why broadcasting is not working correctly on the `pointer` function above. When replaced with any other function that works on the arrays, it broadcasts to the local parts correctly, and returns a distributed result.