# Strange behavior of CuDeviceArrays

**URL:** <https://discourse.julialang.org/t/strange-behavior-of-cudevicearrays/62568>\
**Category:** General Usage\
**Tags:** gpu, cuda\
**Created:** [June 8, 2021, 9:30am UTC](https://discourse.julialang.org/t/strange-behavior-of-cudevicearrays/62568 "2021-06-08T09:30:02Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![fedoroff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fedoroff/32/53209_2.png) [@fedoroff](https://discourse.julialang.org/u/fedoroff)\
**Post date:** [June 8, 2021, 9:30am UTC](https://discourse.julialang.org/t/strange-behavior-of-cudevicearrays/62568/1 "2021-06-08T09:30:02Z")

</div>

I am playing around with passing of custom structures containing arrays into CUDA kernels. In order to be able to access these structure’s arrays inside the kernels, I convert them to CuDeviceArrays using `cudaconvert` function. However, sometimes, I want to see their content. Unfortunately, `Base.collect` function does not work with CuDeviceArrays and I can not explicitly extract their content from the device.

I decided to write my own implementation of `Base.collect`. The idea is to create an empty CuArray, call a CUDA kernel where I transfer the content of CuDeviceArray to the CuArray, element by element, and, finally, collect the resulting CuArray. The code below shows my implementation.

Surprisingly, the obtained results depends on preceding history. The collect function returns the desired content of the CuDeviceArray only if previously I create another CuDeviceArray (which is later never used). If I want to collect CuDeviceArray multiple times, I have to create preventively additional set of CuDeviceArrays.

Can you please explain me what is going on?

```julia
using CUDA

function set_kernel(dest, src)
    id = (blockIdx().x - 1) * blockDim().x + threadIdx().x
    stride = blockDim().x * gridDim().x
    for i=id:stride:length(src)
        dest[i] = src[i]
    end
    return nothing
end

function Base.collect(src::CuDeviceArray)
    dest = CUDA.zeros(size(src))

    N = length(src)
    nthreads = min(N, 256)
    nblocks = cld(N, nthreads)
    @cuda threads=nthreads blocks=nblocks set_kernel(dest, src)

    return collect(dest)
end

# xxx = cudaconvert(CUDA.zeros(1))
# yyy = cudaconvert(CUDA.zeros(1))
# zzz = cudaconvert(CUDA.zeros(1))

a = cudaconvert(CUDA.ones(10))

@show collect(a)
@show collect(a)
@show collect(a)

# xxx, yyy, zzz are commented:
# collect(a) = Float32[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
# collect(a) = Float32[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
# collect(a) = Float32[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

# xxx is uncommented; yyy, zzz are commented:
# collect(a) = Float32[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
# collect(a) = Float32[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
# collect(a) = Float32[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

# xxx, yyy are uncommented; zzz is commented:
# collect(a) = Float32[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
# collect(a) = Float32[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
# collect(a) = Float32[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

# xxx, yyy, zzz are uncommented:
# collect(a) = Float32[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
# collect(a) = Float32[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
# collect(a) = Float32[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]

```

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [June 8, 2021, 10:06am UTC](https://discourse.julialang.org/t/strange-behavior-of-cudevicearrays/62568/2 "2021-06-08T10:06:09Z")

</div>

> [@fedoroff](#):
>
> However, sometimes, I want to see their content.

Just use the original CuArray then? CuDeviceArrays aren’t meant to exist on the CPU, for one, because they don’t keep the original buffer alive. You should just work with the CuArray that backs the CuDeviceArray, and treat the conversion to a GPU-compatible counterpart as an implementation detail.

---

<div class="post-metadata">

**Author:** ![fedoroff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fedoroff/32/53209_2.png) [@fedoroff](https://discourse.julialang.org/u/fedoroff)\
**Post date:** [June 8, 2021, 12:36pm UTC](https://discourse.julialang.org/t/strange-behavior-of-cudevicearrays/62568/3 "2021-06-08T12:36:06Z")

</div>

This is an example of the use case:

```julia
using CUDA

struct Foo{T}
    x :: T
end

function kernel(y, src)
    id = (blockIdx().x - 1) * blockDim().x + threadIdx().x
    stride = blockDim().x * gridDim().x
    for i=id:stride:length(y)
        y[i] = src.x[i]
    end
    return nothing
end

N = 10
y = CUDA.zeros(N)

a = Foo(CUDA.ones(N))
# @cuda threads=N kernel(y, a)
# KernelError: passing and using non-bitstype argument

b = Foo(cudaconvert(CUDA.ones(N)))
@cuda threads=N kernel(y, b)
@show y
# y = Float32[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]

```

With manually converted CuDeviceArrays it is possible to pass custom structures into kernels. Please tell me if there are other methods to do the same.  
If later in the code I wil want to access `x` field of `Foo`, I will have a read only memory error.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [June 8, 2021, 1:01pm UTC](https://discourse.julialang.org/t/strange-behavior-of-cudevicearrays/62568/4 "2021-06-08T13:01:23Z")

</div>

> [@fedoroff](#):
>
> Please tell me if there are other methods to do the same.

Yes, use Adapt:

```julia
julia> using Adapt

julia> Adapt.adapt_structure(to, foo::Foo) = Foo(adapt(to, foo.x))

julia> @cuda threads=N kernel(y, a)
CUDA.HostKernel{typeof(kernel), Tuple{CuDeviceVector{Float32, 1}, Foo{CuDeviceVector{Float32, 1}}}}(kernel, CuContext(0x000000000198a0c0, instance 3d8c8ee076d40ff0), CuModule(Ptr{Nothing} @0x0000000005b27d90, CuContext(0x000000000198a0c0, instance 3d8c8ee076d40ff0)), CuFunction(Ptr{Nothing} @0x00000000087e61d0, CuModule(Ptr{Nothing} @0x0000000005b27d90, CuContext(0x000000000198a0c0, instance 3d8c8ee076d40ff0))))

julia> @show y
y = Float32[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]

```

---

<div class="post-metadata">

**Author:** ![fedoroff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fedoroff/32/53209_2.png) [@fedoroff](https://discourse.julialang.org/u/fedoroff)\
**Post date:** [June 8, 2021, 1:06pm UTC](https://discourse.julialang.org/t/strange-behavior-of-cudevicearrays/62568/5 "2021-06-08T13:06:09Z")

</div>

Wow! Perfect! Thank you.

---

<div class="post-metadata">

**Author:** ![fedoroff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fedoroff/32/53209_2.png) [@fedoroff](https://discourse.julialang.org/u/fedoroff)\
**Post date:** [June 9, 2021, 9:45am UTC](https://discourse.julialang.org/t/strange-behavior-of-cudevicearrays/62568/6 "2021-06-09T09:45:48Z")

</div>

Sorry, one more question.  
How I can pass an array of arrays into the CUDA kernel without manual cudaconvert?  
The illustrating example is the following:

```julia
using CUDA

function kernel(dest, src)
    id = (blockIdx().x - 1) * blockDim().x + threadIdx().x
    stride = blockDim().x * gridDim().x
    for i=id:stride:length(dest)
        dest[i] = src[1][i] + src[2][i]
    end
    return nothing
end

N = 10

out = CUDA.zeros(N)

A = [CUDA.ones(N), CUDA.ones(N)]
# @cuda threads=N kernel(out, A) # passing and using non-bitstype argument
# Ad = CuArray(A) # CuArray only supports bits types
# @cuda threads=N kernel(out, Ad)

B = [cudaconvert(CUDA.ones(N)), cudaconvert(CUDA.ones(N))]
Bd = CuArray(B)
@cuda threads=N kernel(out, Bd)

@show out # out = Float32[2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0]

```

I need it when I try to solve a system of differential equations on GPU using Runge-Kutta methods. In this case I have the array of arrays - vector `ks` containing temporary arrays of length equal to the number of the system equations.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [June 10, 2021, 11:13am UTC](https://discourse.julialang.org/t/strange-behavior-of-cudevicearrays/62568/7 "2021-06-10T11:13:05Z")

</div>

Array element types aren’t converted, only the array itself, so you’d end up with a CuDeviceArray of CuArrays which won’t work. You need to call `cudaconvert` on the inner array manually right now.

---

<div class="post-metadata">

**Author:** ![fedoroff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fedoroff/32/53209_2.png) [@fedoroff](https://discourse.julialang.org/u/fedoroff)\
**Post date:** [June 10, 2021, 11:22am UTC](https://discourse.julialang.org/t/strange-behavior-of-cudevicearrays/62568/8 "2021-06-10T11:22:05Z")

</div>

OK. Thank you.