How to move an array of struct with array members to GPU?

Specific Domains GPU
1

Hello,

Recently I wanted to write a finite volume method navier-stokes solver with CUDA.jl. I was planning to store my mesh as a large array of cell structs, which stores positions, indices, primitive variables, etc for each cell. The MWE is as follows,

using CUDA,Adapt,StaticArrays
struct Point3D{T} <: FieldVector{3, T}
    x::T
    y::T
    z::T
end
struct cell
    a::Int
    b::AbstractArray{Int64,1}
    c::Point3D{Int64}
end

Adapt.@adapt_structure cell
grid = [cell(i, [i+2, i+3], Point3D(i+3, i+4, i+5)) for i=1:10]
grid = Adapt.adapt.(CuArray, grid) # adapt cell?
grid = Adapt.adapt(CuArray, grid) # adapt grid?

function foo(node::cell)
    node.b[2]=node.a*node.b[1]+node.c.y
end
foo.(grid)

I used Adapt.jl hoping to transfer the cell struct to a CUDA-compatible type, and the grid should also be a CuArray{cell,1}.

When I tried to broadcast the grid to some simple functions, it prompted an error as ERROR: CuArray only supports element types that are allocated inline. Any is not allocated inline. I suspected the adapted data is still incompatible with GPU but I checked the types of grid and grid[1].b, they are both CuArray.

So here is my question, can I move an array of struct with array members to GPU, and if it is possible, how to do this?

I am quite new to CUDA.jl and the finite volume method, if such struct is not workable for broadcasting on GPU, I may try other data structures to store my grid and cells.

Thank you for your help!
Best regards.

2

Not easily. The problem is that when launching a kernel, we convert CuArray to CuDeviceArray. When passing objects that contain CuArrays (e.g. your grid with cells here), we would need to allocate new objects containing CuDeviceArrays, which is generally unacceptably expensive for a kernel launch. If you really want to do this, there’s a couple of hacky solutions to be found on Discourse here, which either boil down to you manually putting CuDeviceArrays in your structures, or using a more complicated adaptor like Arrays of arrays and arrays of structures in CUDA kernels cause random errors - #21 by maleadt

Also note that for cell to be GPU compatible, it cannot contain an AbstractArray, but the b type should be parametric.

3 Likes
3

Thank you for clarifying my question! It seems quite complex to hack this. I think it would be better to reconsider my data structure :thinking:

Thank you again! I did not know this before.