# Using MVector in CUDA without memory errors

**URL:** <https://discourse.julialang.org/t/using-mvector-in-cuda-without-memory-errors/105016>\
**Category:** GPU\
**Created:** [October 16, 2023, 1:56pm UTC](https://discourse.julialang.org/t/using-mvector-in-cuda-without-memory-errors/105016 "2023-10-16T13:56:07Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![snowgum](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@snowgum](https://discourse.julialang.org/u/snowgum)\
**Post date:** [October 16, 2023, 1:56pm UTC](https://discourse.julialang.org/t/using-mvector-in-cuda-without-memory-errors/105016/1 "2023-10-16T13:56:07Z")

</div>

I have an application where allocating a small fixed-sized vector for temporary storage inside a kernel is very beneficial. From reading this forum, [issue 340 on CUDAnative.jl](https://github.com/JuliaGPU/CUDAnative.jl/issues/340) and an [issue 38 comment](https://github.com/JuliaGPU/CUDA.jl/issues/38#issuecomment-894315291), I am assuming it is ok to use an MVector.

Using this simplified example seems to work:

```julia
using CUDA
using StaticArrays

function kernel1(a)
    nlevels = 6
    stack = ones(MVector{nlevels, Float32})

    nx = size(a)[1]
    ix = (blockIdx().x - 1) * blockDim().x + threadIdx().x
    
    if (ix <= nx)
        simple_func!(a[ix], stack)
    end
    return
end

function simple_func!(x, y)
    for i in 1:6
        y[i] = x * 2f0
    end
    return nothing
end

```

This runs fine, and there is no memory leak even if I ran it repeatedly (e.g. with `@benchmark`):

```julia
N = 2^20
a = CUDA.fill(1f0, N)
@benchmark CUDA.@sync @cuda threads=1024 blocks=1024 kernel1($a)

```

However, if I modify my kernel slightly to save the results into a second variable:

```julia
function kernel2(a, b)
    nlevels = 6
    tmp = ones(MVector{nlevels, Float32})

    nx = size(a)[1]
    ix = (blockIdx().x - 1) * blockDim().x + threadIdx().x
    
    if (ix <= nx)
        simple_func!(a[ix], tmp)
        for i in 1:nlevels
            b[i, ix] = tmp[i]
        end
    end
    return
end

```

I quickly run into “Out of dynamic GPU memory” errors:

```julia
N = 2^20
a = CUDA.fill(1f0, N)
b = CUDA.fill(0f1, (6, N));

CUDA.@sync @cuda threads=1024 blocks=1024 kernel2(a,b)
# ERROR: Out of dynamic GPU memory (trying to allocate 24 bytes)
# ....

```

Any idea on why I am running out of memory in the second case but not the first?

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [October 16, 2023, 3:25pm UTC](https://discourse.julialang.org/t/using-mvector-in-cuda-without-memory-errors/105016/2 "2023-10-16T15:25:11Z")

</div>

> [@snowgum](#):
>
> ` @cuda threads=1024 blocks=1024 kernel1($a)`

You can use `@device_code_typed` and `@device_code_llvm` to better understand what the compiler does. In your case in `kernel1` The compiler removed almost everything (except for a boundscheck) since it could prove that there are no side-effects.

Looking at `kernel2` I see a call to `%4 = call fastcc {}* @gpu_gc_pool_alloc({ i64, i32 } %state, i64 24)` which explains why you run out of space.

Now `MArray` is finicky, but since Julia uses bounds-checking by default and bounds checking error messages take a reference to the array itself, Julia can’t ellide the allocation for MArray…

```julia
function kernel2(a, b)
           nlevels = 6
           tmp = ones(MVector{nlevels, Float32})

           nx = size(a)[1]
           ix = (blockIdx().x - 1) * blockDim().x + threadIdx().x
           
           if (ix <= nx)
               simple_func!(a[ix], tmp)
               for i in 1:nlevels
                    b[i, ix] = @inbounds tmp[i]
               end
           end
           return
       end

```

So LLVM can’t unroll the loop, thus can’t prove that the boundscheck branch is dead, and can’t ellide the MArray allocation.

---

<div class="post-metadata">

**Author:** ![snowgum](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@snowgum](https://discourse.julialang.org/u/snowgum)\
**Post date:** [October 16, 2023, 8:59pm UTC](https://discourse.julialang.org/t/using-mvector-in-cuda-without-memory-errors/105016/3 "2023-10-16T20:59:21Z")

</div>

Thanks for the quick reply! Indeed that was the problem. Now need to keep in mind that bounds checking will not work with `MArray`…

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [October 17, 2023, 2:46am UTC](https://discourse.julialang.org/t/using-mvector-in-cuda-without-memory-errors/105016/4 "2023-10-17T02:46:19Z")

</div>

We should fix that in StaticArrays.jl I just never found the time to do so.  
Two problems:

1. I think the error is generated by `checkbounds` in Base Julia.
2. any other function that is using an MArray as an argument and is not inlined will cause the same.