# CUDA(.jl) memory errors for very large kernels

**URL:** <https://discourse.julialang.org/t/cuda-jl-memory-errors-for-very-large-kernels/125831>\
**Category:** GPU\
**Tags:** cuda, error, code-generation\
**Created:** [February 12, 2025, 2:19pm UTC](https://discourse.julialang.org/t/cuda-jl-memory-errors-for-very-large-kernels/125831 "2025-02-12T14:19:32Z")\
**Posts on this page:** 5\
**Page:** 2

<div class="post-metadata">

**Author:** ![AntonReinhard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antonreinhard/32/211306_2.png) [@AntonReinhard](https://discourse.julialang.org/u/AntonReinhard)\
**Post date:** [April 17, 2025, 7:57am UTC](https://discourse.julialang.org/t/cuda-jl-memory-errors-for-very-large-kernels/125831/21 "2025-04-17T07:57:37Z")

</div>

> [@MrBelette](#):
>
> In the original example you gave, the kernel used a more complicated object (PhaseSpacePoint?). It may not be an SVector, but it could be a plain struct and not for example a struct of arrays. These won’t be automatically transformed into objects that use global memory. Can you perhaps give another example that uses PhaseSpacePoint in a simple manner?

The `PhaseSpacePoint` object is essentially a collection of `SFourMomentum` objects, stored in tuples and some singletons for meta information. `SFourMomentum` itself is an object with 4 simple data members, but behaves like a 4-SVector. The important part here should only be that it’s entirely `isbits`.

The example I gave should pretty closely match the original kernel, in that it pushes objects to the GPU (allocated in `Vector`/`CuVector`, just like my original example), which themselves contain `SVector`-like objects, in the example case `BiSpinor`/`AdjointBiSpinor`/`DiracMatrix`. I didn’t push `SVector`s of input to the GPU in either case.

> [@MrBelette](#):
>
> I should also say that I’m struggling to see where you have massive parallelization where gpus would be potentially beneficial.

The problem isn’t evaluating the kernel I originally posted once. It has to be evaluated millions/billions of times to be useful (in the end it will be used for many-dimensional Monte Carlo integration). So currently I have this one big monolithic kernel that evaluates `f(psp::PhaseSpacePoint) = y::Float`, and I call this on some large `[Cu]Vector` of `PhaseSpacePoint`s. As we’ve seen, `f` can get arbitrarily big depending on the scattering process I’m considering, so I’m looking for ways to break the kernel into smaller parts to be able to parallelize _inside_ `f`, not just across it.  
I’m not sure if this is the way to go, since this would introduce communication between threads (and lots more code complexity), which is usually something you want to avoid on GPUs.

---

<div class="post-metadata">

**Author:** ![MrBelette](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mrbelette/32/4153_2.png) [@MrBelette](https://discourse.julialang.org/u/MrBelette)\
**Post date:** [April 21, 2025, 11:18pm UTC](https://discourse.julialang.org/t/cuda-jl-memory-errors-for-very-large-kernels/125831/22 "2025-04-21T23:18:33Z")

</div>

I suspect that all of the local load/stores are due to allocations that haven’t been optimized away. In your example, the product of a Bispinor and AdjointBiSpinor is a DiracMatrix. The function \_mul returns a new DiracMatrix(BS \* transpose(aBS)).  
The (roughly) CPU-equivalent code for your example is:

```julia
function testmul(in1, in2, out, n)
    @inbounds A = in1[n]
    @inbounds B = in2[n]
    @inbounds out[n] = A*B
end

```

If one uses Cthulhu.jl with inline\_cost=true, then one sees that the inline cost for \_mul is 128.

 ![image](https://global.discourse-cdn.com/julialang/original/3X/e/4/e4d1f8bbc0ac8616db225e95e68aa037d3d3c087.png)

This exceeds the standard cost threshold of 100, so the function call isn’t automatically inlined. Please see [Inlining 101](https://aviatesk.github.io/posts/inlining-101/#extra_a_quick_dive_into_julias_inlining_cost_model)

I believe that when you use always\_inline=true, or manually annotate a method with @inline, the inlining cost threshold is increased 20x i.e. inline\_cost\_threshold becomes 2000 instead of 100 (for each method). So, your simple case results in inlining, but it may not be true for larger cases, especially if there are hidden dynamic dispatches. Certain statements are hit with a inline\_nonleaf\_penalty=1000, for example. C.f. [julia/base/compiler/optimize.jl at a0ff94433010eeb5d1f3297fb4fffdf4a5ef5f7e · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/blob/a0ff94433010eeb5d1f3297fb4fffdf4a5ef5f7e/base/compiler/optimize.jl)

So, I’m wondering if a better strategy for you might be to manually annotate more of your methods with @inline? For example, you can avoid requiring always\_inline=true in your simple example if you annotate the \_mul functions within multiplication.jl. You can then slowly increase the complexity of your models in order to discover where inlining starts to fail again, and then take appropriate action.

---

<div class="post-metadata">

**Author:** ![islent](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/islent/32/11326_2.png) [@islent](https://discourse.julialang.org/u/islent)\
**Post date:** [April 22, 2025, 7:51am UTC](https://discourse.julialang.org/t/cuda-jl-memory-errors-for-very-large-kernels/125831/23 "2025-04-22T07:51:06Z")

</div>

For me, this error often happens when the temporal variables are not immediately released, especially in the case of carrying out `fft`.

I solve this by manually release the GPU memory using `CUDA.unsafe_free!`.

```julia
a = cu(rand(512, 512, 512))

a = nothing # this will not GC immediately

CUDA.unsafe_free!(a) # GPU memory is immediately released. And you should never use variable a before redefining it

```

---

<div class="post-metadata">

**Author:** ![AntonReinhard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antonreinhard/32/211306_2.png) [@AntonReinhard](https://discourse.julialang.org/u/AntonReinhard)\
**Post date:** [April 22, 2025, 4:35pm UTC](https://discourse.julialang.org/t/cuda-jl-memory-errors-for-very-large-kernels/125831/24 "2025-04-22T16:35:16Z")

</div>

> [@MrBelette](#):
>
> So, I’m wondering if a better strategy for you might be to manually annotate more of your methods with @inline? For example, you can avoid requiring always\_inline=true in your simple example if you annotate the \_mul functions within multiplication.jl. You can then slowly increase the complexity of your models in order to discover where inlining starts to fail again, and then take appropriate action.

It looks like one of the inhibitors to the inlining might be from LinearAlgebra. For example:

```julia
*(x::LinearAlgebra.Transpose{T, <:AbstractVector} where T, A::AbstractMatrix) @ LinearAlgebra ~/.julia/juliaup/julia-1.11.5+0.x64.linux.gnu/share/julia/stdlib/v1.11/LinearAlgebra/src/matmul.jl:96
   ∘ ─ 128 %0 = invoke *(::LinearAlgebra.Transpose{Float64, StaticArraysCore.SVector{4, Float64}},::StaticArraysCore.SMatrix{4, 4, Float64, 16})::LinearAlgebra.Transpose{Float64, StaticArraysCore.SVector{4, Float64}} 

```

I can `@inline` the expression, but it still doesn’t seem to inline all the way…

---

<div class="post-metadata">

**Author:** ![MrBelette](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mrbelette/32/4153_2.png) [@MrBelette](https://discourse.julialang.org/u/MrBelette)\
**Post date:** [April 22, 2025, 6:45pm UTC](https://discourse.julialang.org/t/cuda-jl-memory-errors-for-very-large-kernels/125831/25 "2025-04-22T18:45:39Z")

</div>

Strange, it’s not what I’m seeing for your test kernel. Have you toggled [o]ptimize?

Here’s what I see for your test kernel (hopefully it’s not too small):

 ![image](https://global.discourse-cdn.com/julialang/original/3X/6/e/6eb8a01d1f6eeca190bf81bdc7e1c9594b391bd1.png)

Furthermore, I don’t see any local load/stores for that kernel when I use @device\_code\_ptx without using always\_inline=true, and I inline the \_mul statements. Same for sass code:

```julia
K = @cuda launch=true always_inline=false kernel(in1, in2, out, N)
@show CUDA.memory(K)
@show CUDA.registers(K)

```

gives:  
 ![image](https://global.discourse-cdn.com/julialang/original/3X/f/5/f508dea6c4bd9c2bfa00a0ccb7bec5d99952fd90.png)

[Previous page](https://discourse.julialang.org/t/cuda-jl-memory-errors-for-very-large-kernels/125831.md?page=1)
