# Out of dynamic GPU memory?

**URL:** <https://discourse.julialang.org/t/out-of-dynamic-gpu-memory/76639>\
**Category:** GPU\
**Created:** [February 17, 2022, 5:51pm UTC](https://discourse.julialang.org/t/out-of-dynamic-gpu-memory/76639 "2022-02-17T17:51:25Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![peremato](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/peremato/32/29128_2.png) [@peremato](https://discourse.julialang.org/u/peremato)\
**Post date:** [February 17, 2022, 5:51pm UTC](https://discourse.julialang.org/t/out-of-dynamic-gpu-memory/76639/1 "2022-02-17T17:51:25Z")

</div>

When running a kernel several times (3) I get the following exception (with -g2):

```julia
ERROR: Out of dynamic GPU memory (trying to allocate 64 bytes)
ERROR: Out of dynamic GPU memory (trying to allocate 64 bytes)
ERROR: Out of dynamic GPU memory (trying to allocate 64 bytes)
ERROR: a exception was thrown during kernel execution.
Stacktrace:
ERROR: a exception was thrown during kernel execution.
Stacktrace:
ERROR: a exception was thrown during kernel execution.
Stacktrace:
 [1] gc_pool_alloc at /home/sftnight/.julia/packages/GPUCompiler/1Ajz2/src/runtime.jl:129
 [1] gc_pool_alloc at /home/sftnight/.julia/packages/GPUCompiler/1Ajz2/src/runtime.jl:129
 [1] gc_pool_alloc at /home/sftnight/.julia/packages/GPUCompiler/1Ajz2/src/runtime.jl:129
...

```

I tried to `GC.gc(true); CUDA.reclaim()` between executions but it does not help. I get the crash despite that the memory usage is reduced.

```julia
julia> CUDA.memory_status()
Effective GPU memory usage: 1.41% (209.938 MiB/14.561 GiB)
Memory pool usage: 80.176 KiB (32.000 MiB reserved)
julia> GC.gc(true); CUDA.reclaim()
julia> CUDA.memory_status()
Effective GPU memory usage: 1.19% (177.938 MiB/14.561 GiB)
Memory pool usage: 0 bytes (0 bytes reserved)

```

For information, I am using a CuArray of a Union of 3 structs which is the maximum number of types I can use for the time being (see [Limitation in Union types with CUDA.jl?](https://discourse.julialang.org/t/limitation-in-union-types-with-cuda-jl/76494))  
I there are way to get a real traceback to point to the problem?

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [February 17, 2022, 6:40pm UTC](https://discourse.julialang.org/t/out-of-dynamic-gpu-memory/76639/2 "2022-02-17T18:40:35Z")

</div>

Dynamic memory is memory allocated from within a kernel, and because of how CUDA works that memory is lost after the kernel exist. Basically, don’t allocate within a kernel. The support for that only exists to support some limited cases where we need to allocate an exception object before throwing it.

To find out where the allocations come from, inspect the LLVM code (`@device_code_llvm`) and look for calls to `alloc`-like functions. Escape analysis in 1.8/1.9 is going to improve this, but for now you might have to force-inline some functions or avoid passing complex object to complex functions.

---

<div class="post-metadata">

**Author:** ![peremato](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/peremato/32/29128_2.png) [@peremato](https://discourse.julialang.org/u/peremato)\
**Post date:** [February 18, 2022, 8:14am UTC](https://discourse.julialang.org/t/out-of-dynamic-gpu-memory/76639/3 "2022-02-18T08:14:21Z")

</div>

Thank-you very much. I guess some allocations (i.e. temporary objects) have sneaked in the code that later runs in the kernel. I’ll follow your suggestion.

---

<div class="post-metadata">

**Author:** ![pxl-th](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pxl-th/32/31939_2.png) [@pxl-th](https://discourse.julialang.org/u/pxl-th)\
**Post date:** [February 21, 2022, 12:35am UTC](https://discourse.julialang.org/t/out-of-dynamic-gpu-memory/76639/4 "2022-02-21T00:35:53Z")

</div>

In my case it was due to the use of `StaticArrays.MVector` in kernels. Inlining functions used by the kernel helped eliminate the allocations.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [February 21, 2022, 7:46am UTC](https://discourse.julialang.org/t/out-of-dynamic-gpu-memory/76639/5 "2022-02-21T07:46:26Z")

</div>

A similar case was noted here, [https://github.com/JuliaLang/julia/issues/41800](https://github.com/JuliaLang/julia/issues/41800), which should be fixed in the upcoming 1.8.

---

<div class="post-metadata">

**Author:** ![peremato](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/peremato/32/29128_2.png) [@peremato](https://discourse.julialang.org/u/peremato)\
**Post date:** [February 21, 2022, 12:12pm UTC](https://discourse.julialang.org/t/out-of-dynamic-gpu-memory/76639/6 "2022-02-21T12:12:07Z")

</div>

Indeed I am using `StaticArrays.MVector` ink the kernel. I didn’t know any other way to have a mutable fix length vector.

---

<div class="post-metadata">

**Author:** ![kichappa](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kichappa/32/216306_2.png) [@kichappa](https://discourse.julialang.org/u/kichappa)\
**Post date:** [July 16, 2025, 3:52am UTC](https://discourse.julialang.org/t/out-of-dynamic-gpu-memory/76639/7 "2025-07-16T03:52:59Z")

</div>

> [@maleadt](#):
>
> should be fixed in the upcoming 1.8.

I am finding myself rummaging through this error even now as I use SVector/SMatrices. Was this fixed?

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 16, 2025, 5:47pm UTC](https://discourse.julialang.org/t/out-of-dynamic-gpu-memory/76639/8 "2025-07-16T17:47:11Z")

</div>

Check the LLVM IR (`@device_code_llvm dump_module=true ...`) and look for `gpu_malloc` calls. They can happen when an MArray allocation wasn’t properly optimized away by Julia, resulting in allocations. It happens because mutable StaticArrays are kinda problematic, in that they rely on a Julia optimization kicking in, which doesn’t always happen (as observed here). If you want to avoid running into this, use SArray with `Base.setindex` (i.e. the non-mutating version that returns a new object), which is less likely to run into this.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [July 16, 2025, 11:39pm UTC](https://discourse.julialang.org/t/out-of-dynamic-gpu-memory/76639/9 "2025-07-16T23:39:53Z")

</div>

`SArray`s also allocate if the element type is abstract, even for small `isbits` `Union`s because the backend `Tuple` is exceptionally covariant in its parameters and thus cannot do `Memory`’s inline element optimization. Worth looking out for mistakes in manually specified parameters in constructors, like `SVector{2, Integer}`, or inputs with abstract element types like `SVector{1}(push!([], 1))`.