# Why is it consuming and not freeing GPU memory?

**URL:** <https://discourse.julialang.org/t/why-is-it-consuming-and-not-freeing-gpu-memory/113087>\
**Category:** GPU\
**Created:** [April 17, 2024, 6:34pm UTC](https://discourse.julialang.org/t/why-is-it-consuming-and-not-freeing-gpu-memory/113087 "2024-04-17T18:34:26Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ferran\_Mazzanti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ferran_mazzanti/32/7458_2.png) [@Ferran\_Mazzanti](https://discourse.julialang.org/u/Ferran_Mazzanti)\
**Post date:** [April 17, 2024, 6:34pm UTC](https://discourse.julialang.org/t/why-is-it-consuming-and-not-freeing-gpu-memory/113087/1 "2024-04-17T18:34:26Z")

</div>

Hi,  
I’m running into something I do not understand. My simple CUDA code is taking  
memory all the time and not freeing it. A simple example:

```julia
julia> using CUDA

julia> CUDA.memory_status()
Effective GPU memory usage: 4.16% (501.188 MiB/11.759 GiB)
Memory pool usage: 0 bytes (0 bytes reserved)

julia> kk = CUDA.rand(256,256,256)
julia> aux = CUDA.rand(256,256,256)

CUDA.memory_status()
Effective GPU memory usage: 19.58% (2.302 GiB/11.759 GiB)
Memory pool usage: 128.000 MiB (128.000 MiB reserved)

julia>for i in 1:10
    aux .= CUDA.exp.(kk)
    CUDA.memory_status()
end
Effective GPU memory usage: 19.64% (2.310 GiB/11.759 GiB)
Memory pool usage: 128.000 MiB (128.000 MiB reserved)
Effective GPU memory usage: 19.64% (2.310 GiB/11.759 GiB)
Memory pool usage: 128.000 MiB (128.000 MiB reserved)
Effective GPU memory usage: 19.64% (2.310 GiB/11.759 GiB)
Memory pool usage: 128.000 MiB (128.000 MiB reserved)
Effective GPU memory usage: 19.64% (2.310 GiB/11.759 GiB)
Memory pool usage: 128.000 MiB (128.000 MiB reserved)
Effective GPU memory usage: 19.64% (2.310 GiB/11.759 GiB)
Memory pool usage: 128.000 MiB (128.000 MiB reserved)
Effective GPU memory usage: 19.64% (2.310 GiB/11.759 GiB)
Memory pool usage: 128.000 MiB (128.000 MiB reserved)
Effective GPU memory usage: 19.64% (2.310 GiB/11.759 GiB)
Memory pool usage: 128.000 MiB (128.000 MiB reserved)
Effective GPU memory usage: 19.64% (2.310 GiB/11.759 GiB)
Memory pool usage: 128.000 MiB (128.000 MiB reserved)
Effective GPU memory usage: 19.64% (2.310 GiB/11.759 GiB)
Memory pool usage: 128.000 MiB (128.000 MiB reserved)
Effective GPU memory usage: 19.64% (2.310 GiB/11.759 GiB)
Memory pool usage: 128.000 MiB (128.000 MiB reserved)

```

ok that’s what I would expect, no new memory is being allocated in the GPU.  
But if I run

```julia
julia> for i in 1:10
    aux .= CUDA.exp.(0.01f0*kk)
    CUDA.memory_status()
end
Effective GPU memory usage: 20.26% (2.382 GiB/11.759 GiB)
Memory pool usage: 192.000 MiB (192.000 MiB reserved)
Effective GPU memory usage: 20.79% (2.445 GiB/11.759 GiB)
Memory pool usage: 256.000 MiB (256.000 MiB reserved)
Effective GPU memory usage: 21.32% (2.507 GiB/11.759 GiB)
Memory pool usage: 320.000 MiB (320.000 MiB reserved)
Effective GPU memory usage: 21.85% (2.570 GiB/11.759 GiB)
Memory pool usage: 384.000 MiB (384.000 MiB reserved)
Effective GPU memory usage: 22.39% (2.632 GiB/11.759 GiB)
Memory pool usage: 448.000 MiB (448.000 MiB reserved)
Effective GPU memory usage: 22.92% (2.695 GiB/11.759 GiB)
Memory pool usage: 512.000 MiB (512.000 MiB reserved)
Effective GPU memory usage: 23.45% (2.757 GiB/11.759 GiB)
Memory pool usage: 576.000 MiB (576.000 MiB reserved)
Effective GPU memory usage: 23.98% (2.820 GiB/11.759 GiB)
Memory pool usage: 640.000 MiB (640.000 MiB reserved)
Effective GPU memory usage: 24.51% (2.882 GiB/11.759 GiB)
Memory pool usage: 704.000 MiB (704.000 MiB reserved)
Effective GPU memory usage: 25.04% (2.945 GiB/11.759 GiB)
Memory pool usage: 768.000 MiB (768.000 MiB reserved)

```

it keeps consuming memory and not freeing it. And nothing seems  
yto be releasing it

```julia
julia> CUDA.memory_status()
Effective GPU memory usage: 25.52% (3.000 GiB/11.759 GiB)
Memory pool usage: 768.000 MiB (768.000 MiB reserved)

julia> CUDA.reclaim()

julia> CUDA.memory_status()
Effective GPU memory usage: 25.45% (2.993 GiB/11.759 GiB)
Memory pool usage: 768.000 MiB (768.000 MiB reserved)

```

this is a minimal working example, but in my iteration codes that makes my linux  
box hang sometimes…

What am I doing wrong?

Thanks in advance…

---

<div class="post-metadata">

**Author:** ![tomaklutfu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomaklutfu/32/2411_2.png) [@tomaklutfu](https://discourse.julialang.org/u/tomaklutfu)\
**Post date:** [April 18, 2024, 8:28am UTC](https://discourse.julialang.org/t/why-is-it-consuming-and-not-freeing-gpu-memory/113087/2 "2024-04-18T08:28:52Z")

</div>

> [@Ferran\_Mazzanti](#):
>
> ```julia
> julia> for i in 1:10
> aux .= CUDA.exp.(0.01f0*kk)
> CUDA.memory_status()
> end
> 
> ```

Did you mean to do?

```julia
julia> for i in 1:10
    aux .= CUDA.exp.(0.01f0.*kk)
    CUDA.memory_status()
end

```

Notice `.*` instead of `*` for broadcasting.

---

<div class="post-metadata">

**Author:** ![artemsolod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/artemsolod/32/20704_2.png) [@artemsolod](https://discourse.julialang.org/u/artemsolod)\
**Post date:** [April 18, 2024, 9:33am UTC](https://discourse.julialang.org/t/why-is-it-consuming-and-not-freeing-gpu-memory/113087/3 "2024-04-18T09:33:53Z")

</div>

Judging by the docs, you can try setting `JULIA_CUDA_SOFT_MEMORY_LIMIT` or `JULIA_CUDA_HARD_MEMORY_LIMIT` environment variable (not sure, but I think it needs to be set before `using CUDA`). Also consider calling `CUDA.reclaim()`, `GC.gc()` or manually freeing with `CUDA.unsafe_free!(a)`.

> **[Memory management · CUDA.jl](https://cuda.juliagpu.org/stable/usage/memory/#Garbage-collection)**
>
> Documentation for CUDA.jl.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [April 18, 2024, 10:00am UTC](https://discourse.julialang.org/t/why-is-it-consuming-and-not-freeing-gpu-memory/113087/4 "2024-04-18T10:00:38Z")

</div>

> [@Ferran\_Mazzanti](#):
>
> in my iteration codes that makes my linux  
> box hang sometimes…

You should generally not use the same GPU for driving a display and doing computationally-intensive things. It’s actually the compute that will make the output “hang”, not the use of memory.

And regarding the use of memory, Julia is a garbage collected language, so memory will only be collected when it’s necessary.

> [@artemsolod](#):
>
> Also consider calling `CUDA.reclaim()`, `GC.gc()` or manually freeing with `CUDA.unsafe_free!(a)`.

Please don’t call `CUDA.reclaim()` or even `GC.gc()` unless really necessary, both operations will slow down your application significantly if misused. `unsafe_free!`ing unused memory can be good practice, but as the name implies it’s an unsafe operation so should be done with care.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [April 18, 2024, 2:22pm UTC](https://discourse.julialang.org/t/why-is-it-consuming-and-not-freeing-gpu-memory/113087/5 "2024-04-18T14:22:41Z")

</div>

You can also try [Consider running GC when allocating and synchronizing by maleadt · Pull Request #2304 · JuliaGPU/CUDA.jl · GitHub](https://github.com/JuliaGPU/CUDA.jl/pull/2304), which will consider collecting memory at more points than only when running out of it.

---

<div class="post-metadata">

**Author:** ![Ferran\_Mazzanti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ferran_mazzanti/32/7458_2.png) [@Ferran\_Mazzanti](https://discourse.julialang.org/u/Ferran_Mazzanti)\
**Post date:** [April 18, 2024, 3:55pm UTC](https://discourse.julialang.org/t/why-is-it-consuming-and-not-freeing-gpu-memory/113087/6 "2024-04-18T15:55:25Z")

</div>

Hi,  
well seems that setting the soft limits of the amount of memory to use in the GPU with  
`JULIA_CUDA_SOFT_MEMORY_LIMIT`  
solves the problem, at least in the very first tests I’m conducting now, thanks.  
Best,  
Ferran.