# Freeing memory in the GPU with CUDAdrv / CUDAnative / CuArrays

**URL:** <https://discourse.julialang.org/t/freeing-memory-in-the-gpu-with-cudadrv-cudanative-cuarrays/10946>\
**Category:** GPU\
**Created:** [May 17, 2018, 1:16am UTC](https://discourse.julialang.org/t/freeing-memory-in-the-gpu-with-cudadrv-cudanative-cuarrays/10946 "2018-05-17T01:16:39Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![una-dinosauria](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/una-dinosauria/32/2077_2.png) [@una-dinosauria](https://discourse.julialang.org/u/una-dinosauria)\
**Post date:** [May 17, 2018, 1:16am UTC](https://discourse.julialang.org/t/freeing-memory-in-the-gpu-with-cudadrv-cudanative-cuarrays/10946/1 "2018-05-17T01:16:39Z")

</div>

I am writing some code that calls CUDA kernels via `CUDAdrv`, allocates some `CuArrays` and uses a generic matrix addition (which I think is done via `CUDAnative`).

The problem I have is that after I call this code a couple of times, my GPU runs out of memory, as it seems that calling `gc()` does not free memory in the GPU.

What is the correct way to free memory in the GPU?

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [May 17, 2018, 5:51am UTC](https://discourse.julialang.org/t/freeing-memory-in-the-gpu-with-cudadrv-cudanative-cuarrays/10946/2 "2018-05-17T05:51:33Z")

</div>

GPU memory _is_ managed through the GC, although indirectly: when CuArray instances go out of scope and they are collected by the Julia GC, the GPU memory refcount is lowered and freed if it drops to 0. So make sure your arrays are out of scope before calling `gc()`, and make sure no other objects share the memory (eg. through a view). You can enable debug messagesthat print during finalization using `JULIA_DEBUG=CUDAdrv` on 0.7, and `TRACE=1` with `--compile-cache=no` on 0.6.

Alternatively, you can force early collection by calling `finalize` on an array. IIRC this is a pretty slow call though, and we should probably add a different early-freeing mechanic. It also won’t do anything if the buffer’s refcount hasn’t dropped to 0.

EDIT: of course, I have assumed you’re talking about CUDAdrv’s CuArray. If you’re talking about CuArrays.jl, there’s an additional level of memory pooling. It should try and free by calling `gc()` once it encounters an out-of-memory error during allocation, and other than that the same rules from above (objects should be out of scope, refcounting) apply.

---

<div class="post-metadata">

**Author:** ![una-dinosauria](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/una-dinosauria/32/2077_2.png) [@una-dinosauria](https://discourse.julialang.org/u/una-dinosauria)\
**Post date:** [October 3, 2018, 1:37am UTC](https://discourse.julialang.org/t/freeing-memory-in-the-gpu-with-cudadrv-cudanative-cuarrays/10946/3 "2018-10-03T01:37:12Z")

</div>

Hi again @maleadt,

Sorry, do you have any suggestions on how to debug this?

I have tried to encapsulate all my allocations in a single function, and then calling `GC.gc()` but apparently I am still missing some objects, as I keep running out of memory.

Is there a way to eg. list all the objects allocated in the GPU? Or at least the ones currently in scope?

Any help is much appreciated,

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [October 4, 2018, 6:21am UTC](https://discourse.julialang.org/t/freeing-memory-in-the-gpu-with-cudadrv-cudanative-cuarrays/10946/4 "2018-10-04T06:21:43Z")

</div>

MWE? Or at least some details, from your original post it wasn’t clear if you are using CUDAdrv.CuArray or CuArrays.jl

Assuming the latter, we could add some infrastructure to print the live buffers in the pool (see `memory.jl`), but that would require some engineering. Maybe it would be easier to show some reproducing code and let us have a look

---

<div class="post-metadata">

**Author:** ![una-dinosauria](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/una-dinosauria/32/2077_2.png) [@una-dinosauria](https://discourse.julialang.org/u/una-dinosauria)\
**Post date:** [October 6, 2018, 5:10pm UTC](https://discourse.julialang.org/t/freeing-memory-in-the-gpu-with-cudadrv-cudanative-cuarrays/10946/5 "2018-10-06T17:10:19Z")

</div>

I have CUDA code in a package that I maintain. I create one `CuArray` from `CUDAdrv.jl` (to make space for multiple CURAND generators), and multiple `CuArray`s from `CuArrays.jl` to call kernels I wrote myself, and to use `CuArrays.CUBLAS.gemm`, which itself allocates memory for the return.

I have tried this code on a machine with a GTX 1080 (8 GB) of RAM and it breaks after 10 or so calls. On a machine with a Titan XP (12 GB), the code runs well.

MWE:

```julia
(v1.0) pkg> develop https://github.com/una-dinosauria/Rayuela.jl.git

```

Due to [https://github.com/JuliaLang/Pkg.jl/issues/465](https://github.com/JuliaLang/Pkg.jl/issues/465),

```julia
cd ~/.julia/dev/Rayuela
git pull

```

To get the latest changes. Now, the following script is the MWE:

```julia
using Rayuela

function callGPU(n::Int, nsplits::Int)

  m, h, d = 8, 256, 128

  X = rand(Float32, d, n)
  B = convert(Matrix{Int16}, rand(1:256, m, n))
  C = Vector{Matrix{Float32}}(undef, m)
  for i=1:m; C[i] = rand(Float32, d, h); end

  ilsiters = [4]
  icmiters = 4
  npert = 4
  randord = true

  V = true

  B = Rayuela.encode_icm_cuda(X, B, C, ilsiters, icmiters, npert, randord, nsplits, V)
  B
end

function main(breakit)

  for i = 1:100
    for j = 1:2
      callGPU(100_000, 1)
      GC.gc()
    end
    nsplits = breakit ? 2 : 10
    callGPU(1_000_000, nsplits)
    GC.gc()
  end

end

breakit = true
main(breakit)

```

Hope this is minimal enough, although I can remove the Rayuela dependency if that is too much.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [October 6, 2018, 5:22pm UTC](https://discourse.julialang.org/t/freeing-memory-in-the-gpu-with-cudadrv-cudanative-cuarrays/10946/6 "2018-10-06T17:22:11Z")

</div>

> [@una-dinosauria](#):
>
> develop [GitHub - una-dinosauria/Rayuela.jl: Code for my PhD thesis. Library of quantization-based methods for fast similarity search in high dimensions. Presented at ECCV 18.](https://github.com/una-dinosauria/Rayuela.jl.git)

Note that `add https://github.com/una-dinosauria/Rayuela.jl.git` works just as well.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [October 8, 2018, 8:30am UTC](https://discourse.julialang.org/t/freeing-memory-in-the-gpu-with-cudadrv-cudanative-cuarrays/10946/7 "2018-10-08T08:30:47Z")

</div>

Great, I’ll have a look. Probably not before next week due to deadlines.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [November 13, 2018, 10:54am UTC](https://discourse.julialang.org/t/freeing-memory-in-the-gpu-with-cudadrv-cudanative-cuarrays/10946/8 "2018-11-13T10:54:20Z")

</div>

Sorry for the delay. Was going to have a look, but the code runs into a CUBLAS error:

```julia
julia> using Revise

julia> Revise.includet("src/10946.jl")
[Info: Recompiling stale cache file /home/tbesard/Julia/depot/compiled/v1.0/Rayuela/4wdef.ji for Rayuela [84bd14ec-51ef-568a-9c69-e494d1752004]

julia> main()
Creating 100000 random states... done in 0.08 seconds
 ILS iteration 1/4 done. 0.00% new codes are equal. 100.00% new codes are better.
 ILS iteration 2/4 done. 80.12% new codes are equal. 8.71% new codes are better.
 ILS iteration 3/4 done. 84.26% new codes are equal. 4.95% new codes are better.
 ILS iteration 4/4 done. 87.63% new codes are equal. 2.54% new codes are better.
 Encoding done in 2.15 seconds
Creating 100000 random states... done in 0.02 seconds
ERROR: CUBLASError(code 14, an internal operation failed)
Stacktrace:
 [1] macro expansion at /home/tbesard/Julia/CuArrays/src/blas/error.jl:45 [inlined]
 [2] gemm!(::Char, ::Char, ::Float32, ::CuArrays.CuArray{Float32,2}, ::CuArrays.CuArray{Float32,2}, ::Float32, ::CuArrays.CuArray{Float32,2}) at /home/tbesard/Julia/CuArrays/src/blas/wrappers.jl:888
 [3] gemm at /home/tbesard/Julia/CuArrays/src/blas/wrappers.jl:903 [inlined]
 [4] encode_icm_cuda_single(::Array{Float32,2}, ::Array{Int16,2}, ::Array{Array{Float32,2},1}, ::Array{Int64,1}, ::Int64, ::Int64, ::Bool, ::Bool) at /home/tbesard/Julia/Rayuela/src/LSQ_GPU.jl:71
 [5] encode_icm_cuda(::Array{Float32,2}, ::Array{Int16,2}, ::Array{Array{Float32,2},1}, ::Array{Int64,1}, ::Int64, ::Int64, ::Bool, ::Int64, ::Bool) at /home/tbesard/Julia/Rayuela/src/LSQ_GPU.jl:231
 [6] main(::Bool) at /home/tbesard/Julia/CuArrays/devel/10946/src/10946.jl:18
 [7] main() at /home/tbesard/Julia/CuArrays/devel/10946/src/10946.jl:23
 [8] top-level scope at none:0

```

Any ideas?

---

<div class="post-metadata">

**Author:** ![una-dinosauria](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/una-dinosauria/32/2077_2.png) [@una-dinosauria](https://discourse.julialang.org/u/una-dinosauria)\
**Post date:** [November 13, 2018, 7:28pm UTC](https://discourse.julialang.org/t/freeing-memory-in-the-gpu-with-cudadrv-cudanative-cuarrays/10946/9 "2018-11-13T19:28:43Z")

</div>

Thanks for looking into this. Currently in crunch time due to CVPR, but will get back ASAP.