# CuArray local scope memory issue

**URL:** <https://discourse.julialang.org/t/cuarray-local-scope-memory-issue/92421>\
**Category:** GPU\
**Created:** [January 2, 2023, 4:57pm UTC](https://discourse.julialang.org/t/cuarray-local-scope-memory-issue/92421 "2023-01-02T16:57:12Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![rmsmsgood](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rmsmsgood/32/20544_2.png) [@rmsmsgood](https://discourse.julialang.org/u/rmsmsgood)\
**Post date:** [January 2, 2023, 4:57pm UTC](https://discourse.julialang.org/t/cuarray-local-scope-memory-issue/92421/1 "2023-01-02T16:57:12Z")

</div>

Hi, recently i’m interested in GPU programming, not purpose on deep learning. In my case, i have quite large boolean vectors and have to get the intersect of them. I think these `.==` and `.&&` are very simple operations so maybe `CUDA.jl` could be a faster way. But i’m in memory trouble.

## Issue

Full example:

```julia-auto
x = rand(Bool, 5000000);
y = rand(Bool, 5000000);
@time for k in 1:100
    bit_x = (x .== 1)
    bit_y = (y .== 1)
    bit_z = bit_x .&& bit_y
end

using CUDA
a = CuArray(x);
b = CuArray(y);
@time for k in 1:100
    bit_a = (a .== 1)
    bit_b = (b .== 1)
    bit_c = bit_a .&& bit_b
end

```

The CPU part:

```julia-auto
julia> x = rand(Bool, 5000000);

julia> y = rand(Bool, 5000000);

julia> @time for k in 1:100       
           bit_x = (x .== 1)      
           bit_y = (y .== 1)      
           bit_z = bit_x .&& bit_y
       end
  1.442580 seconds (763.85 k allocations: 215.714 MiB, 13.73% gc time, 12.20% compilation time)

julia> @time for k in 1:100       
           bit_x = (x .== 1)      
           bit_y = (y .== 1)      
           bit_z = bit_x .&& bit_y
       end
  1.077367 seconds (1.70 k allocations: 180.086 MiB, 1.74% gc time)

```

I have to do above calculation so many times, more, more than 100.

The GPU part:

```julia-auto
julia> using CUDA

julia> a = CuArray(x);

julia> b = CuArray(y);

julia> @time for k in 1:100       
           bit_a = (a .== 1)      
           bit_b = (b .== 1)      
           bit_c = bit_a .&& bit_b
       end
 15.573386 seconds (28.00 M allocations: 1.421 GiB, 4.25% gc time, 35.52% compilation time)

julia> @time for k in 1:100       
           bit_a = (a .== 1)      
           bit_b = (b .== 1)      
           bit_c = bit_a .&& bit_b
       end
  0.017204 seconds (9.20 k allocations: 503.125 KiB)

julia> @time for k in 1:100       
           bit_a = (a .== 1)      
           bit_b = (b .== 1)      
           bit_c = bit_a .&& bit_b
       end
  0.020961 seconds (9.20 k allocations: 503.125 KiB)

```

First `15.573386 sec` is OK. I have understand why julia is slow at first time. Second, third performance are x1000 fast and that’s what i want to get, it’s cool. But at forth time, i got `StackOverFlowError`:

![Honeycam 2023-01-03 01-19-49](https://global.discourse-cdn.com/julialang/original/3X/0/5/05670fbe1efd3746a6a7fe83b475a8c8c935d168.gif)

 ![20230103_012024](https://global.discourse-cdn.com/julialang/original/3X/6/3/63a709ff94462aecef8740e64dc5ee1d5aab4689.png)

Sorry for korean, “전용 GPU 메모리 사용량” in the third row means vram. The snapshot shows the memory got free when i interrupt with `Ctrl+C` in REPL. But why? Why the GPU usage was increased?

## Question

> Why CuArray operation leaks the memory in local scope?

Here is my question, I expected the `bit_c = bit_a .&& bit_b` in GPU part has local scope hence got free when the for-loop was completely executed. In CPU part, `bit_z = bit_x .&& bit_y` performs like my expectation and I think it’s right in julia scope structure.

How can i fix it? Could i understand why it’s happened? I already have read below documentation and `GC.gc(true)` doesn’t work.

> **[Memory management · CUDA.jl](https://cuda.juliagpu.org/stable/usage/memory/)**
>
> Documentation for CUDA.jl.

## My test environment

- window10
- julia 1.8.3
- CUDA.jl v3.12.0
- CUDA v"11.7.0"
- GTX 1060 6GB

I tested 3 different environments. (window11, julia 1.8.3, CUDA ? and GTX 1060 6GB), (window10, julia 1.8.4, CUDA v12.0.0 and GTX 1060 3GB) I have better server so i could use 24 vram but i think that’s not an essential solution to me. I want to know what is difference between CPU and GPU in julia, how can i handle them.

Thank you for reading.

---

<div class="post-metadata">

**Author:** ![rmsmsgood](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rmsmsgood/32/20544_2.png) [@rmsmsgood](https://discourse.julialang.org/u/rmsmsgood)\
**Post date:** [January 2, 2023, 5:04pm UTC](https://discourse.julialang.org/t/cuarray-local-scope-memory-issue/92421/2 "2023-01-02T17:04:19Z")

</div>

In [Memory management · CUDA.jl](https://cuda.juliagpu.org/stable/usage/memory/#Avoiding-GC-pressure), `CUDA.unsafe_free!` maybe helpful in some cases, but still local scope things are not understandable. Is this a kind of computer science stuff?

```julia
julia> @time for k in 1:100
           bit_a = (a .== 1)
           bit_b = (b .== 1)
           bit_c = bit_a .&& bit_b
           CUDA.unsafe_free!(bit_a)
           CUDA.unsafe_free!(bit_b)
           CUDA.unsafe_free!(bit_c)
       end
  0.002051 seconds (9.50 k allocations: 512.500 KiB)

julia> @time for k in 1:100
           bit_a = (a .== 1)
           bit_b = (b .== 1)
           bit_c = bit_a .&& bit_b
           CUDA.unsafe_free!(bit_a)
           CUDA.unsafe_free!(bit_b)
           CUDA.unsafe_free!(bit_c)
       end
  0.001994 seconds (9.50 k allocations: 512.500 KiB)

julia> @time for k in 1:100
           bit_a = (a .== 1)
           bit_b = (b .== 1)
           bit_c = bit_a .&& bit_b
           CUDA.unsafe_free!(bit_a)
           CUDA.unsafe_free!(bit_b)
           CUDA.unsafe_free!(bit_c)
       end
  0.003385 seconds (9.50 k allocations: 512.500 KiB)

julia> @time for k in 1:100
           bit_a = (a .== 1)
           bit_b = (b .== 1)
           bit_c = bit_a .&& bit_b
           CUDA.unsafe_free!(bit_a)
           CUDA.unsafe_free!(bit_b)
           CUDA.unsafe_free!(bit_c)
       end
  0.001997 seconds (9.50 k allocations: 512.500 KiB)

julia> @time for k in 1:100
           bit_a = (a .== 1)
           bit_b = (b .== 1)
           bit_c = bit_a .&& bit_b
           CUDA.unsafe_free!(bit_a)
           CUDA.unsafe_free!(bit_b)
           CUDA.unsafe_free!(bit_c)
       end
  0.002219 seconds (9.50 k allocations: 512.500 KiB)

```

---

<div class="post-metadata">

**Author:** ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Post date:** [January 3, 2023, 3:52pm UTC](https://discourse.julialang.org/t/cuarray-local-scope-memory-issue/92421/3 "2023-01-03T15:52:28Z")

</div>

[Performance Tips · The Julia Language](https://docs.julialang.org/en/v1/manual/performance-tips/#Performance-critical-code-should-be-inside-a-function) applies here - using these arrays at the “toplevel” (outside of a function) can cause issues with freeing of memory; try putting these `for` loops within a function and calling that function with `@time`.

---

<div class="post-metadata">

**Author:** ![jmair](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jmair/32/35117_2.png) [@jmair](https://discourse.julialang.org/u/jmair)\
**Post date:** [January 4, 2023, 1:12am UTC](https://discourse.julialang.org/t/cuarray-local-scope-memory-issue/92421/4 "2023-01-04T01:12:27Z")

</div>

As @jpsamaroo said, you should put the code inside a function. You can also do this with preallocating memory too. You can do operations in-place with `.=` broadcasting. This function should work for GPU arrays and normal arrays:

```julia
function f!(c, a, b)
    c .= (a.==true) .&& (b.==true)
end

```

This can be benchmarked:

```julia
using CUDA
using BenchmarkTools
n=1024
a=CUDA.rand(Bool, n)
b=CUDA.rand(Bool, n)
c=similar(a)
@btime CUDA.@sync f!($c, $a, $b)

```

This shouldn’t have memory problems or require the GC.

Also, as a PS, could you simplify to the following:

```julia
function f!(c, a, b)
    c .= a .&& b
end

```

---

<div class="post-metadata">

**Author:** ![rmsmsgood](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rmsmsgood/32/20544_2.png) [@rmsmsgood](https://discourse.julialang.org/u/rmsmsgood)\
**Post date:** [January 4, 2023, 2:30am UTC](https://discourse.julialang.org/t/cuarray-local-scope-memory-issue/92421/5 "2023-01-04T02:30:20Z")

</div>

Wow, `.=` broadcasting was the key. If I subsitute `c .= (a.==true) .&& (b.==true)` into `c = (a.==true) .&& (b.==true)` then the code causes same issue. (‘toplevel’ thing or writing function are not much related when I tested). Thank you for your detailed and kind answer. Thank you!!