# CuArrays and garbage collection when doing FFT convolutions

**URL:** <https://discourse.julialang.org/t/cuarrays-and-garbage-collection-when-doing-fft-convolutions/12412>\
**Category:** Performance\
**Tags:** gpu, gpuarrays\
**Created:** [July 16, 2018, 10:52am UTC](https://discourse.julialang.org/t/cuarrays-and-garbage-collection-when-doing-fft-convolutions/12412 "2018-07-16T10:52:42Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![rveltz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rveltz/32/2707_2.png) [@rveltz](https://discourse.julialang.org/u/rveltz)\
**Post date:** [July 16, 2018, 10:52am UTC](https://discourse.julialang.org/t/cuarrays-and-garbage-collection-when-doing-fft-convolutions/12412/1 "2018-07-16T10:52:42Z")

</div>

Hi,

I am using `CuArrays` to perform fft convolutions and I am getting the error

`ERROR: CUFFTError(code 2, cuFFT failed to allocate GPU or CPU memory)`.

I have threads about collecting temporaries but I don’t know how to do this here.

I’d be happy to have some hints,

Thank you,

Romain

Here is a MWE (hopefully).

```julia
using CuArrays, GPUArrays, CUDAnative
import Base: *

module Convolution
    export convolution

    struct convolution
        kernel::AbstractArray
        kernel_fft::AbstractArray
        tmp::AbstractArray
        fft_flag::Int
        N::Int
        n::Int
        gpu::Bool
        p_forward
        p_backward

        function convolution(kernel::AbstractArray,gpu = false)
            if gpu
                x = new(kernel,fft(fftshift(kernel)),
                    fft(kernel),0,size(kernel)[1],
                    prod(size(kernel)),gpu,
                    0,
                    0)
            else
                error("not here")
            end
            return x
        end
    end
end

function *(cv::Convolution.convolution, x::AbstractArray)
    # return ifftshift(real(irfft(cv.kernel_fft .* rfft(fftshift(x)),cv.N)))
    if cv.gpu
        return real.( ifftshift(ifft(cv.kernel_fft .* fft(fftshift(x)))))
    else
        return ifftshift(real(ifft(cv.kernel_fft .* fft(fftshift(x)))))
    end
end

TY = Float32

dev = CUDAnative.CuDevice(0)
const gpu = cu

N = 2^10
L = 100
hx = 2L/N |> TY
println("\n\n###############\n Neural Field solution , N = $N, dx= $hx\n"*"#"^20)
X = TY.(-L + hx * collect(0:N-1) )
g = TY(1e-4)*exp.(-(1 * X.^2 .+ 1 * X'.^2)/10) |> gpu
J = Convolution.convolution(g,true)

v = rand(N,N) |> gpu;
v2=zeros(v)
for ii=1:20000
	v2 .= J * v
	(ii,GPUArrays.free_global_memory(dev)) |> println
end

```

---

<div class="post-metadata">

**Author:** ![rveltz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rveltz/32/2707_2.png) [@rveltz](https://discourse.julialang.org/u/rveltz)\
**Post date:** [July 17, 2018, 6:20am UTC](https://discourse.julialang.org/t/cuarrays-and-garbage-collection-when-doing-fft-convolutions/12412/2 "2018-07-17T06:20:10Z")

</div>

Should I be pessimistic about this? Or maybe a macro ala `@af_gc` is possible like in `ArrayFire`?

---

<div class="post-metadata">

**Author:** ![rveltz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rveltz/32/2707_2.png) [@rveltz](https://discourse.julialang.org/u/rveltz)\
**Post date:** [July 23, 2018, 7:03pm UTC](https://discourse.julialang.org/t/cuarrays-and-garbage-collection-when-doing-fft-convolutions/12412/3 "2018-07-23T19:03:35Z")

</div>

Does anybody has an advice? my code is not usable with this issue…

---

<div class="post-metadata">

**Author:** ![hskramer](https://avatars.discourse-cdn.com/v4/letter/h/58f4c7/32.png) [@hskramer](https://discourse.julialang.org/u/hskramer)\
**Post date:** [July 24, 2018, 12:12am UTC](https://discourse.julialang.org/t/cuarrays-and-garbage-collection-when-doing-fft-convolutions/12412/4 "2018-07-24T00:12:04Z")

</div>

I have two P5000 and it did finish but it required 16gb of gpu ram the only option I can think of is unified memory. I know the ArrayFire library has support for unified memory so I would look into it and see if it’s been implemented in julia’s ArrayFire.  
Good Luck  
H. Kramer

---

<div class="post-metadata">

**Author:** ![Ralph\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ralph_smith/32/10344_2.png) [@Ralph\_Smith](https://discourse.julialang.org/u/Ralph_Smith)\
**Post date:** [July 24, 2018, 4:06am UTC](https://discourse.julialang.org/t/cuarrays-and-garbage-collection-when-doing-fft-convolutions/12412/5 "2018-07-24T04:06:35Z")

</div>

If you use saved plans to do the transforms there is less stress. (It looks as if you were intending to do that at some point.) Also, putting the `for` loop in a function seems to give the system a better chance to clean up. With these changes I could run your problem on a small GPU, although Julia did grab all of its memory during the loop.

The problem is not strictly related to FFTs - just creating the temporaries for the `fftshift` calls leads to huge memory usage.

---

<div class="post-metadata">

**Author:** ![rveltz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rveltz/32/2707_2.png) [@rveltz](https://discourse.julialang.org/u/rveltz)\
**Post date:** [July 24, 2018, 5:22am UTC](https://discourse.julialang.org/t/cuarrays-and-garbage-collection-when-doing-fft-convolutions/12412/6 "2018-07-24T05:22:31Z")

</div>

Thank you for your input.

---

<div class="post-metadata">

**Author:** ![rveltz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rveltz/32/2707_2.png) [@rveltz](https://discourse.julialang.org/u/rveltz)\
**Post date:** [July 24, 2018, 5:22am UTC](https://discourse.julialang.org/t/cuarrays-and-garbage-collection-when-doing-fft-convolutions/12412/7 "2018-07-24T05:22:51Z")

</div>

How did you come to your last conclusion?

---

<div class="post-metadata">

**Author:** ![Ralph\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ralph_smith/32/10344_2.png) [@Ralph\_Smith](https://discourse.julialang.org/u/Ralph_Smith)\
**Post date:** [July 24, 2018, 12:01pm UTC](https://discourse.julialang.org/t/cuarrays-and-garbage-collection-when-doing-fft-convolutions/12412/8 "2018-07-24T12:01:31Z")

</div>

I just replaced your product expression with `ifftshift(cv.kernel .* fftshift(x))` and watched the memory status. On second thought, my last conclusion is probably wrong - the garbage collection seems to run only when needed, and this works for the cases without repeated plan generation, but not with your original version.

---

<div class="post-metadata">

**Author:** ![sdanisch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sdanisch/32/1406_2.png) [@sdanisch](https://discourse.julialang.org/u/sdanisch)\
**Post date:** [July 24, 2018, 2:50pm UTC](https://discourse.julialang.org/t/cuarrays-and-garbage-collection-when-doing-fft-convolutions/12412/9 "2018-07-24T14:50:43Z")

</div>

You should be able to plan the fft && preallocate!

E.g. like I do here:  
[https://github.com/SimonDanisch/SchroedingersSmoke.jl/blob/master/src/parallel.jl#L79](https://github.com/SimonDanisch/SchroedingersSmoke.jl/blob/master/src/parallel.jl#L79)  
[https://github.com/SimonDanisch/SchroedingersSmoke.jl/blob/master/src/parallel.jl#L170](https://github.com/SimonDanisch/SchroedingersSmoke.jl/blob/master/src/parallel.jl#L170)

Should in theory work with CLArrays & CuArrays like that!

---

<div class="post-metadata">

**Author:** ![rveltz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rveltz/32/2707_2.png) [@rveltz](https://discourse.julialang.org/u/rveltz)\
**Post date:** [July 24, 2018, 5:45pm UTC](https://discourse.julialang.org/t/cuarrays-and-garbage-collection-when-doing-fft-convolutions/12412/10 "2018-07-24T17:45:41Z")

</div>

You think the `plan` is allocating more than your temporary `isf.fc`?

---

<div class="post-metadata">

**Author:** ![sdanisch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sdanisch/32/1406_2.png) [@sdanisch](https://discourse.julialang.org/u/sdanisch)\
**Post date:** [July 24, 2018, 10:21pm UTC](https://discourse.julialang.org/t/cuarrays-and-garbage-collection-when-doing-fft-convolutions/12412/11 "2018-07-24T22:21:04Z")

</div>

I don’t really understand that question…  
You plan an inplace fft and preallocate a buffer for it, so you can have allocation free hot loops, like the one in `possion_solve` - calling `poission_solve` should pretty much have 0 allocations.

---

<div class="post-metadata">

**Author:** ![rveltz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rveltz/32/2707_2.png) [@rveltz](https://discourse.julialang.org/u/rveltz)\
**Post date:** [July 25, 2018, 7:50pm UTC](https://discourse.julialang.org/t/cuarrays-and-garbage-collection-when-doing-fft-convolutions/12412/12 "2018-07-25T19:50:12Z")

</div>

Oh this is nice indeed! I will try it, thank you

---

<div class="post-metadata">

**Author:** ![rveltz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rveltz/32/2707_2.png) [@rveltz](https://discourse.julialang.org/u/rveltz)\
**Post date:** [March 6, 2019, 8:39pm UTC](https://discourse.julialang.org/t/cuarrays-and-garbage-collection-when-doing-fft-convolutions/12412/13 "2019-03-06T20:39:01Z")

</div>

Hi,

Thank you for the suggestion, it worked!

The only allocation is now from fftshift. Is there an inplace version?