# Efficiency when handling jobs larger than VRAM

**URL:** <https://discourse.julialang.org/t/efficiency-when-handling-jobs-larger-than-vram/19570>\
**Category:** GPU\
**Created:** [January 12, 2019, 10:09pm UTC](https://discourse.julialang.org/t/efficiency-when-handling-jobs-larger-than-vram/19570 "2019-01-12T22:09:22Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![wsphillips](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wsphillips/32/6777_2.png) [@wsphillips](https://discourse.julialang.org/u/wsphillips)\
**Post date:** [January 14, 2019, 8:24pm UTC](https://discourse.julialang.org/t/efficiency-when-handling-jobs-larger-than-vram/19570/3 "2019-01-14T20:24:03Z")

</div>

Hey thanks for the feedback!

Data transfer time \>\>\> than compute time no matter what size stack I haul over to the GPU. So I don’t think parallel kernels will make much of a difference.

CUDAdrv has indeed helped solve some issues. The other was not calling `destroy_plan()`. The first code example below _exactly_ produces no memory accumulation. Some things to note: calls to `GC.gc()` don’t free memory ever in my hands. Calling `finalize()` on any CuArray similarly fails. Only CUDAdrv’s `Mem.free(::Buffer)` works consistently. Also (oddly) calling finalize on the context **AFTER** running `destroy!` frees memory that otherwise never gets collected.

And yes, async via streams/events is where I am headed, or at least trying to. A couple hurdles right now that maybe (please?) you could look into…

1. Implementation of `cudaMallocHost` in a straightforward way (e.g. returning an indexable array wrapping the memory). I found burried in `CUDAdrv.Mem.alloc()` that I can set ATTACH\_HOST and get what looks like the correct behavior. Have a look at the second code sample. Its super crude and obviously not stable but I get memory transfer rates between host and device that are 5X as fast…

2. Handles for `cufftSetStream()` and possibly `cufftEstimate*D()` would be super duper helpful too.

Again thanks for your hard work. I dont have the expertise to help with coding to offer extra manpower, but perhaps I can do some documentation work and submit as pull requests as a thank you. It would help me in the end to have some functions described when searched.

```julia
using FFTW
using CuArrays, CuArrays.CUFFT
using CUDAdrv, CUDAnative, CUDAdrv.Mem

A = round.(rand(ComplexF32, 383, 765, 3000).*100)
OUT = zeros(Float32, 765, 765, 3000)
chunksize = 100

@inline function bigrfft(A,OUT,chunksize)

    dev = CuDevice(0)
    ctx = CuContext(dev)
    CUDAdrv.activate(ctx)

    A_GPU = CuArray(A[:,:,1:chunksize])
    abuffer = CuArrays.buffer(A_GPU)
    OUT_GPU = cuzeros(Float32, size(OUT[:,:,1:chunksize]))
    outbuffer = CuArrays.buffer(OUT_GPU)
    p = plan_brfft(A_GPU, 765, [1,2])

    for j in 1:cld(size(A,3),chunksize)
        gpustart = 1 + (j-1)*chunksize
        gpustop = j*chunksize
        Mem.upload!(abuffer, A[:,:,gpustart:gpustop])
        copyto!(OUT_GPU,p*A_GPU)
        CUDAdrv.synchronize(ctx)
        Mem.download!(OUT[:,:,gpustart:gpustop], outbuffer)
    end

    Mem.free(abuffer)
    Mem.free(outbuffer)
    CuArrays.CUFFT.destroy_plan(p)
    CUDAdrv.destroy!(ctx)
    ctx = finalize(ctx)

    return
end

bigrfft(A,OUT,chunksize)

```

Second example. Using pinned memory **results in 5X host-gpu bandwidth** on my workstation. Advisable to evaluate block by block because I can’t guarantee stability.

```julia
using CuArrays, CUDAdrv, CUDAdrv.Mem, CUDAnative
using Test

A = round.(rand(ComplexF32, 383, 765, 300).*100)
Asize = sizeof(A)

#Allocate memory to HOST that the GPU has pinned
function pintoHost(hostArray::Array{T}) where T
    arraysize = sizeof(hostArray)
    arraybuffer = CUDAdrv.Mem.alloc(arraysize,true; flags=Mem.ATTACH_HOST)
    arrayptr = Base.unsafe_convert(Ptr{T}, arraybuffer.ptr)
    pinnedarray = unsafe_wrap(Array, arrayptr, size(hostArray); own = false)
    copyto!(pinnedarray,hostArray)
    return (pinnedarray,arraybuffer)
end
# Verify x is a bonafide copy
(x, xbuffer) = pintoHost(A)
@test x ≈ A

zero_GPU = cuzeros(ComplexF32,size(A))
zero1_GPU = cuzeros(ComplexF32, size(A))
zerobuffer = CuArrays.buffer(zero_GPU)
zero1buffer = CuArrays.buffer(zero1_GPU)

# Pinned memory transfer ≈ 5X faster than regular upload (for me)
@time Mem.transfer!(zerobuffer, xbuffer, sizeof(A))
@time Mem.upload!(zero1buffer, A)
@test collect(zero_GPU) ≈ A
@test collect(zero1_GPU) ≈ A

Mem.free(zerobuffer)
Mem.free(zero1buffer)
zero_GPU = nothing
zero1_GPU = nothing
zerobuffer = finalize(zerobuffer)
zero1buffer = finalize(zero1buffer)

# Test download for similar behavior
B = zeros(ComplexF32, size(A))
A_GPU = CuArray(rand(ComplexF32, size(A)).*100)
agpu_buffer = CuArrays.buffer(A_GPU)
(b, bbuffer) = pintoHost(B)

# Similarly fast
@time Mem.transfer!(bbuffer, agpu_buffer, sizeof(A))
@test collect(A_GPU) ≈ b

Mem.free(agpu_buffer)
A_GPU = finalize(A_GPU)
agpu_buffer = nothing

# NOTE: MUST BE IN THIS ORDER OR SEG FAULT CITY = YOU
x = finalize(x)
Mem.free(xbuffer)
xbuffer = finalize(xbuffer)

b = finalize(b)
Mem.free(bbuffer)
bbuffer = finalize(bbuffer)

```

---

_[View the full topic](https://discourse.julialang.org/t/efficiency-when-handling-jobs-larger-than-vram/19570)._