# Reseting Device

**URL:** <https://discourse.julialang.org/t/reseting-device/63997>\
**Category:** GPU\
**Created:** [July 3, 2021, 6:53pm UTC](https://discourse.julialang.org/t/reseting-device/63997 "2021-07-03T18:53:46Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Fabrice\_Rosay](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabrice_rosay/32/15689_2.png) [@Fabrice\_Rosay](https://discourse.julialang.org/u/Fabrice_Rosay)\
**Post date:** [July 3, 2021, 6:53pm UTC](https://discourse.julialang.org/t/reseting-device/63997/1 "2021-07-03T18:53:46Z")

</div>

Is there any way to reset device state, or at least wipe memory allocated, but keeping already compiled kernel ?  
I keep seeing slow down during iterated call to CUDA and those disapear if i stop and rerun the execution. But as it is expensive to recompile everything, it is a waste of time to do so. I still think it’s about how the memory is handled but it is very difficult to point how exactly.  
I tried running without allocator just in case but it’s obviously a no no. Maybe it is possible to get hands on host memory directly as in C++, I know it’s against the law, but in my particular use case I would like to try it if it’s possible ?

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 3, 2021, 8:44pm UTC](https://discourse.julialang.org/t/reseting-device/63997/2 "2021-07-03T20:44:08Z")

</div>

CUDA.jl has `device_reset!`.

---

<div class="post-metadata">

**Author:** ![Fabrice\_Rosay](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabrice_rosay/32/15689_2.png) [@Fabrice\_Rosay](https://discourse.julialang.org/u/Fabrice_Rosay)\
**Post date:** [July 4, 2021, 7:41am UTC](https://discourse.julialang.org/t/reseting-device/63997/3 "2021-07-04T07:41:47Z")

</div>

Thanks for the answer but this doesn’t work with 11.2 and the use of allocator.  
You have to use another Pool, and then things are slowed down by the pool I believe.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 4, 2021, 12:29pm UTC](https://discourse.julialang.org/t/reseting-device/63997/4 "2021-07-04T12:29:12Z")

</div>

Correct, that’s due to a bug in CUDA. So update your driver 🙂

---

<div class="post-metadata">

**Author:** ![Fabrice\_Rosay](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabrice_rosay/32/15689_2.png) [@Fabrice\_Rosay](https://discourse.julialang.org/u/Fabrice_Rosay)\
**Post date:** [July 5, 2021, 7:26am UTC](https://discourse.julialang.org/t/reseting-device/63997/5 "2021-07-05T07:26:49Z")

</div>

I updated my drivers to latest and everything worked, but it had no effect on the speed.  
Is it possible to create Device array that would not be managed by GC? I would like to avoid GC triggering as much as possible.  
I was thinking something like that:

```julia
function newCu(N)
	buf=Mem.alloc(CUDA.Mem.DeviceBuffer,32*prod(N))
    A=unsafe_wrap(CuArray, convert(CuPtr{Float32}, buf), prod(N),own=false)
	A.=0f0
	return A,buf
end

```

But I have to admit that I don’t really understand if I’m going in the right direction, also I wonder how to destroy them when not needed anymore?

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 5, 2021, 7:30am UTC](https://discourse.julialang.org/t/reseting-device/63997/6 "2021-07-05T07:30:18Z")

</div>

You mentioned a slowdown because of compilation, but now you’re mentioning its GC related?

Anyway, disabling the GC is not a magic bullet. And it requires you to do your own memory management, which is going to be very tricky. If you want to go that far, just use `CUDA.unsafe_free!` to inform CUDA.jl about allocations that can be collected, that should get you pretty far without actually doing your own memory management. Do note that this only drops the allocation’s refcount, so if you have multiple outstanding objects using that buffer – e.g. a view – calling that function on a single instance won’t do anything.

---

<div class="post-metadata">

**Author:** ![Fabrice\_Rosay](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabrice_rosay/32/15689_2.png) [@Fabrice\_Rosay](https://discourse.julialang.org/u/Fabrice_Rosay)\
**Post date:** [July 5, 2021, 7:49am UTC](https://discourse.julialang.org/t/reseting-device/63997/7 "2021-07-05T07:49:41Z")

</div>

I probably was unclear about the slow down, it is still the same problem i mentioned in [GC hitting hard](https://discourse.julialang.org/t/gc-hitting-hard/63102)  
Changing initialization lowered memory pressure, but the slow down keep hitting. I’m pretty sure it is related to memory problem as it strikes faster when working on bigger games. So I was thinking that maybe reseting completly the device would make the next iteration as fast as the first, and I was afraid that functions would then have to be recompiled which it is not the case.  
But reseting didn’t prevent slowdown to happen and it had the effect to make CUDA create a new pool wasting a lot of time so it’s definetly not a solution.  
So all I’m left with is trying to manage memory manually and see if it works.  
`CUDA.unsafe_free!` was not working on unmanaged arrays created with ‘unsafe\_wrap’, what I try is to free the buffer with ‘CUDA.Mem.free’.  
As a side note, I know disabling GC is not magical, but I know also that there is a problem when mixing GC pressure and CUDA.jl and I would like to find a solution, just to see if I can get my implementation as fast as the C++ one. (I also talked to Jonathan Laurent creator of AlphaZero.jl and he told me he also have memory issues).  
Thank you for your help

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 5, 2021, 8:04am UTC](https://discourse.julialang.org/t/reseting-device/63997/8 "2021-07-05T08:04:33Z")

</div>

> [@Fabrice\_Rosay](#):
>
> `CUDA.unsafe_free!` was not working on unmanaged arrays created with ‘unsafe\_wrap’, what I try is to free the buffer with ‘CUDA.Mem.free’.

You’re leaving out crucial details here. If you’re using `unsafe_wrap` to manage your own allocations, you’re not using the memory pool. As a consequence, allocations will be much more expensive, but also you end up with two competing pools which may negatively impact memory management.

---

<div class="post-metadata">

**Author:** ![Fabrice\_Rosay](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabrice_rosay/32/15689_2.png) [@Fabrice\_Rosay](https://discourse.julialang.org/u/Fabrice_Rosay)\
**Post date:** [July 5, 2021, 8:22am UTC](https://discourse.julialang.org/t/reseting-device/63997/9 "2021-07-05T08:22:57Z")

</div>

It’s not easy to talk like that 🙂  
To be clear the original implementation rely only on safe functionality of CUDA.jl, it does not use any low level ones, only `CuArrays` and everything is managed by CUDA.jl. But as it is way slower than c++ counter part, It should be possible to get better, that’s what i try to do with unsafe etc but it is hard…

At a high level the problem could be stated like that: let `f()` be a function (calling CUDA and other stuff).  
Then

```julia
for i in 1:n
   f()
end

```

get slower and slower at each iteration, which before seeing it, I couldn’t imagine it could happen.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 5, 2021, 8:26am UTC](https://discourse.julialang.org/t/reseting-device/63997/10 "2021-07-05T08:26:35Z")

</div>

> [@Fabrice\_Rosay](#):
>
> But as it is way slower than c++ counter part, It should be possible to get better, that’s what i try to do with unsafe etc but it is hard…

But why was it slower? Without knowing that, jumping into unsafe operations that here result in issues with memory management, doesn’t seem like a good idea. Did you profile the code? Was it a specific array operation that was slow?

---

<div class="post-metadata">

**Author:** ![Fabrice\_Rosay](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabrice_rosay/32/15689_2.png) [@Fabrice\_Rosay](https://discourse.julialang.org/u/Fabrice_Rosay)\
**Post date:** [July 5, 2021, 8:42am UTC](https://discourse.julialang.org/t/reseting-device/63997/11 "2021-07-05T08:42:54Z")

</div>

Well I don’t know why it gets slower. As a newbie I was not able to make profile tools work so i’m in the dark. I “profiled” with time() some portions of the code, all I’m sure of is that globally there is a slow down, and I’m pretty sure it’s related to memory issues. The slow down can happen on portions of the code that are only on cpu (i manually timed many portions of the code)  
The problem is I can not make a simple iteration that would replicate the issue, maybe that’s what i should try to do.  
Maybe the problem with Julia and CUDA.jl is that you made so powerful tool that anybody can use them, but the profiling tools are still reserved for clever guys…

---

<div class="post-metadata">

**Author:** ![Fabrice\_Rosay](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabrice_rosay/32/15689_2.png) [@Fabrice\_Rosay](https://discourse.julialang.org/u/Fabrice_Rosay)\
**Post date:** [July 5, 2021, 4:08pm UTC](https://discourse.julialang.org/t/reseting-device/63997/12 "2021-07-05T16:08:29Z")

</div>

Trying to find minimalist code to replicate i found this strange behaviour which might be the source of the problem.

```julia
function test(n)
    L=32*1024

    r=[]
    for i in 1:n
        
        working_batch=CUDA.rand(128,L)
        policy_final=CUDA.zeros(L,81)
        
        for j in 1:50
            for k in 1:64
                prob=CUDA.rand(L,81)
                synchronize()
            end
            policy,batch=Array(policy_final),Array(working_batch)    
            for k in 1:L
                push!(r,policy[k,:])
                if length(r)>2000000
                    popfirst!(r)
                end
            end
        end 
        #GC.gc()
        println(length(r))
    end
end

```

If you execute `test(10)` you will see that memory consumed explodes, it seems GC is not triggered.

If you comment the first inner loop then GC is triggered and memory usage keeps low.

If you comment the second inner loop and the line starting with `policy,batch=...` then no allocation is made as intended.

if you force GC with `GC.gc()` just before print then memory is reclaimed.

I don’t get what is happening, but it seems there is a bad interplay between CUDA and GC. Or maybe I’m fooling myself again.

---

<div class="post-metadata">

**Author:** ![stillyslalom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stillyslalom/32/45687_2.png) [@stillyslalom](https://discourse.julialang.org/u/stillyslalom)\
**Post date:** [July 5, 2021, 6:22pm UTC](https://discourse.julialang.org/t/reseting-device/63997/13 "2021-07-05T18:22:56Z")

</div>

I tried various changes, none of which fixed the allocation issue, but you can get a significant speedup by transposing `policy_final` so that you’re taking memory-contiguous slices with `policy[:,k]`. As a matter of general practice, it’s ill-advised to initialize empty arrays like `r = []`:

```julia
julia> r = []
Any[]

julia> eltype(r)
Any

```

For type stability, it’s better to annotate the eltype like `r = Vector{Float32}[]`.

---

<div class="post-metadata">

**Author:** ![jonathan-laurent](https://avatars.discourse-cdn.com/v4/letter/j/ecae2f/32.png) [@jonathan-laurent](https://discourse.julialang.org/u/jonathan-laurent)\
**Post date:** [July 5, 2021, 6:42pm UTC](https://discourse.julialang.org/t/reseting-device/63997/14 "2021-07-05T18:42:29Z")

</div>

I just want to mention that these types of issues have been a serious headache for [AlphaZero.jl](https://github.com/jonathan-laurent/AlphaZero.jl) from the very start. To be fair, the situation is much better now that is used to be: I remember a time under CuArrays 1.2 where **90%** of training time was spent in GC! But I am still probably leaving a 2x performances factor on the table, just because of bad memory management (AlphaZero.jl may be hit even harder that @Fabrice_Rosay’s implementation as it performs more allocations for the sake of modularity).

As was discussed in [this](https://discourse.julialang.org/t/is-garbage-collection-bad-for-gpu-programming/48942) thread, this is one of the rare places where Python’s ref-counting strategy is actually a great win. And I have never been completely satisfied by the answers provided in the thread I cited, which basically come down to “this is not a big deal in Julia as Julia makes it pretty easy not to allocate when necessary”.

I am still wondering how much of this problem could be solved simply with a better runtime and how much will always come down to having developers eliminate allocations in their code and free resources manually when needed. In the latter case, having powerful tooling to identify memory management issues and fix them strikes me as particularly important.

---

<div class="post-metadata">

**Author:** ![Fabrice\_Rosay](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabrice_rosay/32/15689_2.png) [@Fabrice\_Rosay](https://discourse.julialang.org/u/Fabrice_Rosay)\
**Post date:** [July 5, 2021, 7:44pm UTC](https://discourse.julialang.org/t/reseting-device/63997/15 "2021-07-05T19:44:31Z")

</div>

Thanks for the answer, this part of the loop in real case is not bottleneck it accounts for less than 1% of the time. And actually policy\_final is calculated on the gpu and I think that it is faster to access [thread, index] than reverse order, that way thread in a warp work in close memory. The real problem is to have to manually call GC which is very slow. I don’t understand why GC can handle things when you remove the first loop which does nothing related to the second one.

---

<div class="post-metadata">

**Author:** ![stillyslalom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stillyslalom/32/45687_2.png) [@stillyslalom](https://discourse.julialang.org/u/stillyslalom)\
**Post date:** [July 5, 2021, 7:57pm UTC](https://discourse.julialang.org/t/reseting-device/63997/16 "2021-07-05T19:57:16Z")

</div>

`policy_final` is calculated on the GPU, but you’re indexing into `policy[k,:]`, which was created on the CPU with `policy = Array(policy_final)`, so Julia’s typical column-major memory ordering will still hold.

You’re right - these loops shouldn’t have anything to do with one another. As a band-aid, you can run the GC only when memory pressure is high:

```julia
if Sys.free_memory() < 0.2 * Sys.total_memory()
    GC.gc()
end
```

---

<div class="post-metadata">

**Author:** ![Fabrice\_Rosay](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabrice_rosay/32/15689_2.png) [@Fabrice\_Rosay](https://discourse.julialang.org/u/Fabrice_Rosay)\
**Post date:** [July 5, 2021, 8:02pm UTC](https://discourse.julialang.org/t/reseting-device/63997/17 "2021-07-05T20:02:16Z")

</div>

I just found that adding `CUDA.unsafe_free(prob)` after prob in the first loop and the allocation problem disappear. I think it point towards pressure on the GC coming from the CUDA part.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 6, 2021, 7:11am UTC](https://discourse.julialang.org/t/reseting-device/63997/18 "2021-07-06T07:11:12Z")

</div>

> [@Fabrice\_Rosay](#):
>
> Maybe the problem with Julia and CUDA.jl is that you made so powerful tool that anybody can use them, but the profiling tools are still reserved for clever guys…

That’s never the intention. Did you see the documentation at [Benchmarking & profiling · CUDA.jl](https://cuda.juliagpu.org/stable/development/profiling/)?

But yes, a GC for GPU applications comes at a cost. But as @jonathan-laurent noted above, the situation is already massively better than it used to be. And with some appropriate compiler transformations (essentially doing `unsafe_free!` for you), we should be able to remove this problem for most users. In the meantime, just use `unsafe_free!` yourself, essentially communicating some high-level knowledge about array lifetimes to the run-time. Other workaround shouldn’t be required.

---

<div class="post-metadata">

**Author:** ![Fabrice\_Rosay](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabrice_rosay/32/15689_2.png) [@Fabrice\_Rosay](https://discourse.julialang.org/u/Fabrice_Rosay)\
**Post date:** [July 6, 2021, 12:23pm UTC](https://discourse.julialang.org/t/reseting-device/63997/19 "2021-07-06T12:23:37Z")

</div>

Sir! Yes Sir!  
I will had `unsafe_free!` manually  
To be serious:

1. [quote=“Fabrice\_Rosay, post:11, topic:63997”]  
Maybe the problem with Julia and CUDA.jl is that you made so powerful tool that anybody can use them, but the profiling tools are still reserved for clever guys…  
[/quote] was kinda ironic and not aimed against CUDA.jl
2. that said, it took me 15 days to write a first version of AlphaGPU (around 4 times slower than c++ couterpart), before trying i never had written a single line of CUDA, all credits goes to CUDA.jl
3. another 3 weeks or so to make it more generic and optimize some parts.
4. something like 4 weeks to figure that adding this `unsafe_free!` add a x2 gain (which I don’t complain about, neither did I, it’s just at least imho couter intuitive and maybe it could prove useful to add this example in the section on how to relieve pressure on GC using `unsafe_free!`
5. and I never managed to make any profiler work , although reading to the corresponding section of the CUDA.jl doc, but again I never said anything against your work, it is just a fact.

All in all that made me a litle bit surprise by the harsh ton of your answer, but probably I miss the irony, that happen.  
“Qui bene amat, bene castigat”

Thanks for your time and your answers

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 6, 2021, 12:29pm UTC](https://discourse.julialang.org/t/reseting-device/63997/20 "2021-07-06T12:29:35Z")

</div>

I’m having a hard time understanding which parts of my comment you interpreted as “harsh”, but that obviously wasn’t the intention. I’m just trying to help you out by linking to the documentation in case you missed it, and providing some background on the issue with Julia’s GC / `unsafe_free!`… I’ll try adding some emoticons in the future 🙂

[Next page](https://discourse.julialang.org/t/reseting-device/63997.md?page=2)