# How to reset GPU after launch failure

**URL:** https://discourse.julialang.org/t/how-to-reset-gpu-after-launch-failure/28768
**Category:** General Usage
**Tags:** cudanative, debug, cuda
**Created:** [September 14, 2019, 9:06pm UTC](https://discourse.julialang.org/t/how-to-reset-gpu-after-launch-failure/28768 "2019-09-14T21:06:25Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![Michael\_Benton](https://avatars.discourse-cdn.com/v4/letter/m/e68b1a/32.png) [@Michael\_Benton](https://discourse.julialang.org/u/Michael_Benton)
#### Post date: [September 14, 2019, 9:06pm UTC](https://discourse.julialang.org/t/how-to-reset-gpu-after-launch-failure/28768/1 "2019-09-14T21:06:25Z")

</div>

If I write a kernel that fails for whatever reason, julia becomes unable to use the GPU for the remainder of the session

example :  
julia\>cu([1.,2.])

CUDA error: unspecified launch failure (code #719, ERROR\_LAUNCH\_FAILED)

---

<div class="post-metadata">

### Author: ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [September 16, 2019, 4:53pm UTC](https://discourse.julialang.org/t/how-to-reset-gpu-after-launch-failure/28768/2 "2019-09-16T16:53:43Z")

</div>

Sadly, CUDA doesn’t allow us to do that, so the only course of action is restarting Julia.

---

<div class="post-metadata">

### Author: ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [June 6, 2020, 7:15am UTC](https://discourse.julialang.org/t/how-to-reset-gpu-after-launch-failure/28768/3 "2020-06-06T07:15:03Z")

</div>

Since somebody just liked this post: Note that nowadays, on recent hardware (Volta or higher), Julia exceptions are supported and will not leave the GPU in a unrecoverable state.

---

<div class="post-metadata">

### Author: ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)
#### Post date: [March 18, 2022, 11:20am UTC](https://discourse.julialang.org/t/how-to-reset-gpu-after-launch-failure/28768/4 "2022-03-18T11:20:14Z")

</div>

Hey maleadt,

This topic with error code 719 (ERROR\_LAUNCH\_FAILED) is really crucial .  
How can I do the reset with the Volta or higher? If I can help for you, let me know what do you advice what should I implement to do the “reset” feature on this error case. (I think this is something that can help for thousands of people later on.)

---

<div class="post-metadata">

### Author: ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [March 18, 2022, 11:41am UTC](https://discourse.julialang.org/t/how-to-reset-gpu-after-launch-failure/28768/5 "2022-03-18T11:41:14Z")

</div>

There’s nothing to enable. If your GPU has compute capability 7 or higher, Julia exceptions will result in a recoverable KernelException. On older hardware, they will result in an unrecoverable CUDA error that requires a process restart. And of course, many CUDA errors (like illegal memory accesses) are always unrecoverable, and there’s nothing we can do about that.

---

<div class="post-metadata">

### Author: ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)
#### Post date: [March 18, 2022, 12:47pm UTC](https://discourse.julialang.org/t/how-to-reset-gpu-after-launch-failure/28768/6 "2022-03-18T12:47:37Z")

</div>

That could explain why I didn’t find any solution on this topic… ☹  
I cannot believe this really an issue in 2022. 😃 Damn!

---

<div class="post-metadata">

### Author: ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [March 18, 2022, 1:08pm UTC](https://discourse.julialang.org/t/how-to-reset-gpu-after-launch-failure/28768/7 "2022-03-18T13:08:36Z")

</div>

There is a solution to this topic, namely to use Volta hardware or more recent. You haven’t specified which error you are running into. If you are triggering an ILLEGAL\_MEMORY\_ACCESS and want to recover, complain to NVIDIA. In general, you shouldn’t be able to trigger such errors with CUDA.jl; kernel array accesses are bounds checked, and will result in a recoverable exception.

---

<div class="post-metadata">

### Author: ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)
#### Post date: [March 18, 2022, 1:43pm UTC](https://discourse.julialang.org/t/how-to-reset-gpu-after-launch-failure/28768/8 "2022-03-18T13:43:15Z")

</div>

Indeed I have computing capability 6.1. ☹ I will try to acquire a better videocard.

I am using kernel functions and having Out of bounds errors somewhere. I am debugging it, but the restart times are really long so it is hard. I just got use to the instantaneous Revise development workflow…  
Also this was pretty interesting as I didn’t face with the bug when I rewrote the `@cuda kernelfn` into a cpu version `for i... kernelfn` version. It was a day ago… I still don’t understand how the CPU code was running properly with out of bound indexing… it was a rollercoaster to find this out of bound indexing, rly… That is why I came to the forum, weather there is a better option to debug things like this.

Thank you for your answers Tim!

---

<div class="post-metadata">

### Author: ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [March 18, 2022, 1:49pm UTC](https://discourse.julialang.org/t/how-to-reset-gpu-after-launch-failure/28768/9 "2022-03-18T13:49:16Z")

</div>

You can try disabling this branch: [https://github.com/JuliaGPU/CUDA.jl/blob/71e9760f02bc6e1bc76aaaf32e98be10a41bae6b/src/compiler/gpucompiler.jl#L35-L37](https://github.com/JuliaGPU/CUDA.jl/blob/71e9760f02bc6e1bc76aaaf32e98be10a41bae6b/src/compiler/gpucompiler.jl#L35-L37)  
But beware that it may lead to miscompilations, so I wouldn’t recommend using it for any other purpose than finding your bug. Also be sure to run with `-g2` for better stack traces.
