Is there a way to reset CUDA without restarting julia and be sure everything in memory related to gpu is wiped ? (multiple call to GC.gc and CUDA.reclaim or unsafe_free! seems to miss some objects, leading to increase memory usage and call to GC in my use case). I have a loop that takes quite a long time to execute and i would like to make such a reset after each iteration of the loop.
Not really. We kind-of support resetting the device’s context with CUDA.device_reset!, but doing so complicates lots of things (e.g. using a CuArray from before a reset will instantly trigger illegal memory errors that necessitate restarting the process) and even breaks some NVIDIA libraries. So I would advise against this.
CUDA’s new “green contexts” may be a way to implement this more robustly, but I haven’t looked into this.