# Using GPU via PyCall causes non-reusable memory allocation

**URL:** <https://discourse.julialang.org/t/using-gpu-via-pycall-causes-non-reusable-memory-allocation/55140>\
**Category:** GPU\
**Tags:** question, pycall, gpu, pytorch, garbage-collection\
**Created:** [February 12, 2021, 11:47am UTC](https://discourse.julialang.org/t/using-gpu-via-pycall-causes-non-reusable-memory-allocation/55140 "2021-02-12T11:47:34Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![JosePereiraUA](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/josepereiraua/32/37640_2.png) [@JosePereiraUA](https://discourse.julialang.org/u/JosePereiraUA)\
**Post date:** [February 12, 2021, 11:47am UTC](https://discourse.julialang.org/t/using-gpu-via-pycall-causes-non-reusable-memory-allocation/55140/1 "2021-02-12T11:47:34Z")

</div>

My name is José Pereira, I’m a portuguese PhD student currently using Julia to develop a light-weight protein design package. I’m trying to implement [TorchANI](https://github.com/aiqm/torchani), a known ML model for molecular energy calculation, developed in Python using the Torch package. My first attempt was to use the [PyCall](https://github.com/JuliaPy/PyCall.jl)package to call the Python code directly, something like:

```julia
using PyCall
@pyimport torch
@pyimport torchani

device = torch.device("cuda")
model = torchani.models.ANI2x(periodic_table_index = true).to(device)
model(...)

```

However, after a few hundred calls to the “model”, memory allocation causes a “CUDA out of memory” error. [This has been previously observed by others.](https://github.com/JuliaPy/PyCall.jl/issues/529) The problem seems to be related to the garbage collection mechanism, as calling `GC.gc(false)` seems to help. I’ve used multiple profiling tools and verified that certain Python lines of code are allocating memory, such as:

`p12_all = torch.triu_indices(num_atoms, num_atoms, 1, device=current_device)`

This allocation, when running the same code in Python (not via PyCall in Julia) get’s re-used on the next iteraction/call. However, in Julia, it is being allocated every step of a loop until an explicit call to `GC.gc(false)` is performed (which, of course, leads to extremely low performance). At this point the allocated memory in the GPU, as we know, does not get freed, it remains allocated and is eventually re-used.

Therefore, my question is: **is this behavior expected, or implemented by default? Is there anything I can do to re-use the memory allocated by Python?**

Thank you 😄

---

<div class="post-metadata">

**Author:** ![groot](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/groot/32/13081_2.png) [@groot](https://discourse.julialang.org/u/groot)\
**Post date:** [February 13, 2021, 10:26pm UTC](https://discourse.julialang.org/t/using-gpu-via-pycall-causes-non-reusable-memory-allocation/55140/2 "2021-02-13T22:26:29Z")

</div>

Had a similar problem. The issue you have is about the GPU cache on the PyTorch side. Try

```julia
torch.cuda.empty_cache()

```

after model call

See [How can we release GPU memory cache? - PyTorch Forums](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530)

---

<div class="post-metadata">

**Author:** ![JosePereiraUA](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/josepereiraua/32/37640_2.png) [@JosePereiraUA](https://discourse.julialang.org/u/JosePereiraUA)\
**Post date:** [February 16, 2021, 3:39pm UTC](https://discourse.julialang.org/t/using-gpu-via-pycall-causes-non-reusable-memory-allocation/55140/3 "2021-02-16T15:39:02Z")

</div>

Thanks for your answer. I tried to simply add `torch.cuda.empty_cache()` after each model call, but it didn’t change a thing:

![Captura de ecrã 2021-02-16 153806](https://global.discourse-cdn.com/julialang/original/3X/b/4/b4c48f55440b7b4b84a168b95580999960679c61.png)

this is the memory allocation after just a few iterations. The memory keeps leaking.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [February 16, 2021, 4:07pm UTC](https://discourse.julialang.org/t/using-gpu-via-pycall-causes-non-reusable-memory-allocation/55140/4 "2021-02-16T16:07:04Z")

</div>

> [@JosePereiraUA](#):
>
> calling `GC.gc(false)` seems to help […] Is there anything I can do to re-use the memory allocated by Python?

It sounds like Julia’s garbage collection just isn’t running frequently enough for you, probably because Julia doesn’t know that memory is running low on the CUDA side?

You can explicitly tell Python you are done with an object `o` from PyCall by calling `pydecref(o)`. (This is safe if you are done with the object: it gets mutated to a NULL object to prevent it from being decref’ed again. Perhaps the function should have been called `pydecref!`…) Equivalently, you can just call `finalize(o)`.

See also [stop using finalizers for resource management? · Issue #11207 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/11207) and [`with` for deterministic destruction · Issue #7721 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/7721) for discussion of this general issue for resource management in Julia.

---

<div class="post-metadata">

**Author:** ![JosePereiraUA](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/josepereiraua/32/37640_2.png) [@JosePereiraUA](https://discourse.julialang.org/u/JosePereiraUA)\
**Post date:** [February 16, 2021, 6:33pm UTC](https://discourse.julialang.org/t/using-gpu-via-pycall-causes-non-reusable-memory-allocation/55140/5 "2021-02-16T18:33:14Z")

</div>

**I will take a look at these links and ideas and report back.**  
So far my solution has been to:

1. Measure current GPU allocation every step
2. Once current GPU allocation surpasses my established threshold (say, 50% of total available GPU memory), set the number of elapsed steps as the maximum allowed number of calls before a `GC.gc(false)` is called (`N`).
3. Call `GC.gc(false)` every `N` steps.