PythonCall.jl and garbage collection

General Usage
1

I am trying out PythonCall.jl and I am loving it so far. However, it seems that when using some Python libraries that call C/C++ under the hood (for example NumPy) Python objects created with PythonCall.jl are never automatically collected from memory. For example:

pkg> activate --temp
pkg> add PythonCall, CondaPkg
julia> using CondaPkg
pkg> conda pip_add numpy
julia> using PythonCall
julia> np = pyimport("numpy")
julia> function createarray()
         a = np.random.rand(10000,10000)
         return nothing
       end
julia> createarray() # memory usage increases
julia> createarray() # memory usage increases even more
julia> createarray() # memory usage keeps increasing
# ...
# if you keep calling createarray(), the memory usage keeps going up 
# until eventually it fills up and the julia process is killed by the OS

Inserting manual calls to GC.gc() and/or PythonCall.pydel!(object) helps releasing the memory, but force the user to do “manual memory management” (i.e. having to keep track of when specific Python objects stop being needed and inserting appropriate function calls). Failing to do so correctly quickly lead to memory issues if you are dealing with somewhat large data structures, complicating the usage of Python libraries from Julia. This behavior is not unique to NumPy, as I noticed it also when trying to use the OR-Tools library.

EDIT: Please ignore the following example, it was meant to be about Python lists, but I just realized I was creating a Julia Vector instead, and not even correctly. I am sorry about that :sweat_smile:. Everything else should still be valid. END OF EDIT

In constrast, Python lists do not seem to suffer from the same problem:

julia> random = pyimport("random")
julia> function createlist()
         a = [[random.random() for _ in 10000] for _ in 10000]
         return nothing
       end
julia> createlist()
julia> createlist() # memory usage stays constant

I hoped that while using PythonCall.jl we could still have take advantage of the automatic memory management that makes languages like Python and Julia so convenient to use.

My question is: is it expected behavior that some Python objects (like NumPy arrays) are never collected unless the user manually does it? Or am I using the package wrong somehow?

2

It’s been a while since I used PythonCall (and I really used it the other way with juliacall), and I don’t pretend I know how Julia, Python, or PythonCall works on any deep level. But considering the basics:

  1. CPython counts references to promptly free non-cyclic garbage, which is most of it in programs with numerical arrays.
  2. CPython and Julia have tracing GCs triggered by allocation thresholds. I don’t know the details of how the number and sizes of allocations are considered, and the existing documentation is not necessarily up to date.
  3. In PythonCall, Julia wrapper objects keep Python objects alive by incrementing reference counts. GCing the wrapper objects decrements the counts.
  4. In juliacall, Python wrapper objects index a global vector referencing the Julia objects. GCing the wrapper objects kicks the Julia objects out of the vector.

We have a plausible explanation. In createarray, you make 1 small PyArray wrapper of a huge 10^8 NumPy matrix. The Julia side didn’t allocate the matrix, so it has no idea how big it is. The repeated calls discarded a handful of wrappers at most, not enough to spur Julia’s GC. Underinformed GC delays are unfortunately very familiar for interop in general. I can’t rule out some interaction with recent changes to CPython’s reference counting and GC, but I doubt downgrading Python helps.

In createlist, random.random() -> float is converted to Float64, so each float value imminently loses all references. You could make a version that makes Python lists, I’d expect it to run into the same issue but with PyList.

3

Thank you very much for your informative reply, what you are saying makes a lot of sense to me and is very helpful.

If your explanation is right (and I understood it correctly), then I would expect this problem to appear in general anytime you are trying to use a number of large Python objects, where “a number” and “large” means enough to fill up your memory before the corresponding PyArray wrappers can trigger Julia’s GC. Essentially, if you are using large Python objects, you cannot rely on them being garbage collected appropriately, because Julia actually “thinks they are small objects” (because it only sees the corresponding wrapper).

I also imagine that if Python’s allocations could somehow contribute to reaching the threshold needed for triggering Julia’s GC, then this problem would not happen. Do you think some sort of solution on these lines could exist, or is it technically not feasible?

4

Indeed, this version with Python lists has the same problem

julia> @pyexec """
       global random
       import random

       def createlistp():
         return [random.random() for _ in range(10000000)]
       """ => createlistp

julia> function createlistj()
         a = createlistp()
         return nothing
       end

julia> createlistj()

julia> createlistj()

# ... and so on, memory usage always goes up
# but manual call to GC.gc() works

while, interestingly, a version that calls directly PyList does not

julia> function createlistj2()
         a = PyList([rand() for _ in 1:10000000])
         return nothing
       end

julia> createlistj2()
julia> createlistj2()
# ... and so on, memory usage stays roughly constant