Julia crashed when using PyCall in multiple threads

General Usage
1

I first make multiple copies of a PyObject, and then access them in multiple threads. Finally, I meet this error.
julia -t5

using PyCall

py"""
class A():
    x = 0.0
a = A()
"""

alist = [deepcopy(py"a") for i in 1:10]
# @show alist

for j in 1:100
    Threads.@threads for i in 1:10
        alist[i].x
    end
end

error:

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7fff61685962 -- port with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7fff61685962 --  at 0x7fff61685962 -- ION with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7fff61685962 --  at 0x7fff61685962 -- ION with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7fff61685962 --  at 0x7fff61685962 --  at 0x7fff61685962 -- ION with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7fff61685962 -- PyObject_GenericGetAttrWithDict at C:\Users\wly\.julia\conda\3\x86_64\python310.dll (unknown line)

Julia version

Julia Version 1.9.3
Commit bed2cd540a1 (2023-08-24 14:43 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, goldmont)

Thanks.

2

I find a similar question about this https://juliacommunity.github.io/zulip-archive/stream/274208-helpdesk-(published)/topic/using.20PyCall.20object.20in.20threads.html, which gives some reason but does not provide a possible solution.

3

Can I suggest using PythonCall.jl? It seems to be the more mature option for calling Python from Julia, and it has some notes on multithreading FAQ & Troubleshooting · PythonCall & JuliaCall?

1 Like
4

This is know to fail.

1 Like
5

Is PythonCall/JuliaCall thread safe?
No.
Some rules if you are writing multithreaded code:

  • Only call Python functions from the first thread.

Thanks for your suggestion. It seems that it also does not realize this.

1 Like
6

I mean, Python itself has a global interpreter lock and cannot handle more than one thread in parallel… There is not much anybody can do about it… Either you serialize the calls to Python, or you create multiple Julia processes calling multiple instances of the Python interpreter…

1 Like
7

What is serialize the calls to Python? Can you give me some examples or links?
Thanks.

8

It means you use a lock such that only one thread of your Julia program can access your Python call at any given time…

I did not try this yet, it might work, but it also might mean that you need a handler in the main thread that communicates with your other threads and calls Python…

9

Stumbled across this very problem and after some searching found the following solution:

using PythonCall
using ThreadPools

macro pythread(expr)
    quote
        fetch(@tspawnat 1 begin
            $(esc(expr))
        end)
    end
end

function babel_convert(input, informat = "cdxml", outformat = "mol")
    @pythread begin
        ob = pyimport("openbabel.openbabel")
        conv = ob.OBConversion()
        conv.SetInAndOutFormats(informat, outformat)
        
        mol = ob.OBMol()
        conv.ReadString(mol, input)
        
        if ! Bool(mol.Has2D()) && ! Bool(mol.Has3D())
            pgen = ob.OBOp.FindType("gen2D")
            pgen !== nothing && pgen.Do(mol)
        end

        pyconvert(String, conv.WriteString(mol))
    end
end

PythonCall.GC.disable()
Threads.@threads for i in 1:10000
    babel_convert(cdxml)
end
PythonCall.GC.enable()
1 Like
10

Just tried to use @pythread in a more complicated scenario and found that it essential to call PythonCall.GC.isable() from the Main thread as well.
As this is not always easy to do, it may be favorable to define @pythread as follows:

macro pythread(expr)
    quote
        fetch(@tspawnat 1 begin
            PythonCall.GC.disable()
            res = $(esc(expr))
            PythonCall.GC.enable()
            res
        end)
    end
end