Julia crashed when using PyCall in multiple threads

General Usage
1

I first make multiple copies of a PyObject, and then access them in multiple threads. Finally, I meet this error.
julia -t5

using PyCall

py"""
class A():
    x = 0.0
a = A()
"""

alist = [deepcopy(py"a") for i in 1:10]
# @show alist

for j in 1:100
    Threads.@threads for i in 1:10
        alist[i].x
    end
end

error:

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7fff61685962 -- port with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7fff61685962 --  at 0x7fff61685962 -- ION with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7fff61685962 --  at 0x7fff61685962 -- ION with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7fff61685962 --  at 0x7fff61685962 --  at 0x7fff61685962 -- ION with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7fff61685962 -- PyObject_GenericGetAttrWithDict at C:\Users\wly\.julia\conda\3\x86_64\python310.dll (unknown line)

Julia version

Julia Version 1.9.3
Commit bed2cd540a1 (2023-08-24 14:43 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, goldmont)

Thanks.

2

I find a similar question about this https://juliacommunity.github.io/zulip-archive/stream/274208-helpdesk-(published)/topic/using.20PyCall.20object.20in.20threads.html, which gives some reason but does not provide a possible solution.

3

Can I suggest using PythonCall.jl? It seems to be the more mature option for calling Python from Julia, and it has some notes on multithreading FAQ & Troubleshooting · PythonCall & JuliaCall?

1 Like
4

This is know to fail.

1 Like
5

Is PythonCall/JuliaCall thread safe?
No.
Some rules if you are writing multithreaded code:

  • Only call Python functions from the first thread.

Thanks for your suggestion. It seems that it also does not realize this.

1 Like
6

I mean, Python itself has a global interpreter lock and cannot handle more than one thread in parallel… There is not much anybody can do about it… Either you serialize the calls to Python, or you create multiple Julia processes calling multiple instances of the Python interpreter…

7

What is serialize the calls to Python? Can you give me some examples or links?
Thanks.

8

It means you use a lock such that only one thread of your Julia program can access your Python call at any given time…

I did not try this yet, it might work, but it also might mean that you need a handler in the main thread that communicates with your other threads and calls Python…