Julia crashed when using PyCall in multiple threads

General Usage
1

I first make multiple copies of a PyObject, and then access them in multiple threads. Finally, I meet this error.
julia -t5

using PyCall

py"""
class A():
    x = 0.0
a = A()
"""

alist = [deepcopy(py"a") for i in 1:10]
# @show alist

for j in 1:100
    Threads.@threads for i in 1:10
        alist[i].x
    end
end

error:

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7fff61685962 -- port with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7fff61685962 --  at 0x7fff61685962 -- ION with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7fff61685962 --  at 0x7fff61685962 -- ION with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7fff61685962 --  at 0x7fff61685962 --  at 0x7fff61685962 -- ION with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7fff61685962 -- PyObject_GenericGetAttrWithDict at C:\Users\wly\.julia\conda\3\x86_64\python310.dll (unknown line)

Julia version

Julia Version 1.9.3
Commit bed2cd540a1 (2023-08-24 14:43 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, goldmont)

Thanks.

2

I find a similar question about this https://juliacommunity.github.io/zulip-archive/stream/274208-helpdesk-(published)/topic/using.20PyCall.20object.20in.20threads.html, which gives some reason but does not provide a possible solution.

3

Can I suggest using PythonCall.jl? It seems to be the more mature option for calling Python from Julia, and it has some notes on multithreading FAQ & Troubleshooting · PythonCall & JuliaCall?

1 Like
4

This is know to fail.

1 Like
5

Is PythonCall/JuliaCall thread safe?
No.
Some rules if you are writing multithreaded code:

  • Only call Python functions from the first thread.

Thanks for your suggestion. It seems that it also does not realize this.

1 Like
6

I mean, Python itself has a global interpreter lock and cannot handle more than one thread in parallel… There is not much anybody can do about it… Either you serialize the calls to Python, or you create multiple Julia processes calling multiple instances of the Python interpreter…

1 Like
7

What is serialize the calls to Python? Can you give me some examples or links?
Thanks.

8

It means you use a lock such that only one thread of your Julia program can access your Python call at any given time…

I did not try this yet, it might work, but it also might mean that you need a handler in the main thread that communicates with your other threads and calls Python…

9

Stumbled across this very problem and after some searching found the following solution:

using PythonCall
using ThreadPools

macro pythread(expr)
    quote
        fetch(@tspawnat 1 begin
            $(esc(expr))
        end)
    end
end

function babel_convert(input, informat = "cdxml", outformat = "mol")
    @pythread begin
        ob = pyimport("openbabel.openbabel")
        conv = ob.OBConversion()
        conv.SetInAndOutFormats(informat, outformat)
        
        mol = ob.OBMol()
        conv.ReadString(mol, input)
        
        if ! Bool(mol.Has2D()) && ! Bool(mol.Has3D())
            pgen = ob.OBOp.FindType("gen2D")
            pgen !== nothing && pgen.Do(mol)
        end

        pyconvert(String, conv.WriteString(mol))
    end
end

PythonCall.GC.disable()
Threads.@threads for i in 1:10000
    babel_convert(cdxml)
end
PythonCall.GC.enable()
1 Like
Run multiple python instances with pycall in different threads
10

Just tried to use @pythread in a more complicated scenario and found that it essential to call PythonCall.GC.isable() from the Main thread as well.
As this is not always easy to do, it may be favorable to define @pythread as follows:

macro pythread(expr)
    quote
        fetch(@tspawnat 1 begin
            PythonCall.GC.disable()
            res = $(esc(expr))
            PythonCall.GC.enable()
            res
        end)
    end
end
11

After updating to Julia 1.12.4, the julia kernel is crashing inconsistently on some PyPlot calls with the error:

Fatal Python error: PyThreadState_Get: the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL) Python runtime state: initialized Thread 0x00000001f6bc3ac0 (most recent call first): File "~/.julia/conda/3/aarch64/lib/python3.10/site-packages/PIL/ImageFile.py", line 559 in _encode_tile File "~/.julia/conda/3/aarch64/lib/python3.10/site-packages/PIL/ImageFile.py", line 540 in _save File "~/.julia/conda/3/aarch64/lib/python3.10/site-packages/PIL/PngImagePlugin.py", line 1412 in _save File "~/.julia/conda/3/aarch64/lib/python3.10/site-packages/PIL/Image.py", line 2459 in save File "~/.julia/conda/3/aarch64/lib/python3.10/site-packages/matplotlib/image.py", line 1656 in imsave File "~/.julia/conda/3/aarch64/lib/python3.10/site-packages/matplotlib/backends/backend_agg.py", line 446 in _print_pil File "~/.julia/conda/3/aarch64/lib/python3.10/site-packages/matplotlib/backends/backend_agg.py", line 497 in print_png File "~/.julia/conda/3/aarch64/lib/python3.10/site-packages/matplotlib/backend_bases.py", line 2043 in <lambda> File "~/.julia/conda/3/aarch64/lib/python3.10/site-packages/matplotlib/backend_bases.py", line 2193 in print_figure [5469] signal 6: Abort trap: 6 in expression starting at none:1

The Kernel crashed while executing code in the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure. Click [here](https://aka.ms/vscodeJupyterKernelCrash) for more info. View Jupyter [log](command:jupyter.viewOutput) for further details.

The vscode advice and logs are very generic, but the upper part of the message suggests the julia kernel or its vscode extensions are triggering errors for using python/PyCall in a non-thread-safe way.

12

You have three options:

  • use PythonCall/ PythonPlot instead of PyCall/PyPlot (now threadsafe, not in the past)
  • run Julia with julia -t 1,0 (single threaded)
  • do the plotting in a separate, single-threaded process
13

Thanks, ufechner7. I am running a Jupyter notebook in vscode. It is using a single-threaded IJulia kernel, but perhaps julia 1.12.4 or PyCall is spawning multiple threads somehow.

The problem might be my reuse of PyPlot figures, which I just clear in a new cell with clf() and keep plotting.
I am now using close(gcf()) instead of clf(). It seems to work, but it’s hard to test for this intermittent problem.

I’ll also try migrating to PythonCall and PythonPlot.

14

Running julia -t 1 results in Julia using two threads (from version 1.12 onwards), one normal and one interactive thread. Only if you run julia -t 1,0 does it really use only one thread.

1 Like
15

I started getting strange VSCode crashes too, especially when PyPlot (installed through Conda/PyCall) plots were open (julia 1.12.4, VScode version code-1.108.2-1769004860, Julia extension 1.176.2 on linux). I’d get an error message which frustratingly disappeared as soon as the crash happened, so I resorted to screenshotting waiting for a crash … It seems to be something to do with Revise.jl and PyObjects, from the incomplete screen grab. Downgrading to Revise 3.13.0 (I had run update which brought me to 3.13.1 yesterday) or ensuring Revise is not loaded on VSCode startup seems to fix this – Phew!

2 Likes