PyJulia multithreading segfaults

General Usage
, , ,
1

I am wondering if the Julia community has any suggestions for importing code from Julia to Python in some manner other than the PyJulia package or suggestions on how to properly use that package as the GitHub seems to be inactive.

I imagine a lot of the advice will be to rewrite the code in Julia, and while I’d like to do that eventually, I’d prefer to have a working solution soon and know this is not a Julia bug.

I am suffering from a segmentation fault 11 error when multithreading using joblib.Parallel to run a small Julia function.

I import a function in Python, modeled after this attempted solution

from julia.api import Julia
jl = Julia(compiled_modules=False)
from julia import Main
Main.include("fastsum.jl")
from julia.Main import greenfunction

where the function itself is

using Tullio, LoopVectorization
function greenfunction(mu, wns, sigwns, energy, dosnorm)
    new_array = Array{ComplexF64}(undef, length(wns))
    @tullio threads=false new_array[i] = 1 / (mu + wns[i] * 1im - energy[j] - sigwns[i]) * dosnorm[j]
    dosnorm = Nothing
    energy  = Nothing
    sigwns  = Nothing
    wns     = Nothing
    
    return new_array
end

Clearing some of the variables seems to make the number of processes I can run increase from something around 10 to 80, but eventually it still segfaults, with error as

signal (15): Terminated
in expression starting at none:0
mul_fast at ./fastmath.jl:167 [inlined]
mul_fast at ./fastmath.jl:219 [inlined]
...
... ( a lot of PyCall directories)
...
unknown function (ip: (nil))
Allocations: 76861098 (Pool: 76845028; Big: 16070); GC: 116

signal (11): Segmentation fault
in expression starting at none:0
...
joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.

The more detailed errors are here

Is there a way to clear the Julia cache as this seems to be the problem? Or import Julia less frequently somehow? I call that import function several hundred times.

2

I would bet it’s a GIL problem. I don’t think PyCall / PyJulia handles the GIL at all.

Have you tried PythonCall / juliacall?

It would also be helpful to include a full MWE.