Floop threading and Juliacall produce segmentation fault

General Usage
, , ,
1

Hi! I’m trying to speed up some calculations using julia to call from python. I’m trying to use juliacall as follows:

  1. I have a julia threaded function to produce samples
#"file.jl"
using FLoops

function my_func(N)
    samples=Array{Any}(undef,N)
    @floop for i in 1:N
        samples[i]=sum(ones(100000)*i)
    end
    return samples
end
  1. I try to call it from python using
from juliacall import Main as jl
jl.include("/home/carlos/Documentos/Trabajo de grado/Tesis/New code/file.jl")
sa=jl.my_func(10000)
print(sa)

but it produces a segmentation fault error, even if it works with the original julia code in the REPL. Someone knows how to fix it?

1 Like
2

You have to disable the garbage collection while running the threaded Julia code. Something like this:

def mddf(*args, **kwargs) : 
     jl.GC.enable(False) 
     result = jl.cm.mddf(*args, **kwargs)  # this is multi-threaded in Julia 
     jl.GC.enable(True) 
     return result
5 Likes
3

Why is this necessary?

4
1 Like
5

Is this a problem even if you aren’t passing any python object to Julia? Does PyCall & Co have the same limitation? This sounds like a severe limitation in JuliaCall and PythonCall :confused:

6

It seems that there is something similar, although I didn’t go into the details to see if it is the same thing: thread safety · Issue #882 · JuliaPy/PyCall.jl · GitHub

7

Surprisingly it works without much effort using PyJulia, without disabling the garbage collector. However, I haven’t tested it works well enough yet, without bugs.

8

Sorry, it didn’t work, segmentation fault is still there

9

Uhm… it worked here, but note that your function allocates a lot, so I cannot run it here without having the GC turned on. Thus, I modified it such that it does not allocate as much. In that case, it worked:

In [1]: from juliacall import Main as jl

In [2]: jl.seval("using FLoops")

In [3]: jl.seval("""
   ...: function my_func(N)
   ...:     samples=Array{Float64}(undef,N)
   ...:     @floop for i in 1:N
   ...:         samples[i]=sum(i for i in 1:10^4)
   ...:     end
   ...:     return samples
   ...: end
   ...: """)
Out[3]: my_func (generic function with 1 method)

In [4]: jl.GC.enable(False)
Out[4]: True

In [5]: jl.my_func(10000)
Out[5]: 
10000-element Vector{Float64}:
 5.0005e7
...
 5.0005e7

In [9]: jl.my_func(10000)
Out[9]: 
10000-element Vector{Float64}:
 5.0005e7
...
 5.0005e7

In [10]: jl.my_func(10000)
Out[10]: 
10000-element Vector{Float64}:
 5.0005e7
...
 5.0005e7

In [11]: jl.my_func(10000)
Out[11]: 
10000-element Vector{Float64}:
 5.0005e7
...
 5.0005e7

In [12]: jl.my_func(10000)
Out[12]: 
10000-element Vector{Float64}:
 5.0005e7
...
 5.0005e7

In [13]: %timeit jl.my_func(10000)
43.4 µs ± 2.72 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [14]: jl.GC.enable(True)
Out[14]: False

In [15]: %timeit jl.my_func(10000)
Segmentation fault (core dumped)

I ran the function several times without GC, and didn’t get the segmentation fault, and then with it I get it. The %timeit macro (from ipython3) runs the function several times.

If I run your original function the section gets Killed because of lack of memory, but that’s another, albeit related, issue.

10

That is, this limitation does require that the threaded code is minimally allocating, or one can have memory issues given that GC is off on the Julia side.

Anyway, that is a workaround, it may not work in every case, and I’m not a specialist on it.

11

Does Python replace/override the fault handlers of Julia?

When multi-threaded GC uses read-protected pages to implement safepoints (e.g. to ask other threads to stop doing work). This causes a benign segmentation fault, but it must be handled by the Julia segmentation handler.

12

This would do it.

13

Sorry, can you be a bit more explicit? How can I change it? What would be different?/