How to wrap a Python function in a Julia package using PythonCall.jl

General Usage
1

I am developing a package that wants to wrap and re-export a Python function. The relevant code is below, and it works well within the same file it was called in, but it breaks when trying to test it in the package I am developing. Is there a way to use PythonCall to execute a Python function on a Julia array under the hood? Is there something I am doing that is noticeably wrong?

This works within the same file:

## /src/scipy.jl
using Pkg
Pkg.activate("..")
using CondaPkg; CondaPkg.add("scipy")

scipy = pyimport("scipy")
ndimage = scipy.ndimage

function transform(array, tfm::Scipy)
	return pyconvert(Array{Float32}, ndimage.distance_transform_edt(array))
end

x = [
	1 1 0 0
	0 1 1 0
	0 1 0 1
	0 1 0 0
]
test = transform(x, Scipy())

This is broken:

## /test/scipy.jl
x = [
	1 1 0 0
	0 1 1 0
	0 1 0 1
	0 1 0 0
]
test = transform(x, Scipy())
2

I am not a Python user nor have I ever done this. But I have used odsIO.jl very successfully, so there may be clues in that package.

3

Test segfaults with this error

signal (11): Segmentation fault: 11
in expression starting at /Users/daleblack/Library/CloudStorage/GoogleDrive-djblack@uci.edu/My Drive/dev/julia/DistanceTransforms/test/scipy2.jl:4
PyObject_GetAttr at /private/var/folders/t3/_k26tgtj7cv96l4vy3pxk5nw0000gn/T/jl_xyJmot/.CondaPkg/env/lib/libpython3.11.dylib (unknown line)
PyObject_GetAttr at /Users/daleblack/.julia/packages/PythonCall/ZzOaq/src/cpython/pointers.jl:299 [inlined]
pygetattr at /Users/daleblack/.julia/packages/PythonCall/ZzOaq/src/abstract/object.jl:60
getproperty at /Users/daleblack/.julia/packages/PythonCall/ZzOaq/src/Py.jl:272
unknown function (ip: 0x2a1d1c09b)
ijl_apply_generic at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
transform at /Users/daleblack/Library/CloudStorage/GoogleDrive-djblack@uci.edu/My Drive/dev/julia/DistanceTransforms/src/scipy.jl:39
unknown function (ip: 0x2a1d100e7)
ijl_apply_generic at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
do_call at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
eval_body at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
eval_body at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
eval_body at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_interpret_toplevel_thunk at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_toplevel_eval_flex at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_toplevel_eval_flex at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_toplevel_eval_flex at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
ijl_toplevel_eval_in at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
eval at ./boot.jl:368 [inlined]
include_string at ./loading.jl:1428
ijl_apply_generic at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
_include at ./loading.jl:1488
include at ./client.jl:476
unknown function (ip: 0x100174057)
ijl_apply_generic at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
do_call at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
eval_body at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_interpret_toplevel_thunk at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_toplevel_eval_flex at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_toplevel_eval_flex at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
ijl_toplevel_eval_in at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
eval at ./boot.jl:368 [inlined]
include_string at ./loading.jl:1428
ijl_apply_generic at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
_include at ./loading.jl:1488
include at ./client.jl:476
unknown function (ip: 0x100174057)
ijl_apply_generic at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
do_call at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
eval_body at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_interpret_toplevel_thunk at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_toplevel_eval_flex at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_toplevel_eval_flex at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
ijl_toplevel_eval_in at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jlplt_ijl_toplevel_eval_in_13487 at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
eval at ./boot.jl:368 [inlined]
exec_options at ./client.jl:276
_start at ./client.jl:522
jfptr__start_59061 at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
ijl_apply_generic at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
true_main at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_repl_entrypoint at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
Allocations: 72936179 (Pool: 72902612; Big: 33567); GC: 40
ERROR: Package DistanceTransforms errored during testing (exit code: 139)
4

Thanks, that might have some useful tips but I don’t see anything specific as of right now. Nice simple package though and uses PyCall, which is similar I think!

5

With PyCall the key thing is to do the pyimport in the __init__() function of the module. Precompilation saves memory pointers to Python objects otherwise, which won’t be valid in future sessions. I expect PythonCall behaves the same in this respect.

6

Okay, that makes some sense. I don’t see any documentation though for PythonCall. So with PyCall, if you have “PackageX” you would would create an __init__() function inside of that main Module e.g. (PackageX.jl) or could if be in any src file?

7

This looks like it might be the answer PyMNE.jl/PyMNE.jl at main · beacon-biosignals/PyMNE.jl · GitHub

8

Adding this code inside the main module still results in a segfault

module DistanceTransforms
using PythonCall

const scipy = PythonCall.pynew()

function __init__()
    PythonCall.pycopy!(scipy, pyimport("scipy"))
end

export scipy
10

Can you post a full MWE please?

11

Yes, I just put together a simple example package (GitHub - Dale-Black/DT.jl). This segfaults whenever trying to precompile and when trying to run the test. Thanks btw for the amazing package @cjdoris

12 
ndimage = scipy.ndimage

function transform(array, tfm::Scipy)
    return pyconvert(Array{Float32}, ndimage.distance_transform_edt(array))
end

Be careful with all globals related to Python objects, as they get saved by the precompilation with stale pointers. Does it work better if you eliminate ndimage and call scipy.ndimage.distance_transform_edt?

2 Likes
13

Oh that looks like it fixed the problem! One more question, any recommended way to automatically install python packages like scipy inside of the Julia package if not already installed? I added this but I am guessing it’s not recommended

module DT

using CondaPkg
CondaPkg.add("scipy")

using PythonCall

const scipy = PythonCall.pynew()

function __init__()
    PythonCall.pycopy!(scipy, pyimport("scipy"))
end

export scipy

include("scipy.jl")

end
14

You just ship the CondaPkg.toml file that is created with your package. Any dependencies in there are automatically installed.

1 Like
15

Hey, just double checking. So I don’t need the CondaPkg dependency for this, right?

16

As long as I have the CondaPkg.toml file, then PythonCall will handle the rest?

17

Yes, any package which depends on CondaPkg either directly or indirectly will have it’s CondaPkg.toml dependencies automatically installed.

Since PythonCall depends on CondaPkg, it’s enough to depend only on PythonCall.

2 Likes
18

Thank you :slight_smile:

(OdsIO developer looking for PythonCall documentation and ended up on this thread :slight_smile: :slight_smile: :slight_smile: )