How to wrap a Python function in a Julia package using PythonCall.jl

Dale_James_Black · February 23, 2023, 7:59pm

I am developing a package that wants to wrap and re-export a Python function. The relevant code is below, and it works well within the same file it was called in, but it breaks when trying to test it in the package I am developing. Is there a way to use PythonCall to execute a Python function on a Julia array under the hood? Is there something I am doing that is noticeably wrong?

This works within the same file:

## /src/scipy.jl
using Pkg
Pkg.activate("..")
using CondaPkg; CondaPkg.add("scipy")

scipy = pyimport("scipy")
ndimage = scipy.ndimage

function transform(array, tfm::Scipy)
	return pyconvert(Array{Float32}, ndimage.distance_transform_edt(array))
end

x = [
	1 1 0 0
	0 1 1 0
	0 1 0 1
	0 1 0 0
]
test = transform(x, Scipy())

This is broken:

## /test/scipy.jl
x = [
	1 1 0 0
	0 1 1 0
	0 1 0 1
	0 1 0 0
]
test = transform(x, Scipy())

Jake · February 23, 2023, 8:10pm

I am not a Python user nor have I ever done this. But I have used odsIO.jl very successfully, so there may be clues in that package.

Dale_James_Black · February 23, 2023, 8:17pm

Test segfaults with this error

signal (11): Segmentation fault: 11
in expression starting at /Users/daleblack/Library/CloudStorage/GoogleDrive-djblack@uci.edu/My Drive/dev/julia/DistanceTransforms/test/scipy2.jl:4
PyObject_GetAttr at /private/var/folders/t3/_k26tgtj7cv96l4vy3pxk5nw0000gn/T/jl_xyJmot/.CondaPkg/env/lib/libpython3.11.dylib (unknown line)
PyObject_GetAttr at /Users/daleblack/.julia/packages/PythonCall/ZzOaq/src/cpython/pointers.jl:299 [inlined]
pygetattr at /Users/daleblack/.julia/packages/PythonCall/ZzOaq/src/abstract/object.jl:60
getproperty at /Users/daleblack/.julia/packages/PythonCall/ZzOaq/src/Py.jl:272
unknown function (ip: 0x2a1d1c09b)
ijl_apply_generic at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
transform at /Users/daleblack/Library/CloudStorage/GoogleDrive-djblack@uci.edu/My Drive/dev/julia/DistanceTransforms/src/scipy.jl:39
unknown function (ip: 0x2a1d100e7)
ijl_apply_generic at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
do_call at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
eval_body at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
eval_body at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
eval_body at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_interpret_toplevel_thunk at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_toplevel_eval_flex at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_toplevel_eval_flex at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_toplevel_eval_flex at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
ijl_toplevel_eval_in at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
eval at ./boot.jl:368 [inlined]
include_string at ./loading.jl:1428
ijl_apply_generic at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
_include at ./loading.jl:1488
include at ./client.jl:476
unknown function (ip: 0x100174057)
ijl_apply_generic at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
do_call at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
eval_body at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_interpret_toplevel_thunk at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_toplevel_eval_flex at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_toplevel_eval_flex at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
ijl_toplevel_eval_in at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
eval at ./boot.jl:368 [inlined]
include_string at ./loading.jl:1428
ijl_apply_generic at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
_include at ./loading.jl:1488
include at ./client.jl:476
unknown function (ip: 0x100174057)
ijl_apply_generic at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
do_call at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
eval_body at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_interpret_toplevel_thunk at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_toplevel_eval_flex at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_toplevel_eval_flex at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
ijl_toplevel_eval_in at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jlplt_ijl_toplevel_eval_in_13487 at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
eval at ./boot.jl:368 [inlined]
exec_options at ./client.jl:276
_start at ./client.jl:522
jfptr__start_59061 at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
ijl_apply_generic at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
true_main at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_repl_entrypoint at /Users/daleblack/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/lib/julia/libjulia-internal.1.8.dylib (unknown line)
Allocations: 72936179 (Pool: 72902612; Big: 33567); GC: 40
ERROR: Package DistanceTransforms errored during testing (exit code: 139)

Dale_James_Black · February 23, 2023, 8:18pm

Thanks, that might have some useful tips but I don’t see anything specific as of right now. Nice simple package though and uses PyCall, which is similar I think!

GunnarFarneback · February 23, 2023, 9:03pm

With PyCall the key thing is to do the pyimport in the __init__() function of the module. Precompilation saves memory pointers to Python objects otherwise, which won’t be valid in future sessions. I expect PythonCall behaves the same in this respect.

Dale_James_Black · February 23, 2023, 9:34pm

Okay, that makes some sense. I don’t see any documentation though for PythonCall. So with PyCall, if you have “PackageX” you would would create an __init__() function inside of that main Module e.g. (PackageX.jl) or could if be in any src file?

Dale_James_Black · February 23, 2023, 9:37pm

This looks like it might be the answer PyMNE.jl/PyMNE.jl at main · beacon-biosignals/PyMNE.jl · GitHub

Dale_James_Black · February 23, 2023, 9:47pm

Adding this code inside the main module still results in a segfault

module DistanceTransforms
using PythonCall

const scipy = PythonCall.pynew()

function __init__()
    PythonCall.pycopy!(scipy, pyimport("scipy"))
end

export scipy

cjdoris · February 23, 2023, 10:27pm

Can you post a full MWE please?

Dale_James_Black · February 24, 2023, 9:40pm

Yes, I just put together a simple example package (GitHub - Dale-Black/DT.jl). This segfaults whenever trying to precompile and when trying to run the test. Thanks btw for the amazing package @cjdoris

GunnarFarneback · February 25, 2023, 8:41am

ndimage = scipy.ndimage

function transform(array, tfm::Scipy)
    return pyconvert(Array{Float32}, ndimage.distance_transform_edt(array))
end

Be careful with all globals related to Python objects, as they get saved by the precompilation with stale pointers. Does it work better if you eliminate ndimage and call scipy.ndimage.distance_transform_edt?

Dale_James_Black · February 25, 2023, 5:14pm

Oh that looks like it fixed the problem! One more question, any recommended way to automatically install python packages like scipy inside of the Julia package if not already installed? I added this but I am guessing it’s not recommended

module DT

using CondaPkg
CondaPkg.add("scipy")

using PythonCall

const scipy = PythonCall.pynew()

function __init__()
    PythonCall.pycopy!(scipy, pyimport("scipy"))
end

export scipy

include("scipy.jl")

end

cjdoris · February 25, 2023, 7:15pm

You just ship the CondaPkg.toml file that is created with your package. Any dependencies in there are automatically installed.

Dale_James_Black · May 19, 2023, 2:06am

Hey, just double checking. So I don’t need the CondaPkg dependency for this, right?

Dale_James_Black · May 19, 2023, 2:08am

As long as I have the CondaPkg.toml file, then PythonCall will handle the rest?

cjdoris · May 19, 2023, 7:10pm

Yes, any package which depends on CondaPkg either directly or indirectly will have it’s CondaPkg.toml dependencies automatically installed.

Since PythonCall depends on CondaPkg, it’s enough to depend only on PythonCall.