Hi all,
I am having issues dereferencing Ptr{Cvoid} inside of a shared object library. I suspect it is due to garbage collection.
I am working on a Python project that requires a Julia library that does not work with PyCall. I am using PackageCompiler.jl to compile .jl into .so which I call using ctypes in python.
As a proof of concept I have tried to parse a sparse matrix, create a reference to it, and then pass the resulting pointer back to python. In the below code this is called pointers_to_sparse and it works fine.
Rather than parse the sparse array every time I really would rather just hold onto the pointer to a Julia object and dereference it when called. I do this with open_jl_sparse. In my experiments I am able to pass the correct pointer hex address back and forth and I am able to deref it without a segfault. but when i try to actually get to the values stored in the ref, I get a segfault. This is on the line in the below code: vals=arrayref[];
module laplacians_so_src
using Base
using SparseArrays
Base.@ccallable function pointers_to_sparse(dataptr::Ptr{Cdouble},rowptr::Ptr{Cint},colptr::Ptr{Cint},numel::Cint)::Ptr{Cvoid}
# given pointer to a data vector, assign elements
# to the rows and columns of a sparse matrix
# indexed by the vector at rowptr and colptr.
data = unsafe_wrap(Array, dataptr,(numel,),own=true);
column_idx = unsafe_wrap(Array, colptr, (numel,),own=true);
row_idx = unsafe_wrap(Array, rowptr, (numel,),own=true);
array = sparse(row_idx,column_idx,data);
println(array); #for debugging to see that the array is correct.
arrayptr = pointer_from_objref(Ref(array));
println(arrayptr); #for debugging to compare to the address of what is printed in open_jl_sparse
return arrayptr
end
Base.@ccallable function open_jl_sparse(arrayptr::Ptr{Cvoid})::Cint
#take a Ptr{CVoid} that references a Julia Ref{} and unpack it.
println(arrayptr); #debug
arrayref = unsafe_pointer_to_objref(arrayptr);
vals = arrayref[]; #segfault happens here.
println(vals);
return 0
end
end
Here is what the python code that calls this looks like. Notice that I turned off the garbage collector in order to see that helps. I do not think it is a ctypes problem as when I remove the vals line from above I dont get segfaults.
import ctypes
from numpy.ctypeslib import ndpointer
import numpy as np
from scipy.sparse import random
#make synthetic data
x = random(20,5,0.2)
dat = x.data.astype(ctypes.c_double)
col_idx = (x.nonzero()[1]+1).astype(ctypes.c_int)
row_idx = (x.nonzero()[0]+1).astype(ctypes.c_int)
numel = x.nnz
#load and configure the .so
lib = ctypes.CDLL(b"/full/path/to/builddir/laplacians_so_src.so", ctypes.RTLD_GLOBAL)
lib.jl_init_with_image__threading.argtypes = (ctypes.c_char_p, ctypes.c_char_p)
lib.jl_init_with_image__threading(None, b"/full/path/to/builddir/laplacians_so_src.so")
#turn off jl garbage collector
lib.jl_gc_enable(0)
#setup the jl function for making a sparse matrix pointer
ptr_to_sparse = lib.pointers_to_sparse
ptr_to_sparse.restype = ctypes.POINTER(ctypes.c_void_p)
ptr_to_sparse.argtypes =[ndpointer(ctypes.c_double, flags="C_CONTIGUOUS"),ndpointer(ctypes.c_int, flags="C_CONTIGUOUS"),ndpointer(ctypes.c_int, flags="C_CONTIGUOUS"),ctypes.c_int]
# set up the jl function for derefing a Ptr{Void} to a Julia Ref
ptr_open = lib.open_jl_sparse
ptr_open.argtypes = [ctypes.POINTER(ctypes.c_void_p)]
ptr_open.restype = ctypes.c_int
# Obtain a pointer to a Julia Ref{SparseMatrixCSC{Cdouble,Cint}}
# by parsing a sparse scipy matrix using COO format
y = ptr_to_sparse(dat,row_idx,col_idx,ctypes.c_int(numel)) #this works
# Try to deref it in Julia
ptr_open(y) # this is where the segfault is
The Julia code is compiled using juliac.jl -vas laplacians_so_src.jl
Any tips? Given the way that it is dereferencing and only failing when I actually try to index it with [] it seems to me that maybe the address is getting cleared. Additionally, the pointer returned by pointers_to_sparse is pointing to the same hex address as the one received by open_jl_sparse, so again it seems like that address is missing something.