Passing Julia pointers back and forth from Julia PackageCompiled shared object libraries

General Usage
1

Hi all,

I am having issues dereferencing Ptr{Cvoid} inside of a shared object library. I suspect it is due to garbage collection.

I am working on a Python project that requires a Julia library that does not work with PyCall. I am using PackageCompiler.jl to compile .jl into .so which I call using ctypes in python.

As a proof of concept I have tried to parse a sparse matrix, create a reference to it, and then pass the resulting pointer back to python. In the below code this is called pointers_to_sparse and it works fine.

Rather than parse the sparse array every time I really would rather just hold onto the pointer to a Julia object and dereference it when called. I do this with open_jl_sparse. In my experiments I am able to pass the correct pointer hex address back and forth and I am able to deref it without a segfault. but when i try to actually get to the values stored in the ref, I get a segfault. This is on the line in the below code: vals=arrayref[];

module laplacians_so_src
using Base
using SparseArrays

Base.@ccallable function pointers_to_sparse(dataptr::Ptr{Cdouble},rowptr::Ptr{Cint},colptr::Ptr{Cint},numel::Cint)::Ptr{Cvoid}
# given pointer to a data vector, assign elements 
# to the rows and columns of a sparse matrix
# indexed by the vector at rowptr and colptr.
    data = unsafe_wrap(Array, dataptr,(numel,),own=true);
    column_idx = unsafe_wrap(Array, colptr, (numel,),own=true);
    row_idx = unsafe_wrap(Array, rowptr, (numel,),own=true);
    array = sparse(row_idx,column_idx,data);
    println(array); #for debugging to see that the array is correct.
    arrayptr = pointer_from_objref(Ref(array));
    println(arrayptr); #for debugging to compare to the address of what is printed in open_jl_sparse
    return arrayptr
end

Base.@ccallable function open_jl_sparse(arrayptr::Ptr{Cvoid})::Cint
#take a Ptr{CVoid} that references a Julia Ref{} and unpack it.
    println(arrayptr); #debug
    arrayref = unsafe_pointer_to_objref(arrayptr);
    vals = arrayref[]; #segfault happens here.
    println(vals);
    return 0
end
end

Here is what the python code that calls this looks like. Notice that I turned off the garbage collector in order to see that helps. I do not think it is a ctypes problem as when I remove the vals line from above I dont get segfaults.


import ctypes
from numpy.ctypeslib import ndpointer
import numpy as np
from scipy.sparse import random

#make synthetic data
x = random(20,5,0.2)
dat = x.data.astype(ctypes.c_double)
col_idx = (x.nonzero()[1]+1).astype(ctypes.c_int)
row_idx = (x.nonzero()[0]+1).astype(ctypes.c_int)
numel = x.nnz

#load and configure the .so
lib = ctypes.CDLL(b"/full/path/to/builddir/laplacians_so_src.so", ctypes.RTLD_GLOBAL)
lib.jl_init_with_image__threading.argtypes = (ctypes.c_char_p, ctypes.c_char_p)
lib.jl_init_with_image__threading(None, b"/full/path/to/builddir/laplacians_so_src.so")
#turn off jl garbage collector
lib.jl_gc_enable(0)

#setup the jl function for making a sparse matrix pointer
ptr_to_sparse = lib.pointers_to_sparse
ptr_to_sparse.restype = ctypes.POINTER(ctypes.c_void_p)
ptr_to_sparse.argtypes =[ndpointer(ctypes.c_double, flags="C_CONTIGUOUS"),ndpointer(ctypes.c_int, flags="C_CONTIGUOUS"),ndpointer(ctypes.c_int, flags="C_CONTIGUOUS"),ctypes.c_int]

# set up the jl function for derefing a Ptr{Void} to a Julia Ref
ptr_open = lib.open_jl_sparse
ptr_open.argtypes = [ctypes.POINTER(ctypes.c_void_p)]
ptr_open.restype = ctypes.c_int

# Obtain a pointer to a Julia Ref{SparseMatrixCSC{Cdouble,Cint}} 
# by parsing a sparse scipy matrix using COO format
y = ptr_to_sparse(dat,row_idx,col_idx,ctypes.c_int(numel)) #this works
# Try to deref it in Julia
ptr_open(y) # this is where the segfault is

The Julia code is compiled using juliac.jl -vas laplacians_so_src.jl

Any tips? Given the way that it is dereferencing and only failing when I actually try to index it with [] it seems to me that maybe the address is getting cleared. Additionally, the pointer returned by pointers_to_sparse is pointing to the same hex address as the one received by open_jl_sparse, so again it seems like that address is missing something.

2

For the future:

No need to use Ptr{Cvoid} here. You can simply use the Julia Ref{} and then treat it as a c_void_p in python. Julia can deref it then. Here’s the new julia code.

module laplacians_so_src
using Base
using SparseArrays

Base.@ccallable function pointers_to_sparse(dataptr::Ptr{Cdouble},rowptr::Ptr{Cint},colptr::Ptr{Cint},numel::Cint)::Ref{}
# given pointer to a data vector, assign elements 
# to the rows and columns of a sparse matrix
# indexed by the vector at rowptr and colptr.
    data = unsafe_wrap(Array, dataptr,(numel,),own=true);
    column_idx = unsafe_wrap(Array, colptr, (numel,),own=true);
    row_idx = unsafe_wrap(Array, rowptr, (numel,),own=true);
    array = sparse(row_idx,column_idx,data);
    println(array); #for debugging to see that the array is correct.
    arrayptr = (Ref(array));
    println(arrayptr); #for debugging to compare to the address of what is printed in open_jl_sparse
    return arrayptr
end

Base.@ccallable function open_jl_sparse(arrayptr::Ref{})::Cint
#take a Ptr{CVoid} that references a Julia Ref{} and unpack it.
    println(arrayptr); #debug
    arrayval = arrayptr[];
    println(arrayval)
    return 0
end
end