Ccall with a C struct containing a pointer

General Usage
1

I’m writing a Julia interface for a C library. I have a C struct containing C-style array. I’ve wrapped said struct in Julia like so:

mutable struct Example
   data::Ptr{Int64} # in the C, this is an array.
end
x = Int64[1,2,3]
s = Example(pointer(x))
@ccall lib.someFunc(s::Example)::Cvoid #this works as desired

How do I create a Julia class to encapsulate this functionality? Writing functions that take the type Example as an argument seems ripe for memory issues: what if the array that data points to goes out of scope?

In C++, I’d have data point to an newed array allocated on the heap, which I then would delete at some point. From the docs on wrapping C and Fortran code in Julia, if I were calling a C function that took a C-style array, I would construct a myArray of type Vector{T} then pass myArray::Ref{T} into the @ccall. But the interface of the C function requires the struct: I cannot pass a bare array into the C function.

2

One option is to manually root the array during the @ccall, to make sure that it stays allocated until the call is complete, e.g.

GC.@preserve x @ccall lib.someFunc(s::Example)::Cvoid

Another option is to pad your struct on the Julia side with additional fields that store references to the Julia arrays, e.g.

mutable struct Example
    data::Ptr{Int64} # in the C, this is an array.
    _data::Vector{Int64} # keep a ref to the Julia array
    Example(d::Vector{Int64}) = new(pointer(d), d)
end
3

I think there’s a pattern involving cconvert and unsafe_convert here, with two structs. One for using on the Julia side, one for the actual FFI interop, which would only exist for the duration of the FFI call. I’m not 100% sure on the exact definitions necessary, so the following may not necessarily be correct:

// The C example just returns the first element of the array

typedef struct {
  int* data;
} CExample;

int printarr(CExample* ex) {
  return ex->data[0];
}

mutable struct JLExample
    data::Vector{Int64}
end

mutable struct CExample
    data::Ptr{Int64}
end

Base.cconvert(::Type{Ptr{CExample}}, jle::JLExample) = CExample(pointer(jle.data))

function Base.unsafe_convert(::Type{Ptr{CExample}}, ce::CExample)
    return convert(Ptr{CExample}, pointer_from_objref(ce))
end

The reason I’m not 100% sure on this is because of that pointer(jle.data) in cconvert. This is likely illegal - cconvert isn’t supposed to take pointers, that’s the job of unsafe_convert here. The manual seems self contradictory here, since it says this:

Base.cconvert normally just calls convert, but can be defined to return an arbitrary new object more appropriate for passing to C. This should be used to perform all allocations of memory that will be accessed by the C code. For example, this is used to convert an Array of objects (e.g. strings) to an array of pointers.

but also this:

This can be used to allocate memory that will be accessed by the ccall. If multiple objects need to be allocated, a tuple of the objects can be used as return value.

Neither convert nor cconvert should take a Julia object and turn it into a Ptr.

which seems to imply that the conversion of an Array of objects to an array of pointers would be illegal…

Either way, it would be used like so:

julia> function bar()
           ex = JLExample([15,2,3])
           GC.@preserve ex begin
               @ccall "./example.so".printarr(ex::Ref{CExample})::Cint
           end
       end
bar (generic function with 1 method)

julia> bar()
15

which works, but certainly feels a little ugly due to the need for that GC.@preserve. The GC.@preserve should make the definitions above safe, since ex (and by extension, the wrapped array) is not allowed to be freed until the end of that begin block. This shouldn’t be necessary though, and I’m pretty sure I’m missing the correct way of doing this :thinking: Would be really great to get such an example involving a more complicated struct into the docs…

4

Here’s one other variation that might be more appropriate, and could be safer by default, if I’m using the interface correctly:

mutable struct JLExample
    data::Vector{Int64}
end

struct CExample # not mutable!
    data::Ptr{Int64}
end

Base.cconvert(::Type{Ref{CExample}}, jle::JLExample) = (jle, Ref{CExample}())

function Base.unsafe_convert(::Type{Ref{CExample}}, ce::Tuple{JLExample, Base.RefValue{CExample}})
    jle, ref = ce
    ref[] = CExample(pointer(jle.data))
    return Base.unsafe_convert(Ptr{CExample}, ref)
end

Basically, this ensures the C-FFI-struct has a safe place to go, by making sure both the source object jle and the inline data is preserved. This should be the case, since the manual says that the result of cconvert will “behave as if passed to GC.@preserve”. Then, in unsafe_convert the actual construction of the immutable data being passed to C is performed. Since that object is stored in the already allocated Ref (and is itself isbits, so will be stored inline in the Ref), all that’s left to do is convert that Ref to a Ptr, which will be passed to the ccall. (I’m not exactly sure why this last unsafe_convert is needed, but without it Julia complained about expecting a Ptr…)

This also seems to be zero-overhead on the calling side:

julia> ex = JLExample([15,2,3])
JLExample([15, 2, 3])

julia> function bar(ex)
           @ccall "./example.so".printarr(ex::Ref{CExample})::Cint
       end
bar (generic function with 1 method)

julia> bar(ex)
15

# no allocations!!
julia> @time bar(ex)
  0.000002 seconds
15

I should also mention that both of my posts assume that the original memory is allocated & managed by Julia. If the data is instead allocated & managed by C, the dance is going to be a bit different, I think.

5

Yet a third variation that seems to work:

mutable struct JLExample
    data::Vector{Int64}
end

mutable struct CExample
    data::Ptr{Int64}
    CExample() = new()
end

Base.cconvert(::Type{Ref{CExample}}, jle::JLExample) = (jle, CExample())

function Base.unsafe_convert(::Type{Ref{CExample}}, ce::Tuple{JLExample, CExample})
    jle, ref = ce
    ref.data = pointer(jle.data)
    return convert(Ptr{CExample}, Base.pointer_from_objref(ref))
end

This is basically the same as the one above, except that CExample itself is mutable, instead of storing an immutable in a Ref. This way, we can take the pointer to our data in unsafe_convert, and just write that pointer into the existing struct, which is then passed to C. I think this one, as well as the previous one are equally good, but there’s still a little lingering uneasiness about the whole thing.

6

In GTPSA.jl, we solved this problem by allocating the array directly in C. E.g. for a member array coef, we allocate in Julia using:

coef = @ccall jl_malloc(sz::Csize_t)::Ptr{T}

We then attach the finalizer to it:

f(t) = @ccall jl_free(t.coef::Ptr{Cvoid})::Cvoid
finalizer(f, t)

This solution has the benefits of keeping the C-style array owned by the C, safe from Julia’s garbage collector. I then pass to C functions like:

s = TPS() # mutable type like your example
@ccall lib.someFunc(s::Ref{TPS})::Cvoid

Of course, you must make sure that flexible array members are not used in the C code for this to work.