Follow up question from Ccall with a C struct containing a pointer: how do I make this cconvert/unsafe_convert dance work when the struct is immutable? Re-assigning dm.data to pointer(m) inside unsafe_convert is then out of the question. If I was writing C/C++, I’d allocate space for the buffer by calling new in cconvert, and then write its contents by calling memcpy in unsafe_convert. How do I do this in Julia, such that my new buffer doesn’t get garbage-collected?
Create a Ref in cconvert, wrapping your (unfinished) immutable (or uninitialized but with the correct type, like Ref{MyType}(). In unsafe_convert, replace the content of the Ref with a newly created immutable containing the correct pointer.
This should work, because immutables are (by design) defined by their bitequality instead of their address in memory. By themselves, they don’t have a stable memory address. The Ref provides that stable address for the ccall (assuming you need to pass a pointer to your immutable into the C function), and the C side can write to that pointer as much as it wants. To retrieve any potentially changed data, simply index the Ref like r[].
assuming you need to pass a pointer to your immutable into the C function
Unfortunately, I want to pass the C function a value, not a pointer. (These discussions have made me realize that this would be a lot easier if my C library wasn’t this way.)
An elaboration to my original post: allocating a large enough buffer inside cconvert then writing its contents inside unsafe_convert is a work-around. Ideally I’d rather not memcpy at all, and just make the pointer point to the same contents. However, I don’t see how to do that for an immutable struct: the docs seem to say that I should allocate all objects inside cconvert, and only convert to pointers inside unsafe_convert. But it’s immutable, so I can’t allocate-then-set-field.
If you want to pass the object by value, you should be able to just create the immutable with the correct pointer value in unsafe_convert, though I admittedly haven’t tried that. That’s the one variation that didn’t come up in your previous thread 
I’m a bit confused by this - you shouldn’t need to memcpy anything around here, if your immutable struct already contains the Ptr object to your GC-tracked object. You only need to make sure the GC tracked object the Ptr refers to stays alive, which you can do either through GC.@preserve <myobj> around the entire ccall, or returning the object in question from cconvert (e.g. through a tuple, or directly or w\e). There should be no issue with creating a C-compatible immutable object containing the pointer in unsafe_convert, which is then passed to C by value.
Or am I misunderstanding what you’re referring to when you say “an immutable containing a pointer”?
I think I understand. The cconvert/unsafe_convert call pattern is just a convenience thing: Julia will automatically GC.@preserve whatever is returned from cconvert, then pass it on to unsafe_convert. I’m free to work with pointers elsewhere: I just need to manually GC.@preserve the pointed-to object. The following should work:
struct DenseMatrix # immutable!
rowCount::Cint
columnCount::Cint
data::Ptr{Cdouble}
end
function Base.cconvert(::Type{DenseMatrix}, m::Matrix{Float64})
# the GC.@preserve around the @ccall makes this safe.
return DenseMatrix(size(m)[1], size(m)[2], pointer(m))
end
function CWrapper(arg1::Matrix{Float64})
GC.@preserve arg1 @ccall lib.fname(arg1::DenseMatrix)
end
Yes, except I think for your example the GC.@preserve is unnecessary - you should be able to just do
struct DenseMatrix # immutable!
rowCount::Cint
columnCount::Cint
data::Ptr{Cdouble}
end
Base.cconvert(::Type{DenseMatrix}, m::Matrix{Float64}) = m
function Base.unsafe_convert(::Type{DenseMatrix}, m::Matrix{Float64})
return DenseMatrix(size(m, 1), size(m, 2), pointer(m))
end
function CWrapper(arg1::Matrix{Float64})
@ccall lib.fname(arg1::DenseMatrix)::<rettype>
end
since the DenseMatrix will be passed by-value (i.e. as a copy) anyway (it’s isbits after all) and itself doesn’t need to be GC preserved, as I understand it. Just returning m from cconvert already @preserves the matrix itself:
julia> isbitstype(DenseMatrix)
true
shell> cat mwe.c
#include <math.h>
typedef struct {
int rowCount;
int columnCount;
double* data;
} DenseMatrix;
double foo(DenseMatrix foo) {
if (foo.columnCount <= 0)
return NAN;
if (foo.rowCount <= 0)
return INFINITY;
int idxa = foo.columnCount - 1;
int idxb = foo.rowCount - 1;
return foo.data[idxb*(foo.columnCount) + idxa];
}
julia> data = rand(Float64, 15, 37)
15×37 Matrix{Float64}:
0.599364 0.22981 0.96945 … 0.344901 0.416426 0.192333
0.0278234 0.0835467 0.460888 0.665467 0.825422 0.430733
0.365078 0.155395 0.0188155 0.968414 0.797662 0.628757
0.457193 0.256214 0.374096 0.0370763 0.261907 0.141956
0.181229 0.0202176 0.0417381 0.595971 0.425398 0.873739
0.166689 0.799661 0.30146 … 0.379971 0.0946295 0.709757
0.602595 0.0282483 0.327141 0.211264 0.93809 0.991113
0.446278 0.25726 0.617319 0.422344 0.473222 0.343918
0.331866 0.346359 0.464125 0.649524 0.568934 0.910309
0.00314365 0.0951955 0.523954 0.366409 0.239934 0.273698
0.625282 0.484357 0.646132 … 0.0714788 0.949366 0.181078
0.446451 0.203708 0.637352 0.57657 0.238058 0.800698
0.528139 0.915547 0.15079 0.561895 0.812107 0.940406
0.398491 0.664749 0.708104 0.979059 0.0340735 0.347522
0.965296 0.236917 0.781048 0.507677 0.866314 0.377035
julia> CWrapper(data)
0.3770354806819668
julia> CWrapper(data) === last(data)
true
julia> function foo(s1, s2)
data = rand(Float64, s1, s2)
CWrapper(data) == last(data)
end
foo (generic function with 1 method)
julia> @allocated foo(10, 15)
1296
julia> 10*15*8 # size of the matrix allocated in foo, plus some overhead for the `Vector` struct etc
1200
One thing you will have to be careful about is row- vs column-majorness, since Julia is column major and C-libraries usually do row-major.