Question on ccall for complex types

General Usage
1

We are getting deprecation warnings for uses of & in ccall’s in Nemo.jl. It’s time we replaced them, but I don’t currently understand what is required.

To make the question concrete, I give an example. On the C side we basically have a struct, called fmpz_poly, say. Various functions take pointers to such structs as both inputs and as outputs, e.g. we have functions which effectively look like this:

/* set r = a + b */
void fmpz_poly_add(fmpz_poly * r, const fmpz_poly * a, const fmpz_poly * c)
{
/* ... */
}

On the Julia side, we define:

type fmpz_poly
   # some fields
end

Now we wish to pass fmpz_poly’s created on the Julia side, to the C function fmpz_poly_add say, safely and efficiently.

At present we do the following:

function +(a::fmpz_poly, b::fmpz_poly)
   r = fmpz_poly()
   ccall((:fmpz_poly_add, :libflint), Void, (Ptr{fmpz_poly}, Ptr{fmpz_poly}, Ptr{fmpz_poly}), &r, &a, &b)
   return r
end

The recommendation seems to be to use Ref{fmpz_poly} instead of & and Ptr{fmpz_poly}. But I want to understand what this is doing.

The documentation says that Ref{fmpz_poly} prevents Julia gc from cleaning up the reference until there is no longer any reference to the Ref itself. However, the ccall interface also says that it maintains a reference to its arguments to prevent gc whilst the ccall is being executed. And it says that the Ref argument is converted to a native pointer before passing to the C side (by calling some function for this purpose).

This confuses me, since I don’t see the purpose of using a Ref, since the arguments to ccall are automatically protected from gc.

Basically, I can’t tell from the documentation which of the following is expected, and what the performance ramifications of each solution is (or whether they are valid Julia):

function +(a::fmpz_poly, b::fmpz_poly)
   r = fmpz_poly()
   reff = Ref{fmpz_poly}(r)
   refa = Ref{fmpz_poly}(a)
   refb = Ref{fmpz_poly}(b)
   ccall((:fmpz_poly_add, :libflint), Void, (Ref{fmpz_poly}, Ref{fmpz_poly}, Ref{fmpz_poly}), refr, refa, refb)
   return r
end

function +(a::fmpz_poly, b::fmpz_poly)
   r = fmpz_poly()
   ccall((:fmpz_poly_add, :libflint), Void, (Ref{fmpz_poly}, Ref{fmpz_poly}, Ref{fmpz_poly}), 
          Ref{fmpz_poly}(r), Ref{fmpz_poly}(a), Ref{fmpz_poly}(b))
   return r
end

function +(a::fmpz_poly, b::fmpz_poly)
   r = fmpz_poly()
   ccall((:fmpz_poly_add, :libflint), Void, (Ref{fmpz_poly}, Ref{fmpz_poly}, Ref{fmpz_poly}), r, a, b)
   return r
end

Clearly, we do need Julia to not gc cleanup the objects a, b and r whilst the ccall is in effect. But our entire library is full of ccalls and we obviously want to do things as efficiently as possible. So which of the above is the best strategy?

2

Ref is not doing that. That part of the doc seems to be more confusing than helping.

This is not true. Properly defined cconvert and unsafe_convert prevents that.

All of them are equivalent and should have the same performance. The first two are exactly the same. There’s no significance of assigning to a local variable.

3

To quote the (latest) documentation:

ccall automatically arranges that all of its arguments will be preserved from garbage collection until the call returns.

You mentioned cconvert and unsafe_convert. Am I supposed to define these for my types, or are there sensible Julia defaults that will do the job in most cases?

4

There’s no “sensible Julia defaults”. There are specific meanings when you use Ref as arguments type that must not be overwriten. If the convertion you want is not passing a type by reference, you may need to define your own to do that and you must not use a type combination that conflicts with the base one. (i.e. the three pieces of code should call the base definition to pass ::fmpz_poly by reference, if you want to do something else using auto conversion, you must not use Ref{fmpz_poly} as the ccall argument types).

5

Just to clarify what I think you are saying: there is no need to supply definitions of cconvert and unsafe_convert in any of the three code examples I gave, for my types, (assuming I want the values to be passed by reference).

6

I’m saying it is invalid to supply cconvert and unsafe_convert definitions for the code you have

7

Or in another word, there is a sensible definition but it is not the default, it is the only valid definition. You must not provide your own using the same types