Macro for linking Julia fcn to @ccall of C fcn, given names and arg types

I have a bunch of C functions, that I want to write Julia wrappers for. Here’s an example:

const LIBSPARSE = "/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/"*
                    "vecLib.framework/libSparse.dylib"
# ...definitions of cconvert and unsafe convert...
SparseMultiply(arg1::SparseMatrix{Cfloat},
                                arg2::StridedMatrix{Cfloat},
                                arg3::StridedMatrix{Cfloat}) = 
            @ccall LIBSPARSE._Z14SparseMultiply18SparseMatrix_Float17DenseMatrix_FloatS0_(arg1::SparseMatrix{Cfloat},
                                                                                        arg2::DenseMatrix{Cfloat},
                                                                                        arg3::DenseMatrix{Cfloat})::Cvoid

I realize this isn’t quite self contained: copy-paste the first 200ish lines from here for the definition of SparseMatrix and cconvert. Many of these functions have the same Julia name, and only differ by their arguments. The types passed to the c function are mostly the same as the Julia types, with a few exceptions: there’s a couple enums that become Cuint32s, and Strided(Vector/Matrix){Cfloat/Cdouble} become Dense(Vector/Matrix){Cfloat/Cdouble}. (See previous link for definitions of these structs: their internals aren’t important for the question.) Here’s my best attempt at writing a macro that does this for me:

macro generateDemangled(jlName, cName, retType, jlArgTypes...)
    local jlArgExprs = map(enumerate(jlArgTypes)) do (i, T)
        Expr(:(::), esc(Symbol("arg$i")), esc(T))
    end
    local jlCall = Expr(:(::), Expr(:call, esc(jlName), jlArgExprs...), esc(retType))
    local cArgTypesList = Vector{Type}()
    local ALL_ENUMS = [:(SparseFactorization_t), :(SparseOrder_t), :(SparseScaling_t),
                                                    :(SparseStatus_t), :(SparseControl_t)]
    for jlType in jlArgTypes
        if jlType in ALL_ENUMS
            push!(cArgTypesList, Cuint)
        elseif jlType == :(StridedVector{Cfloat})
            push!(cArgTypesList, DenseVector{Cfloat})
        elseif jlType == :(StridedVector{Cdouble})
            push!(cArgTypesList, DenseVector{Cdouble})
        elseif jlType == :(StridedMatrix{Cfloat})
            push!(cArgTypesList, DenseMatrix{Cfloat})
        elseif jlType == :(StridedMatrix{Cdouble})
            push!(cArgTypesList, DenseMatrix{Cdouble})
        else
            push!(cArgTypesList, eval(jlType)) # Issue #1
        end
    end
    local cArgTypesExpr = Expr(:tuple, cArgTypesList...)
    local functionAndLibrary = Expr(:tuple, esc(cName), LIBSPARSE)
    local autoNamedArgs = [esc(Symbol("arg$i")) for i in 1:length(jlArgTypes)]
    # Issue #2
    local ccallFunction = Expr(:call, :ccall, functionAndLibrary, esc(retType), cArgTypesExpr, autoNamedArgs...)
    return Expr(Symbol("="), jlCall, ccallFunction)
end

Running

@macroexpand @generateDemangled(SparseMultiply, 
    :_Z14SparseMultiply18SparseMatrix_Float17DenseMatrix_FloatS0_,
    Cvoid,
    SparseMatrix{Cfloat}, StridedMatrix{Cfloat}, StridedMatrix{Cfloat})

in the REPL generates something that looks pretty close to what I want. But there’s still 2 issues with this:

  1. I’m calling eval inside of a macro. That’s bad form: macros ought to manipulate expressions at parse time.
  2. When I tried to use @ccall via Expr(:macrocall, Symbol("@ccall"), ...) it’d give me errors for not passing a LineNumberNode as the 2nd argument. I reverted to ccall, but now this means I ought to explicitly invoke GC.@preserve, cconvert, and unsafe_convert. (These structs have pointers as fields.) Ick.

Is there a way to resolve these 2 issues? Or is there a better approach that avoids macros entirely? Mostly I’m just trying to cut down on code repetition. I guess I could provide the C types as well, which would eliminate issue #1…but then that brings back a good portion of the code repetition.

I do care about performance, though: are there benefits that come with using a macro, versus other ways of generating code? I’m currently using @eval to generate multiple functions, by looping over 2 possible type parameters, to cut the amount of code duplication in half. Code snippet here. But I don’t expect that to impact performance: both types are hard-coded.

I think you don’t need to eval(jlType) and instead push!(cArgTypesList(esc(jlType))). The eval just gets you an early resolution of the type name to the actual type, so you then proceed putting that type into the AST directly which is not necessary.

For 2, macro usage always generates these LineNumberNodes:

julia> Meta.@dump @macro arg
Expr
  head: Symbol macrocall
  args: Array{Any}((3,))
    1: Symbol @macro
    2: LineNumberNode
      line: Int64 1
      file: Symbol REPL[15]
    3: Symbol arg

So if you want to construct the macrocall manually, you can make a LineNumberNode yourself:

macro showmacro(x)
    Expr(:macrocall, Symbol("@show"), :(LineNumberNode(@__LINE__, @__FILE__)), x)
end

@showmacro sin(3)
# sin(3) = 0.1411200080598672

You can look at what we do with ccalls in Arblib.jl, e.g. here:

### Memory management
arbcall"void acb_init(acb_t x)"
arbcall"void acb_clear(acb_t x)"
#mo arbcall"acb_ptr _acb_vec_init(slong n)" # clashes with similar method for arb
arbcall"void _acb_vec_clear(acb_ptr v, slong n)"
arbcall"slong acb_allocated_bytes(const acb_t x)"
arbcall"slong _acb_vec_allocated_bytes(acb_srcptr vec, slong len)"
#mo arbcall"double _acb_vec_estimate_allocated_bytes(slong len, slong prec)" # clashes with similar method for arb

### Basic manipulation
arbcall"void acb_zero(acb_t z)"
arbcall"void acb_one(acb_t z)"
arbcall"void acb_onei(acb_t z)"
arbcall"void acb_set(acb_t z, const acb_t x)"
arbcall"void acb_set_ui(acb_t z, ulong x)"
arbcall"void acb_set_si(acb_t z, slong x)"
arbcall"void acb_set_d(acb_t z, double x)"
[...]

all of these are string macros that generate the corresponding julia functions. For example a C-signature

int acb_mat_lu_classical(slong * perm, acb_mat_t LU, const acb_mat_t A, slong prec)

is translated as follows:

julia> sgn = "int acb_mat_lu_classical(slong * perm, acb_mat_t LU, const acb_mat_t A, slong prec)";
julia> fn = Arblib.ArbCall.ArbFunction(sgn)
Arblib.ArbCall.ArbFunction{Int32}("acb_mat_lu_classical", Arblib.ArbCall.Carg[Arblib.ArbCall.Carg{Vector{Int64}}(:perm, false), Arblib.ArbCall.Carg{AcbMatrix}(:LU, false), Arblib.ArbCall.Carg{AcbMatrix}(:A, true), Arblib.ArbCall.Carg{Int64}(:prec, false)])

julia> Arblib.ArbCall.jlargs(fn)
(Expr[:(perm::Vector{<:Integer}), :(LU::AcbMatrixLike), :(A::AcbMatrixLike)], Expr[:(prec::Integer)])

julia> Arblib.ArbCall.jlcode(fn)
quote
    #= /home/kalmar/.julia/dev/Arblib/src/ArbCall/ArbFunction.jl:175 =#
    function lu_classical!(perm::Vector{<:Integer}, LU::AcbMatrixLike, A::AcbMatrixLike, prec::Integer)
        #= /home/kalmar/.julia/dev/Arblib/src/ArbCall/ArbFunction.jl:152 =#
        #= /home/kalmar/.julia/dev/Arblib/src/ArbCall/ArbFunction.jl:153 =#
        __ret = ccall(#= /home/kalmar/.julia/dev/Arblib/src/ArbCall/ArbFunction.jl:154 =# Arblib.@libflint("acb_mat_lu_classical"), Int32, (Ref{Int64}, Ref{Arblib.acb_mat_struct}, Ref{Arblib.acb_mat_struct}, Int64), perm, LU, A, prec)
        #= /home/kalmar/.julia/dev/Arblib/src/ArbCall/ArbFunction.jl:159 =#
        __ret
    end
    #= /home/kalmar/.julia/dev/Arblib/src/ArbCall/ArbFunction.jl:176 =#
    lu_classical!(perm::Vector{<:Integer}, LU::AcbMatrixLike, A::AcbMatrixLike; prec::Integer) = begin
            #= /home/kalmar/.julia/dev/Arblib/src/ArbCall/ArbFunction.jl:176 =#
            lu_classical!(perm, LU, A, prec)
        end
end

this code is generated at precompile time and never written to file;
As you can see we spent some time structuring the translation but it definitely pays off in the long run. Thanks to this we wrap effortlessly virtually the whole Arb (which to authors credit is very well structured itself) and adding new types/functions requires little to no human intervention (except parsing the latest documentation).

I think you don’t need to eval(jlType) and instead push!(cArgTypesList(esc(jlType))) .

Aha! That’s nice and simple. I thought I had tried that, but apparently not.

So if you want to construct the macrocall manually, you can make a LineNumberNode yourself

So that’s how I’d construct a suitable line node. Thanks! That answers both my questions. Here’s the final macro:

macro generateDemangled(jlName, cName, retType, jlArgTypes...)
    local jlArgExprs = map(enumerate(jlArgTypes)) do (i, T)
        Expr(:(::), esc(Symbol("arg$i")), esc(T))
    end
    local jlCall = Expr(:(::), Expr(:call, esc(jlName), jlArgExprs...), esc(retType))
    local ALL_ENUMS = [:(SparseFactorization_t), :(SparseOrder_t), :(SparseScaling_t),
                                                    :(SparseStatus_t), :(SparseControl_t)]
    local cArgTypes = Vector{Any}()
    # convert the julia argument types to their C equivalents. 
    for sym in jlArgTypes
        if sym in ALL_ENUMS
            push!(cArgTypes, Cuint)
        elseif sym == :(StridedVector{Cfloat})
            push!(cArgTypes, DenseVector{Cfloat})
        elseif sym == :(StridedVector{Cdouble})
            push!(cArgTypes, DenseVector{Cdouble})
        elseif sym == :(StridedMatrix{Cfloat})
            push!(cArgTypes, DenseMatrix{Cfloat})
        elseif sym == :(StridedMatrix{Cdouble})
            push!(cArgTypes, DenseMatrix{Cdouble})
        else
            push!(cArgTypes, esc(sym))
        end
    end
    local cArgExprs = [Expr(:(::), esc(Symbol("arg$i")), T) for (i,T) in enumerate(cArgTypes)]
    local LIBSPARSE = "/System/Library/Frameworks/Accelerate.framework/Versions"*
                    "/A/Frameworks/vecLib.framework/libSparse.dylib"
    local funcAndLibrary  = Expr(:(.), LIBSPARSE, esc(cName))
    # due to my use case, I don't need to worry about return type translation.
    local cCall = Expr(:(::), Expr(:call, funcAndLibrary, cArgExprs...), esc(retType))
    local ccallMarco = Expr(:macrocall, Symbol("@ccall"), :(LineNumberNode(@__LINE__, @__FILE__)), cCall)
    return Expr(Symbol("="), jlCall, ccallMarco)
end

# example use
@generateDemangled(SparseMultiply, # Julia name
    :_Z14SparseMultiply18SparseMatrix_Float17DenseMatrix_FloatS0_, # C name
    Cvoid, # return type
    SparseMatrix{Cfloat}, StridedMatrix{Cfloat}, StridedMatrix{Cfloat}) # Julia argument types

That Arblib.jl stuff looks real fancy. I may go that route if I need to do this for 100+ functions, but right now I’m only at about 30 or so. Also, part of me just wanted to know where I was going wrong: this is my first time trying to write a marco.

1 Like