Interpolation using $ with @cfunction in a macro reported as a closure, thus won't work on Apple Silicon

General Usage
#1

I’m still learning about macros and only recently learnt about closures. I am trying to use the ArchGDAL package on my Apple Silicon Macbook, but it reported an error during precompilation: ERROR: LoadError: cfunction: closures are not supported on this platform.

I have managed to isolate a MWE:

macro cplprogress(progressfunc)
    @cfunction($(esc(progressfunc)), Cint, (Cdouble, Cstring, Ptr{Cvoid}))
end

progressfunc = x -> x

@cplprogress(progressfunc)

This results in the following stacktrace:

ERROR: LoadError: cfunction: closures are not supported on this platform
Stacktrace:
 [1] var"@cplprogress"(__source__::LineNumberNode, __module__::Module, progressfunc::Any)
   @ Main ~/mwe.jl:2
in expression starting at /mwe.jl:7
in expression starting at /mwe.jl:7

From what I understand, the $ in the macro is not in fact indicating a closure, but is actually just interpolation for the macro?
Either way I would really appreciate some guidance to fixing it so the closure can be removed, or otherwise fixed!

Thanks in advance!

Julia: 1.7.1
OS: MacOS Monterey 12.1
CPU: M1 Pro
Arch: darwin-aarch64

1 Like
#2

The ArchGDAL.jl package is apparently using this for the GDALProgressFunc callback API.

It seems like the solution is to change the Julia package to use the void* argument (pProgressArg) to pass the closure argument, rather than forcing Julia to compile a new callback for every progressfunc argument it wants to handle (which apparently isn’t supported on M1). That is, they should define something like:

function progressfunc_wrapper(dfComplete::Cdouble, pszMessage::Ptr{UInt8}, pProgressArg::Any)::Cint
    pszMessage == C_NULL && return pProgressArg(dfComplete)
    return pProgressArg(dfComplete, unsafe_string(pszMessage))
end
const progressfunc_wrapper_c = @cfunction(progressfunc_wrapper, Cint, (Cdouble, Cstring, Any))

and then, any place you actually want to pass a progressfunc, you instead pass progressfunc_wrapper_c as the callback function and progressfunc as the callback-data argument.

In APIs that support a callback-data argument, like this one, the above is the approach I would recommend. It puts a lot fewer demands on the compiler.

(Also, then their progresscallback functions are simpler and more Julian. They are passed a Julia String as their optional “message” argument rather than a pointer, and can return true or false instead of a Cint, since progressfunc_wrapper handles the conversions.)

3 Likes
#3

Thanks @stevengj . A wrapper function sounds like a good idea, I’ll see if I can get something like that working then hopefully submit a PR.

Do you know if the $ in the original implementation is actually creating a closure though? Or is that just part of the macro syntax.

Cheers

#4

Yes, because it’s splicing in a runtime closure from the call site.

Looking at how it’s used in ArchGDAL.jl, to be a drop-in replacement you might want a “dumber” progressfunc_wrapper than I wrote above, which just calls:

return pProgressArg(dfComplete, pszMessage)

to keep their existing progressfunc API. Then, when passing the callback to a GDAL function like is done here, you would pass progressfunc_wrapper_c instead of @cplprogress(progressfunc), and pass (complete, msg) -> progressfunc(complete, msg, progressdata) instead of progressdata. (The @cplprogress macro can be entirely deleted.)

In the longer run, I would re-think the progressfunc Julia API to make it more Julian (no raw-pointer or data arguments), but in the short run this will let you make a PR that keeps everything in the API the same and only improves the implementation.

1 Like
#5

Your MWE is actually a fairly weird looking macro because it’s invoking @cfunction as part of the macro expansion. So the esc here doesn’t really make sense.

Was this unintentional, and did you mean to quote the @cfunction syntax?

macro cplprogress(progressfunc)
    quote
        @cfunction($(esc(progressfunc)), Cint, (Cdouble, Cstring, Ptr{Cvoid}))
    end
end
#6

This example was just taken verbatim from the ArchGDAL package. I tried it with the quote structure but got a invalid syntax error.

#7

This seems like a bug in Base, in particular in the internals of macroexpand which doesn’t understand Expr(:cfunction). The following should also work, but doesn’t.

macro cplprogress(progressfunc)
    quote
        f = $(esc(progressfunc))
        @cfunction(f, Cint, (Cdouble, Cstring, Ptr{Cvoid}))
    end
end
#8

Actually, no, I take that back. The problem here is that writing a macro to wrap a macro which recognizes $ syntax is quite subtle. This is because normal syntax interpolation will see and remove the $ before the inner quoted macro (the @cfunction in this case) gets to do so. I think the correct macro would be something like:

macro cplprogress(progressfunc)
    quote
        @cfunction($(Expr(:$, esc(progressfunc))), Cint, (Cdouble, Cstring, Ptr{Cvoid}))
    end
end

Then we’d have:

julia> @macroexpand @cplprogress x->x
quote
    #= REPL[41]:3 =#
    $(Expr(:cfunction, Base.CFunction, :(x->begin
          #= REPL[43]:1 =#
          x
      end), :(Main.Cint), :(Core.svec(Main.Cdouble, Main.Cstring, Main.Ptr{Main.Cvoid})), :(:ccall)))
end

In any case, yes, the original macro is using @cfunction outside a quote block, but with a $. So this is using the closure syntax.

#9

Sorry for bumping, just wanting to point out that we simply don’t support a custom callback function on Apple Silicon right now in ArchGDAL, which makes it work again. This fix is in v0.8.5.