# Generate a function via macro and pass to @cfunction. Possible?
**URL:** https://discourse.julialang.org/t/generate-a-function-via-macro-and-pass-to-cfunction-possible/123988
**Category:** General Usage
**Tags:** question, macros, c
**Created:** [December 19, 2024, 11:38am UTC](https://discourse.julialang.org/t/generate-a-function-via-macro-and-pass-to-cfunction-possible/123988 "2024-12-19T11:38:06Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@tqml](https://discourse.julialang.org/u/tqml)
#### Post date: [December 19, 2024, 11:38am UTC](https://discourse.julialang.org/t/generate-a-function-via-macro-and-pass-to-cfunction-possible/123988/1 "2024-12-19T11:38:06Z")
I’m trying to write a macro that takes a function, creates a wrapper for it, passes the wrapper to `@cfunction` and finally passes the function pointer to an external C library.
To further clarify the use-case here: I want to create a type-stable wrapper of the function and pass this to the C-Library. The wrapper should extract the arguments from a C pointer and then call the function, all without runtime dispatch.
I tried several things, that dont work, see below.
My question:
How can this could be done? Is this even possible?
### Approach 1: Defining wrapper and passing to `@cfunction`.
```julia
macro _test_macro_func_wrapped_1(func)
func_esc = esc(func) # must be escaped, because macro lives in package
quote
wrapped = function ()
println("Execute Wrapper")
r = $func_esc()
return println("Execute Wrapper result: ", r)
end
wrapped() # works
# ERROR HERE - "wrapped" not defined
ptr = @cfunction(wrapped, Cvoid, ())
@info "Function ptr:" ptr
ptr
end
end
```
### Approach 2:
Try to avoid macro hygiene (but not working)
```julia
macro _test_macro_func_wrapped_2(func)
func_esc = esc(func) # escape the function
wrapper_name = :wrapper
println("Wrapper name: ", wrapper_name)
quote
# Try to avoid auto generated name, but somehow its still generated
function ($wrapper_name)()
println("Execute Wrapper")
r = $func_esc(10)
return println("Execute Wrapper result: ", r)
end
($wrapper_name)() # works
$wrapper_name # works
@info "Method:" $wrapper_name # Works
# ERROR HERE: "wrapper" not defined in Main (but also not in the package)
ptr = @cfunction($wrapper_name, Cvoid, ())
@info "Function ptr:" ptr
ptr
end
end
```
### Approach 3:
Avoid macro hygiene and force definition of the wrapper via `eval`. But fails, because of an “syntax error” if `esc()` is used, or if `esc()` is not used, then the outer function cannot be referenced.
```julia
macro _test_macro_func_wrapped_3(func)
func_esc = esc(func) # escape the function
wrapper_name = :wrapper
package_name = @ __MODULE__
package_wrapper_expr = Expr(:., package_name, :(:wrapper))
println("Wrapper name: ", wrapper_name)
println("Package wrapper expr: ", package_wrapper_expr)
wrapper_def = quote
function ($wrapper_name)()
println("Execute Wrapper")
r = $(func)(10)
return println("Execute Wrapper result: ", r)
end
end
# ERROR here: LoadError: syntax: invalid syntax (escape (outerref my_scalar_function))
# or if esc() is ommited: UndefVarError: `my_scalar_function` not defined in `package`
eval(wrapper_def)
quote
($package_wrapper_expr)()
$package_wrapper_expr
@info "Method:" $package_wrapper_expr
ptr = @cfunction($package_wrapper_expr, Cvoid, ())
@info "Function ptr:" ptr
ptr
end
end
```
---
### Author:  [@Benny](https://discourse.julialang.org/u/Benny)
#### Post date: [December 19, 2024, 12:48pm UTC](https://discourse.julialang.org/t/generate-a-function-via-macro-and-pass-to-cfunction-possible/123988/2 "2024-12-19T12:48:48Z")
You’re hoping to create an expression that `eval`uates to 1) define a wrapped function, then 2) make a function pointer from it. That would work if step 2 was a normal function call; sequential execution is what they do. But `@cfunction` isn’t a function call, it’s a macro call, and those are expanded in the expression before `eval`uation even happens.
Many macro calls are designed to expand to function calls that do the real work, so the sequential behavior still happens. Let’s see what `@cfunction` does:
```julia
julia> @macroexpand begin
function foo(x::Int, y::Int) # docstring example
return x + y
end
@cfunction(foo, Int, (Int, Int))
end
quote
#= REPL[25]:2 =#
function foo(x::Int, y::Int)
#= REPL[25]:2 =#
#= REPL[25]:3 =#
return x + y
end
#= REPL[25]:5 =#
$(Expr(:cfunction, Ptr{Nothing}, :(:foo), :Int, :(Core.svec(Int, Int)), :(:ccall)))
end
```
`@cfunction` expands to a dedicated `Expr`, which is definitely not a function call (`:call`). `@cfunction`’s docstring mentions
> … these arguments will be evaluated in global scope during compile-time
> (not deferred until runtime)…
so the real work happens during evaluation but before the sequential execution (like how `local`/`global` statements still apply to a whole scope despite being written at the tail end of the expression). You’ll need to evaluate that wrapped function into the global scope, then try the `@cfunction` call in a separate top-level expression.
```julia
julia> begin
function foo(x::Int, y::Int)
return x + y
end
@cfunction(foo, Int, (Int, Int)) # macro call errors at compile-time
end
ERROR: UndefVarError: `foo` not defined in `Main`
Suggestion: check for spelling errors or missing imports.
julia> foo # "preceding" definition was not executed
ERROR: UndefVarError: `foo` not defined in `Main`
Suggestion: check for spelling errors or missing imports.
julia> function foo(x::Int, y::Int)
return x + y
end
foo (generic function with 1 method)
julia> @cfunction(foo, Int, (Int, Int))
Ptr{Nothing} @0x000001ddee99b6f0
```
---
### Author:  [@tqml](https://discourse.julialang.org/u/tqml)
#### Post date: [December 19, 2024, 1:24pm UTC](https://discourse.julialang.org/t/generate-a-function-via-macro-and-pass-to-cfunction-possible/123988/3 "2024-12-19T13:24:14Z")
Thanks for the reply and the explanation.
Is there any way to automatically define the wrapper function _before_ `@cfunction` is evaluated?
I want to have the implementation as user-friendly as possible and avoid that the user needs to first define a wrapper in global scope. It would be convenient if all of this could happen within one macro or function call.
---
### Author:  [@Benny](https://discourse.julialang.org/u/Benny)
#### Post date: [December 19, 2024, 11:54pm UTC](https://discourse.julialang.org/t/generate-a-function-via-macro-and-pass-to-cfunction-possible/123988/4 "2024-12-19T23:54:39Z")
I pretty much never interop with C in practice so a lot of it is lost on me. I myself would like to understand why `@cfunction` and `ccall` are like that instead of more straightforward function calls. I’d try my luck with `$`:
> …compile-time (not deferred until runtime). Adding a ‘$’ in front of the function argument changes this to instead create a runtime closure over the local variable `callable` (this is not supported on all architectures).
which is harder to preserve for the macro call when the surrounding quoted expression attempts `$`-interpolation, but that can be pulled off by `$`-interpolating a `Expr(:$, ...)` in other cases.
---
### Author:  [@tqml](https://discourse.julialang.org/u/tqml)
#### Post date: [December 20, 2024, 2:09pm UTC](https://discourse.julialang.org/t/generate-a-function-via-macro-and-pass-to-cfunction-possible/123988/5 "2024-12-20T14:09:16Z")
Good point, that might be the BEST way to achieve this, but its not my preferred way, because its not supported everywhere (e.g. ARM ) 😕
However, I had a look at the implementation of `@cfunction` (which is surprisingly simple!) and found a way to create a dynamic cfunction.
Idea is:
- Create a global dictionary and store the function there
- do not use `@cfunction`, instead build the expression and evaluate it at runtime.
I’m just wondering: How dangerous / future-proof is this?
```julia
REF = Dict{Int, Function}()
function inner_ccall_2()
choice = parse(Int, ARGS[1])
R = rand(1:10)
dynamic_func_inner = dynamic(R)
REF[1] = dynamic_func_inner # store the function in a global variable
#fptr = QuoteNode(:dynamic_func_inner)
# DOESNT WORK (fails with "KeyError")
# ptr = @cfunction(REF[1], Int, (Int,))
# res = ccall(ptr, Int, (Int,), R)
# @info "Result 3" R res
# WORKS (delayed evaluation)
# manually construct cfunction, what would normally do the macro
fptr = QuoteNode(:(REF[1]))
rt = :Int
at = :(Int,)
attr_svec = Expr(:call, GlobalRef(Core, :svec), at.args...)
cfun = Expr(:cfunction, cfunction_type, fptr, rt, attr_svec, QuoteNode(:ccall))
ptr = eval(cfun)
res = ccall(ptr, Int, (Int,), R)
@info "Result 3" R res
# SHOULD ALSO WORK
#cfun_expr_2 = Expr(:macrocall, Symbol("@cfunction"), LineNumberNode(@ __LINE__ , @ __MODULE__ ), :(REF[1]), :Int, :(Int,))
#@show cfun_expr_2
#ptr2 = eval(cfun_expr_2)
#res2 = ccall(ptr2, Int, (Int,), R)
#@info "Result 4" R res2
end
```
---
### Author:  [@mikmoore](https://discourse.julialang.org/u/mikmoore)
#### Post date: [December 20, 2024, 3:32pm UTC](https://discourse.julialang.org/t/generate-a-function-via-macro-and-pass-to-cfunction-possible/123988/6 "2024-12-20T15:32:20Z")
Might FunctionWrappers.jl be relevant here? It at least does the job of getting a type-stable function pointer from an arbitrary function and a call signature. But I don’t know if/how you can pass that to a foreign function. It might just work, or maybe after calling something like `pointer(wrappedfunction)`?
---
### Author:  [@Benny](https://discourse.julialang.org/u/Benny)
#### Post date: [December 20, 2024, 11:07pm UTC](https://discourse.julialang.org/t/generate-a-function-via-macro-and-pass-to-cfunction-possible/123988/7 "2024-12-20T23:07:16Z")
> [@tqml](#):
>
> `cfun = Expr(:cfunction, cfunction_type, fptr, rt, attr_svec, QuoteNode(:ccall))`
What’s `cfunction_type` and how does the overall expression work? It doesn’t seem to match the expressions made by the documented `@cfunction(callable, ReturnType, (ArgumentTypes...,)) -> Ptr{Cvoid}`.
> [@mikmoore](#):
>
> Might [FunctionWrappers.jl](https://juliahub.com/ui/Packages/General/FunctionWrappers) be relevant here?
It has problems keeping up with world age now ([Issue #52635](https://github.com/JuliaLang/julia/issues/52635)). `FunctionWrapper` instantiation appears to get stuck at the first world age where a call signature was compiled, and the internal `reinit_wrapper` only gets you to the next one, not the current one. I don’t know if previous world ages’ code are never collected or if `FunctionWrappers` is keeping that code alive, no idea how the internals work there. But I’d look elsewhere for a runtime generated function pointer.
> **Minimal example so you know for sure what I'm talking about.**
>
> ```julia
> julia> using FunctionWrappers: FunctionWrappers as FWs, FunctionWrapper as FW
>
> julia> foo()=0;
>
> julia> foo()=1;
>
> julia> FW{Int, Tuple{}}(foo)()
> 1
>
> julia> foo()=2;
>
> julia> FW{Int, Tuple{}}(foo)()
> 1
>
> julia> let x = FW{Int, Tuple{}}(foo)
> FWs.reinit_wrapper(x) # internal
> x()
> end
> 2
>
> julia> foo()=3;
>
> julia> foo()
> 3
>
> julia> FW{Int, Tuple{}}(foo)()
> 1
>
> julia> let x = FW{Int, Tuple{}}(foo)
> FWs.reinit_wrapper(x) # internal
> x()
> end
> 2
>
> ```
---
### Author:  [@tqml](https://discourse.julialang.org/u/tqml)
#### Post date: [December 20, 2024, 11:59pm UTC](https://discourse.julialang.org/t/generate-a-function-via-macro-and-pass-to-cfunction-possible/123988/8 "2024-12-20T23:59:09Z")
> [@Benny](#):
>
> What’s `cfunction_type` and how does the overall expression work? It doesn’t seem to match the expressions made by the documented `@cfunction(callable, ReturnType, (ArgumentTypes...,)) -> Ptr{Cvoid}`.
I actually copied this from the Julia source code (its how `@cfunction` is implemented). Internally, the `@cfunction` macro creates a `Expr(:cfunction, ...)` with some argument checks and conversations (tuple becomes a Core.svec). `cfunction_type` is a `Ptr{Cvoid}` (forgot to copy that). In case of a closure it would be a `Expr(:$,...)`, but thats not supported on my machine.
I found out that you can put other things inside `fptr = QuoteNode(...)` and it will still work, as long as the symbol is globally accessible. So its a nice way to delay the macrocall until the function is defined.
Thats the entire definition of `@cfunction`:
```julia
macro cfunction(f, rt, at)
if !(isa(at, Expr) && at.head === :tuple)
throw(ArgumentError("@cfunction argument types must be a literal tuple"))
end
at.head = :call
pushfirst!(at.args, GlobalRef(Core, :svec))
if isa(f, Expr) && f.head === :$
fptr = f.args[1]
typ = CFunction
else
fptr = QuoteNode(f)
typ = Ptr{Cvoid}
end
cfun = Expr(:cfunction, typ, fptr, rt, at, QuoteNode(:ccall))
return esc(cfun)
end
```