# Is it possible to splat into ccall?

**URL:** <https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761>\
**Category:** General Usage\
**Tags:** ccall\
**Created:** [March 8, 2023, 7:49pm UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761 "2023-03-08T19:49:46Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![lmtzx9h4qqnt](https://avatars.discourse-cdn.com/v4/letter/l/74df32/32.png) [@lmtzx9h4qqnt](https://discourse.julialang.org/u/lmtzx9h4qqnt)\
**Post date:** [March 8, 2023, 7:49pm UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/1 "2023-03-08T19:49:46Z")

</div>

As the title says – is it possible? A straightforward attempt doesn’t seem to work:

```julia
julia> ccall(:pow, Cdouble, (Cdouble, Cdouble), (2, 10)...)
ERROR: syntax: more types than arguments for ccall around REPL[65]:1

```

The same question, but relating to `@ccall`, was asked here:

> [@Why is ccall syntax?](https://discourse.julialang.org/t/why-is-ccall-syntax/72427/6):
>
> Is it possible to splat into @ccall? @ccall printf("%d\n"::Cstring, $((i::Cint for c in 1:5)...))::Cint ERROR: LoadError: ArgumentError: args in @ccall need type annotations. '$((i::Cint for c = 1:5)...)' doesn't have one. Stacktrace:

But the answer is not exactly appealing:

> [@Why is ccall syntax?](https://discourse.julialang.org/t/why-is-ccall-syntax/72427/7):
>
> You can splat into the AST which @ccall is a part of, provided you do it before macro expansion. For example: @eval @ccall printf("%d %d %d %d %d\n"::Cstring, $((:($i::Cint) for i in 1:5)...))::Cint But it’s unlikely that it’s a good idea to use @eval like this! What’s your surrounding code look like for this use case?

Translating this to my example above:

```julia
julia> @eval @ccall pow($((:($x::Cdouble) for x in (2, 10))...))::Cdouble
1024.0

```

I suppose I’d even be fine with it being a little ugly, but the need for `@eval` is a showstopper.

---

<div class="post-metadata">

**Author:** ![uniment](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uniment/32/24532_2.png) [@uniment](https://discourse.julialang.org/u/uniment)\
**Post date:** [March 8, 2023, 10:05pm UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/2 "2023-03-08T22:05:00Z")

</div>

Mildly hacky, but maybe to your taste:

```julia
julia> abstract type ccaller{fun, retT, argT} end

julia> @generated ccaller{fun, retT, argT}(arg...) where {fun, retT, argT} =
           :(ccall(fun, $retT, ($((argT isa Tuple ? argT[i] : argT for i=eachindex(arg))...),), $((:(arg[$i]) for i=eachindex(arg))...)))

julia> ccaller{:pow, :Cdouble, :Cdouble}(2, 10)
1024.0

```

Splattable:

```julia
julia> ccaller{:pow, :Cdouble, ((:Cdouble, :Cdouble)...,)}((2, 10)...)
1024.0

```

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 9, 2023, 4:34am UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/3 "2023-03-09T04:34:31Z")

</div>

No, you kind of can’t do this (except for hacks!). And the presence of hacks is a sign that something “might be wrong” and you should be doing it a different way.

Can you explain your actual use case? `pow` isn’t a very realistic example, because you can just index the args instead

```julia
args = (2,10)
ccall(:pow, Cdouble, (Cdouble, Cdouble), args[1], args[2])

```

Presumably there’s some reason why you don’t want to do it this way?

---

<div class="post-metadata">

**Author:** ![lmtzx9h4qqnt](https://avatars.discourse-cdn.com/v4/letter/l/74df32/32.png) [@lmtzx9h4qqnt](https://discourse.julialang.org/u/lmtzx9h4qqnt)\
**Post date:** [March 9, 2023, 5:44pm UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/4 "2023-03-09T17:44:01Z")

</div>

Well, the canonical example would be varargs functions in C (e. g. as linked in my original post):

> [@Why is ccall syntax?](https://discourse.julialang.org/t/why-is-ccall-syntax/72427/7):
>
> You can splat into the AST which @ccall is a part of, provided you do it before macro expansion. For example: @eval @ccall printf("%d %d %d %d %d\n"::Cstring, $((:($i::Cint) for i in 1:5)...))::Cint But it’s unlikely that it’s a good idea to use @eval like this! What’s your surrounding code look like for this use case?

I used `pow` as an example because I wanted to focus on the syntax without adding the complications that come with varargs. Using @uniment’s method, the linked `printf` example becomes

```julia
julia> ccaller{:printf, :Cint, (:Cstring, ntuple(_ -> :Cint, 5)...)}("%d %d %d %d %d\n", (1:5)...);
1 2 3 4 5

```

without `@eval` – much better!

But to be honest, that isn’t even how I came across this question. I just have some C functions (in an external library) with annoyingly long argument lists (9 arguments) that I wanted to call and validate/transform the arguments for. Instead of copy-pasting the same code 9 times to handle each argument, I tried to pack them in a tuple and process them in a loop instead. So now I already have the arguments as a tuple, so the easiest (and least error-prone!) way of passing them to `ccall` would be to use splatting. Of course I _could_ also just index all 9 arguments individually, but that just seems stupid when splatting exists! We’re not writing C here after all.

So just for illustration, what I wanted to write was

```julia
function external_func_wrapper(args::NTuple{9, Integer})
	for x in args
		validate(x) || error(x)
	end
	return ccall(
		:external_func, Cdouble,
		(Cint, Cint, Cint, Cint, Cint, Cint, Cint, Cint, Cint),
		(transform(x) for x in args)...
	)
end

```

But since that’s not allowed, I’d have to change it to

```julia
function external_func_wrapper(args::NTuple{9, Integer})
	for x in args
		validate(x) || error(x)
	end
	arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9 =
		(transform(x) for x in args)
	return ccall(
		:external_func, Cdouble,
		(Cint, Cint, Cint, Cint, Cint, Cint, Cint, Cint, Cint),
		arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9
	)
end

```

which makes me list the entire indexed argument list _twice_, instead of not at all if splatting were allowed.

> [@c42f](#):
>
> And the presence of hacks is a sign that something “might be wrong” and you should be doing it a different way.

I don’t really agree with this. I’d even almost say it’s the opposite: The fact that “hacks” like @uniment’s method exist demonstrates that there isn’t any fundamental reason why this shouldn’t work. Clearly, Julia has all the information necessary to determine whether the call is valid and carry it out. It’s just that `ccall` arbitrarily rejects this specific syntax. As you say yourself in that same thread I linked above:

> [@Why is ccall syntax?](https://discourse.julialang.org/t/why-is-ccall-syntax/72427/3):
>
> #### The basic answer
> 
> It doesn’t need to be syntax. `@ccall` is a more recent and much more attractive syntax for the same thing which doesn’t need special support from the parser or lowering.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [March 9, 2023, 6:14pm UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/5 "2023-03-09T18:14:16Z")

</div>

> [@lmtzx9h4qqnt](#):
>
> I don’t really agree with this. I’d even almost say it’s the opposite: The fact that “hacks” like @uniment’s method exist demonstrates that there isn’t any fundamental reason why this shouldn’t work. Clearly, Julia has all the information necessary to determine whether the call is valid and carry it out. It’s just that `ccall` arbitrarily rejects this specific syntax. As you say yourself in that same thread I linked above:

What `@generated` is doing here is circumventing the restriction the compiler usually enforces, of having the number of declared arguments match the number of passed arguments. Arguably, this is a bug in how `@generated` interacts with `ccall` and should be fixed.  
The reason it’s not allowed is that, in general, what you wrote may lead to calling an external function with a different number of arguments than it expects, which can lead to all kinds of issues, and splatting alone cannot check that this will work at compile time, since the length is not guaranteed to be statically known information.

For example, [`printf` just trusts](https://jameshfisher.com/2016/11/23/c-varargs/) that there are the same number of arguments passed to it as are declared in the format string. Not passing enough can lead to [security vulnerabilities](https://owasp.org/www-community/attacks/Format_string_attack). `ccall` (ordinarily) enforcing that such a mismatch cannot happen from the julia side is a feature, not a bug. For me at least, that’s enough of a “fundamental reason” not to want this.

---

<div class="post-metadata">

**Author:** ![lmtzx9h4qqnt](https://avatars.discourse-cdn.com/v4/letter/l/74df32/32.png) [@lmtzx9h4qqnt](https://discourse.julialang.org/u/lmtzx9h4qqnt)\
**Post date:** [March 9, 2023, 8:07pm UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/6 "2023-03-09T20:07:00Z")

</div>

I’m aware of the importance of ensuring that functions are called with the correct signatures. There is nothing Julia can do to enforce this – when calling external functions, it’s the programmer’s responsibility to ensure that the function signature is defined correctly in Julia and that, for example, arguments correctly matching the format string are passed to functions like `printf`. And in C, this is just how it works – there is no way for `printf` to verify how many arguments were actually passed to it.

When it comes to making sure that the numbers of argument types and argument values are equal, there is no reason that it has to be done at a syntax level or at compile time – in the simplest case, `ccall` could just check if they match at runtime and throw an exception if they don’t. But this is really a flaw in the design of `ccall`: Why are the argument types and values passed separately? If they were passed together (which is exactly how it works with `@ccall`), there wouldn’t even be any need to check that the lengths match!

Rather, what I think is going on here is that this might be an accidental artifact of history rather than a conscious design decision. As @c42f explains in the thread I linked, `ccall` was created at a very early stage:

> [@Why is ccall syntax?](https://discourse.julialang.org/t/why-is-ccall-syntax/72427/3):
>
> #### The historical answer
> 
> `ccall` dates from _very_ early in Julia’s development. Perhaps around [this commit](https://github.com/JuliaLang/julia/commit/18f456233b280f2b5649c033c1977133c73bfd88) where it was first added to the runtime or [this commit](https://github.com/JuliaLang/julia/commit/e9fbc5c52ff123da27b9cc368b77372fa0618574) where the syntax became more similar to its current form.
> 
> This was back when Julia source code was still stored in `.j` files, and the first `ccall` commit was before macros were even implemented as part of [this commit](https://github.com/JuliaLang/julia/commit/a1b9fee4e72e8438693eddc25d769fab4cc98c9d).

Perhaps splatting in its current form just didn’t exist at the time, and `ccall` simply incorrectly interprets a splatting expression as a single argument? The error message is definitely incorrect and misleading:

```julia
ERROR: syntax: more types than arguments for ccall around REPL[65]:1

```

Also, note that in my example, it’s guaranteed even at compile time that the correct number of arguments is passed, because `args` is of type `NTuple{9, Integer}`.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [March 9, 2023, 8:18pm UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/7 "2023-03-09T20:18:42Z")

</div>

> [@lmtzx9h4qqnt](#):
>
> The error message is definitely incorrect and misleading:
> 
> ```julia
> ERROR: syntax: more types than arguments for ccall around REPL[65]:1
> 
> ```
> 
> Also, note that in my example, it’s guaranteed even at compile time that the correct number of arguments is passed, because `args` is of type `NTuple{9, Integer}`.

The error you see is a syntax error, because `ccall` needs special casing during parsing/lowering to be able to be passed through the compiler and handled as a foreign call, as well as for inserting the calls to `cconvert` and `unsafe_convert`. It cannot see what types the splatted object eventually ends up as and it certainly doesn’t know that it will be a `NTuple{9, Integer}`. Since it fails at the syntax level, there is no later type based compiler magic you can do to fix this, you need to (at minimum) change how `ccall` is handled in parsing & lowering.

`@ccall` is the “escape hatch” that we use right now in lieu of modifying the parser to make `ccall` more special - not to mention that any change to the parser there needs to be backwards compatible with the existing syntax, as to not introduce a breaking change. The relevant Scheme code in the parser is [here](https://github.com/JuliaLang/julia/blob/f288f8886b34949a984c04feb606115904494d1f/src/julia-syntax.scm#L1095-L1128) and [here](https://github.com/JuliaLang/julia/blob/f288f8886b34949a984c04feb606115904494d1f/src/julia-syntax.scm#L2613-L2631). There is a mention of `...` in there, so it does already seem to be aware of varargs to some extent, but not so far as to allow splatting of arguments (likely in part because we’d still need to know how many arguments we actually want to pass during link time, if I’m not mistaken). This is something the `@generated` approach avoids via sleight of hand, by not splatting at runtime, but during compilation/parsing, when the `@generated` function is compiled for the passed in (known & fixed length) arguments.

---

<div class="post-metadata">

**Author:** ![lmtzx9h4qqnt](https://avatars.discourse-cdn.com/v4/letter/l/74df32/32.png) [@lmtzx9h4qqnt](https://discourse.julialang.org/u/lmtzx9h4qqnt)\
**Post date:** [March 9, 2023, 9:15pm UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/8 "2023-03-09T21:15:21Z")

</div>

> [@Sukera](#):
>
> The error you see is a syntax error, because `ccall` needs special casing during parsing/lowering to be able to be passed through the compiler and handled as a foreign call, as well as for inserting the calls to `cconvert` and `unsafe_convert`.

This doesn’t seem to match with @c42f’s statement in the linked thread:

> [@Why is ccall syntax?](https://discourse.julialang.org/t/why-is-ccall-syntax/72427/3):
>
> #### The basic answer
> 
> It doesn’t need to be syntax. `@ccall` is a more recent and much more attractive syntax for the same thing which doesn’t need special support from the parser or lowering.

Not sure who’s right here? 😉

> [@Sukera](#):
>
> Since it fails at the syntax level, there is no later type based compiler magic you can do to fix this, you need to (at minimum) change how `ccall` is handled in parsing & lowering.

Yes, this is what I’m saying: This _shouldn’t_ fail at the syntax level. The length check should instead be done at compile- or runtime.

> [@Sukera](#):
>
> `@ccall` is the “escape hatch” that we use right now in lieu of modifying the parser to make `ccall` more special - not to mention that any change to the parser there needs to be backwards compatible with the existing syntax, as to not introduce a breaking change.

What I’m arguing for is making `ccall` _less_ special. Normal function calls in Julia don’t behave like this syntactically, and given that it looks the same, it would be better for `ccall` to behave as closely to a normal function call as possible.

> [@Sukera](#):
>
> The relevant Scheme code in the parser is [here](https://github.com/JuliaLang/julia/blob/f288f8886b34949a984c04feb606115904494d1f/src/julia-syntax.scm#L1095-L1128) and [here](https://github.com/JuliaLang/julia/blob/f288f8886b34949a984c04feb606115904494d1f/src/julia-syntax.scm#L2613-L2631). There is a mention of `...` in there, so it does already seem to be aware of varargs to some extent, but not so far as to allow splatting of arguments

The `...` in the code refers to specifying varargs in the type signature, as in

```julia
julia> ccall(:printf, Cint, (Cstring, Cint...), "%d %d %d\n", 1, 2, 3);
1 2 3

```

which is yet another special-case syntax that only exists within the context of `ccall`.

> [@Sukera](#):
>
> but not so far as to allow splatting of arguments (likely in part because we’d still need to know how many arguments we actually want to pass during link time, if I’m not mistaken)

I don’t know if “link time” refers to something else in the context of Julia, but in C, you can link to functions perfectly fine without even specifying any function signature. And of course, with varargs you’ll also never know how many arguments are going to be passed, and this number can change, too. So I’m not sure what the problem would be here.

> [@Sukera](#):
>
> This is something the `@generated` approach avoids via sleight of hand, by not splatting at runtime, but during compilation/parsing, when the `@generated` function is compiled for the passed in (known & fixed length) arguments.

And the result is a clearer, more consistent, and more flexible syntax, even within the constraints of the current implementation of `ccall`. Wouldn’t it be better if `ccall` worked like this?

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [March 9, 2023, 9:35pm UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/9 "2023-03-09T21:35:16Z")

</div>

> [@lmtzx9h4qqnt](#):
>
> This doesn’t seem to match with @Chris\_Foster’s statement in the linked thread:

```julia
julia> @macroexpand @ccall printf("abcd"::Ptr{Cchar})::Cvoid
quote
    local var"#1#arg1root" = Base.cconvert(Ptr{Cchar}, "abcd")
    local var"#2#arg1" = Base.unsafe_convert(Ptr{Cchar}, var"#1#arg1root")
    $(Expr(:foreigncall, :(:printf), :Cvoid, :(Core.svec(Ptr{Cchar})), 0, :(:ccall), Symbol("#2#arg1"), Symbol("#1#arg1root")))
end

julia> Meta.@lower ccall(:printf, Cvoid, (Ptr{Cchar},), "abcd")
:($(Expr(:thunk, CodeInfo(
    @ none within `top-level scope`
1 ─ %1 = Core.apply_type(Ptr, Cchar)
│ %2 = Base.cconvert(%1, "abcd")
│ %3 = Core.apply_type(Ptr, Cchar)
│ %4 = Base.unsafe_convert(%3, %2)
│ %5 = $(Expr(:foreigncall, :(:printf), :Cvoid, :(Core.svec(Core.apply_type(Ptr, Cchar))), 0, :(:ccall), :(%4), :(%2)))
└── return %5
))))

```

They both use the same `Expr(:foreigncall)` under the hood and both use the same `cconvert`/`unsafe_convert` duality, because they work the same. They are just different interfaces to the same thing. Like I said, `@ccall` is the same as `ccall`, just in macro form.

> [@lmtzx9h4qqnt](#):
>
> The length check should instead be done at compile- or runtime.

The length check cannot in general be done at compile time. A macro like `@ccall`, just like regular `ccall` doesn’t know anything about the type of the object you end up splatting. There’s nothing you can check at that level other than the raw syntax, and since the FFI in julia uses the `Expr(:foreigncall)` mechanism (which you can’t use at runtime, as `Expr` as used here is a compile time concept), you either have to rework how the compiler does FFI calls, or change parsing/lowering.

> [@lmtzx9h4qqnt](#):
>
> What I’m arguing for is making `ccall` _less_ special. Normal function calls in Julia don’t behave like this syntactically, and given that it looks the same, it would be better for `ccall` to behave as closely to a normal function call as possible.

Well, it’s an FFI call - it cannot, and should not, behave the same, I think. There are some limitations you have to account for due to how the C ABI works here and due to the safety the compiler provides/wants to provide, and at what level 🤷 I for one am thankful for `ccall` checking that I can’t even accidentally introduce an argument mismatch here, and for `@ccall` providing that same guarantee by requiring all arguments to have an annotation (so no splatting can happen there).

> [@lmtzx9h4qqnt](#):
>
> is yet another special-case syntax that only exists within the context of `ccall`.

Yes, and that explicitly exists to allow calling of vararg functions (which already require all of the arguments to have the same type). I guess it could work to have splatting work there if you define the argument types to be to a vararg function, but I’m really unsure of how this would work on a type level. Since vararg requires all arguments to have the same type, we’d have to be able to enforce that any object you’d splat there, would end up as a collection of the same type as specified in the type signature. That seems like a lot of additional checking.

> [@lmtzx9h4qqnt](#):
>
> Wouldn’t it be better if `ccall` worked like this?

I don’t know, because relying on runtime information for a (usually/currently) compile time safety feature seems really shaky to me. There isn’t really a whole lot the compiler can trust here, because allowing splatting would mean relying on e.g. `eltype` to be correct for the given iterable. In other cases, trusting an implementation of a user function to be correct at that level was decided not to be worth it, e.g. in discussions about whether or not fallback implementations for `AbstractArray` should use `@inbounds` or not (trusting the user-type to use `@boundscheck` and `@propagate_inbounds` correctly).

---

<div class="post-metadata">

**Author:** ![lmtzx9h4qqnt](https://avatars.discourse-cdn.com/v4/letter/l/74df32/32.png) [@lmtzx9h4qqnt](https://discourse.julialang.org/u/lmtzx9h4qqnt)\
**Post date:** [March 9, 2023, 10:34pm UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/10 "2023-03-09T22:34:31Z")

</div>

> [@Sukera](#):
>
> They both use the same `Expr(:foreigncall)` under the hood and both use the same `cconvert`/`unsafe_convert` duality, because they work the same. They are just different interfaces to the same thing. Like I said, `@ccall` is the same as `ccall`, just in macro form.

Yes, I know. I’m just saying that the `ccall` interface is really strange and unusual because nothing else in the language works like this, while `@ccall` is just another macro.

> [@Sukera](#):
>
> The length check cannot in general be done at compile time. A macro like `@ccall`, just like regular `ccall` doesn’t know anything about the type of the object you end up splatting. There’s nothing you can check at that level other than the raw syntax, and since the FFI in julia uses the `Expr(:foreigncall)` mechanism (which you can’t use at runtime, as `Expr` as used here is a compile time concept), you either have to rework how the compiler does FFI calls, or change parsing/lowering.

Yes, that would obviously require a change to the language (even if just by adding a new mechanism while keeping the old `ccall`). Perhaps I should say it more explicitly – when I said above that

> [@lmtzx9h4qqnt](#):
>
> Clearly, Julia has all the information necessary to determine whether the call is valid and carry it out. It’s just that `ccall` arbitrarily rejects this specific syntax.

I didn’t mean that it must be possible to somehow work around this _in the current implementation of the language_, but rather, that it should be straightforward to modify the implementation in such a way to make this possible.

> [@Sukera](#):
>
> Yes, and that explicitly exists to allow calling of vararg functions (which already require all of the arguments to have the same type).

That’s another thing I don’t get: Vararg functions can have arguments with arbitrary types (see `printf`!). This requirement only seems to be there because anything else couldn’t easily be made to fit the existing `ccall` syntax. And it seems like all it does is turn off the length check! After all, since I explicitly have to individually pass the arguments to `ccall` anyway, instead of using `...`, I could just add however many argument types I need to make them match the argument values.

> [@Sukera](#):
>
> Well, it’s an FFI call - it cannot, and should not, behave the same, I think. There are some limitations you have to account for due to how the C ABI works here and due to the safety the compiler provides/wants to provide, and at what level 🤷

Python doesn’t seem to have a problem with having FFI calls as perfectly normal function calls:

```python
>>> from ctypes import *
>>> libc = CDLL("libc.so.6")
>>> printf = libc.printf
>>> printf(b"%d %g %s\n", 7, c_double(1.1), b"abc")
7 1.1 abc

```

Now obviously the technical details are completely different in Python, but what I’m saying is that _syntactically_, it should work the same. If that perhaps requires multiple steps or a macro, I think that’s still better than a weird special construct with unclear rules that behaves like nothing else in the language.

> [@Sukera](#):
>
> I for one am thankful for `ccall` checking that I can’t even accidentally introduce an argument mismatch here, and for `@ccall` providing that same guarantee by requiring all arguments to have an annotation (so no splatting can happen there).

Again, I’m not saying that there shouldn’t be error checking, just that that it should be done in a way that’s consistent with the rest of the language.

> [@Sukera](#):
>
> I don’t know, because relying on runtime information for a (usually/currently) compile time safety feature seems really shaky to me. There isn’t really a whole lot the compiler can trust here, because allowing splatting would mean relying on e.g. `eltype` to be correct for the given iterable.

But all the checks are still there even with the `@generated` approach, since currently everything eventually has to go through `ccall`! The fact that it’s possible to make the syntax more powerful using `@generated` just proves that the existing `ccall` syntax arbitrarily limits something that’s already possible.

And why would it be necessary to check `eltype`? `cconvert` and `unsafe_convert` are already called for all arguments. If something of the wrong type is passed, this will just give an error.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [March 9, 2023, 11:41pm UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/11 "2023-03-09T23:41:36Z")

</div>

> [@lmtzx9h4qqnt](#):
>
> it should be straightforward to modify the implementation in such a way to make this possible.

Well, changing how `ccall` lowers and the whole [`Expr(:foreigncall)` machinery](https://docs.julialang.org/en/v1/devdocs/ast/#Expr-types) that’s [underpinning `ccall`](https://github.com/JuliaLang/julia/blob/master/src/ccall.cpp) unfortunately isn’t straightforward. Granted, at least `ccall` is still exposed as a regular `Expr(:call)` in the parsed code and it’s only the lowering stage that would need to change, but at the very least, once you actually want to emit the correct call into C, you [need to know how many arguments you’ll end up passing](https://github.com/JuliaLang/julia/blob/f288f8886b34949a984c04feb606115904494d1f/src/ccall.cpp#L1324), and you better have the correct number or you’ll get very nasty vulnerabilities.

> [@lmtzx9h4qqnt](#):
>
> And why would it be necessary to check `eltype`? `cconvert` and `unsafe_convert` are already called for all arguments. If something of the wrong type is passed, this will just give an error.

Let’s take a look at what `Expr(:foreigncall)` needs [right now](https://docs.julialang.org/en/v1/devdocs/ast/#Expr-types):

> `foreigncall` Statically-computed container for `ccall` information. The fields are:
> 
> - `args[1]` : name  
> The expression that’ll be parsed for the foreign function.
> - `args[2]::Type` : RT  
> The (literal) return type, computed statically when the containing method was defined.
> - `args[3]::SimpleVector` (of Types) : AT  
> The (literal) vector of argument types, computed statically when the containing method was defined.
> - `args[4]::Int` : nreq  
> The number of required arguments for a varargs function definition.
> - `args[5]::QuoteNode{Symbol}` : calling convention  
> The calling convention for the call.
> - `args[6:5+length(args[3])]` : arguments  
> The values for all the arguments (with types of each given in args[3]).
> - `args[6+length(args[3])+1:end]` : gc-roots  
> The additional objects that may need to be gc-rooted for the duration of the call. See [Working with LLVM](https://docs.julialang.org/en/v1/devdocs/llvm/#Working-with-LLVM) for where these are derived from and how they get handled.

To generate all those arguments to the `Expr(:foreigncall)`, you _need_ to know how many arguments you end up passing. To know that, you need to be able to trust `length` of the iterable object, which you don’t know the type of during parsing. The `@generated` approach above gets around that by deferring the actual generation of the `ccall`/`Expr(:foreigncall)` to such a time where you do already know the argument types of the call, which you just don’t have with the current `ccall` interface and changing that is tricky, because that’s the documented API we guarantee for `ccall` to use. This is something a user written macro has access to, so any change there has the potential of being breaking, unfortunately.

Just adding a late expansion of the object/a new function barrier ala the `@generated` approach breaks that `Expr(:call)` interface we currently guarantee, but without that you can’t generate all those `cconvert`/`unsafe_convert` you need to ensure safety (and to be able to call the correct function), due to only knowing the syntax (and possibly the type, depending on the stage you want to do this in, which may not be enough). The only option left would be to insert a check based on the `eltype` of the splatted argument (assuming you do know the type), or actually `collect`ing the splatted argument and checking each object for conformity individually in case of a type instability, before being able to hand them off to `ccall` (which would be quite redundant, since `ccall` would then have to do that internally again to not break the existing interface).

* * *

I mean, all of that is totally possible if we can totally reinvent `ccall`, but I shudder at the thought of what happens when there’s a mismatch between what `length` reports and what is actually produced when you iterate that object. You might end up with values being used by C that aren’t rooted in the GC, which the GC then happily frees while C accesses it - a classic use after free. That’s the sort of thing that just cannot happen when you give the syntactic guarantee of having live objects, due to them being used explicitly in the `ccall` (modulo objects you have to `GC.@preserve` manually due to passing pointers to them into C…). Not to mention that all the current machinery we do have is working purely in the syntactic domain, and isn’t dynamic to the extent that it can call back into user code to figure out how many things will be passed to the C function.

All in all, to me it doesn’t seem worth it to add all of this complexity, for a (seemingly) minor benefit in convenience. The whole compiler pipeline has the information you’re looking for, but not any individual piece (and adding more mechanisms that act like `@generated` is bound to make the compiler devs unhappy, from what I heard them say about `@generated` and the headaches it can cause).

---

<div class="post-metadata">

**Author:** ![lmtzx9h4qqnt](https://avatars.discourse-cdn.com/v4/letter/l/74df32/32.png) [@lmtzx9h4qqnt](https://discourse.julialang.org/u/lmtzx9h4qqnt)\
**Post date:** [March 10, 2023, 1:43am UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/12 "2023-03-10T01:43:09Z")

</div>

I feel like this isn’t really going anywhere. I keep saying that the FFI _interface_ in Julia could be improved, and you keep dismissing that by pointing to _current_ implementation details (although I do appreciate the links to the relevant parts of the code!).

> [@Sukera](#):
>
> I mean, all of that is totally possible if we can totally reinvent `ccall`, but I shudder at the thought of what happens when there’s a mismatch between what `length` reports and what is actually produced when you iterate that object.

> [@Sukera](#):
>
> To know that, you need to be able to trust `length` of the iterable object, which you don’t know the type of during parsing.

So just require it to be a tuple, where the length is given by the type parameter (no need to trust `length`). The `returntype` argument to `ccall` is already required to even be a tuple _expression_, so that’s consistent in some sense. I’d be happy to be able to use splatting even if it means I have to have the arguments in a tuple. My main complaint is that it’s not possible _at all_ to pass the arguments in some kind of container.

Perhaps the easiest way to have a more flexible and consistent interface would be to add a `dynamic_ccall` function, where everything is checked and done at runtime like in Python (which would obviously be less efficient than Julia’s current `ccall`!).

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 10, 2023, 2:12am UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/13 "2023-03-10T02:12:34Z")

</div>

> [@lmtzx9h4qqnt](#):
>
> I keep saying that the FFI _interface_ in Julia could be improved

I agree that it’s technically feasible to implement some version of this! Though it would be a fair amount of work and I don’t think it’s an improvement because it papers over a _lot_ of possible foot guns. Getting foreign calls right is hard enough without adding splatting. Explicit is good here!

But if you disagree, it should be possible to build this feature without modifying Julia itself - by implementing your own macro `@myccall` which lowers the surface syntax (including splatting) down to a special type and passes that to the `@generated` function as demoed in the second post here by @uniment.

This is basically how it would need to work if it was a builtin feature - in general not knowing the type of the thing being splatted, we need runtime dispatch on that type which will then generate the associated code to marshal the splatted arguments and conform to the ABI of the foreign function. It’s all a huge footgun though.

A less scary version would be requiring the thing being splatted to be annotated with its length in some way - then it’s quite easy to write a macro which expands to `ccall` with the appropriate number of arguments. As a purely syntactic transformation this seems a lot less dangerous. It could be part of `@ccall` in principle. Though on balance I still feel it’s a feature to have to be very explicit about the argument list in FFI calls.

---

<div class="post-metadata">

**Author:** ![uniment](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uniment/32/24532_2.png) [@uniment](https://discourse.julialang.org/u/uniment)\
**Post date:** [March 10, 2023, 5:28am UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/14 "2023-03-10T05:28:14Z")

</div>

> [@lmtzx9h4qqnt](#):
>
> The `...` in the code refers to specifying varargs in the type signature, as in
> 
> ```julia
> julia> ccall(:printf, Cint, (Cstring, Cint...), "%d %d %d\n", 1, 2, 3);
> 1 2 3
> 
> ```
> 
> which is yet another special-case syntax that only exists within the context of `ccall`.

![KermitWatGIF](https://global.discourse-cdn.com/julialang/original/3X/0/d/0d15251941c4387ab32aa86bdcf1c16886d47bcb.gif)

How is this even considered valid Julia? 🤣

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [March 10, 2023, 6:16am UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/15 "2023-03-10T06:16:30Z")

</div>

> [@lmtzx9h4qqnt](#):
>
> I feel like this isn’t really going anywhere. I keep saying that the FFI _interface_ in Julia could be improved, and you keep dismissing that by pointing to _current_ implementation details (although I do appreciate the links to the relevant parts of the code!).

I’m not dismissing it , and I’m sorry if it comes across that way. I’m giving references and justifications for why the current interface is the way it is, what its limitations are and how they come to be and why I think that due to those tradeoffs, being dynamic to the extent you seek is unlikely to change (not to mention the big amount of work that entails - which, if someone wants to implement that, these are the places you’ll have to touch, at minimum).

There’s another possible limitation of having `ccall` being dynamic to that extent - it requires the runtime (and potentially even codegen) to be available to generate the specialized call, at least in the `@generated` version and probably even in a “taking the length of a tuple” version, due to likely requiring dynamic dispatch (you can of course technically specialize the implementation to emit all the right things to not need that, but tht would entail making this more special again…). On its own that’s pretty ok, but is a downside in a static compilation context, which is becoming more important in julia every day (think pkgimages or PkgCompiler, or a custom sysimage). Calls involving such a dynamic `ccall` cannot be precompiled if the length of the tuple is unknown (and I’m unsure whether they can be with `@generated`), so there’s definitely a bunch of unknown edge cases that need to be worked out.

> [@lmtzx9h4qqnt](#):
>
> So just require it to be a tuple, where the length is given by the type parameter (no need to trust `length`).

Well, yes and no. It’s true that julia internally can assume the type parameter layout of a tuple since it owns the type, but from a higher level, it’d have to go through `length` again. You could make `ccall` a builtin function so people can’t add functions to it, but I imagine the underlying implementation of that builtin will be quite ugly (not to mention, it would be a guaranteed call into the julia runtime, before finally ending up in the library you ultimately want to call). The current `ccall` doesn’t have that limitation, due to everything already happening in the generated code once the call is emitted, not at runtime.

> [@lmtzx9h4qqnt](#):
>
> The `returntype` argument to `ccall` is already required to even be a tuple _expression_, so that’s consistent in some sense.

It’s required to be a tuple expression because that’s one of the few per-element typed containers `Core` has available that early in the game. The tuple (as shown above) ultimately ends up as a `Core.svec`, which even your hypothetical would likely need to end up as (unless you want every such `ccall` to dynamically allocate an array that can’t ever be eliminated, due to crossing the FFI border 😬 ). It ending up as a `Core.svec` during lowering, resulting from a literal tuple, also means we don’t have to trust `length` or the type parameter of a tuple at all.

---

<div class="post-metadata">

**Author:** ![uniment](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uniment/32/24532_2.png) [@uniment](https://discourse.julialang.org/u/uniment)\
**Post date:** [March 10, 2023, 11:47am UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/16 "2023-03-10T11:47:21Z")

</div>

Less hacky, maybe to your taste:

```julia
julia> ccaller(fun, retT, argT, args...) = ccaller(Val(fun), retT, argT, args...)
       ccaller(fun::Val, retT, argT, args...) = ccaller(fun, retT, Tuple{argT...}, args...)
       @generated ccaller(::Val{fun}, ::Type{retT}, ::Type{argT}, args...) where {fun, retT, argT<:Tuple} = let;
           length(argT.parameters) == length(args) || error("bruh not cool")
           quote @ccall $fun($((:(args[$i]::$(argT.parameters[i])) for i=1:length(args))...))::$retT end
       end
ccaller (generic function with 4 methods)

julia> ccaller(:pow, Cdouble, (Cdouble,Cdouble), 2, 10)
1024.0

```

Splattable:

```julia
julia> ccaller(:pow, Cdouble, ((Cdouble,Cdouble)...,), (2, 10)...)
1024.0

```

---

<div class="post-metadata">

**Author:** ![lmtzx9h4qqnt](https://avatars.discourse-cdn.com/v4/letter/l/74df32/32.png) [@lmtzx9h4qqnt](https://discourse.julialang.org/u/lmtzx9h4qqnt)\
**Post date:** [March 10, 2023, 5:16pm UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/17 "2023-03-10T17:16:15Z")

</div>

Wow, amazing! I didn’t think it’d actually be possible to turn this into a standard Julia function! And there doesn’t seem to be any overhead either (at least with these simple examples)! This even seems like something that’d be great to have in the standard library. Would you mind if I turn this into a package?

I have a few small questions/comments, too:

- Why did you use `let` – just to avoid having to type `function`? 😃
- Similarly, is there any particular reason why you used `quote`/`end` instead of `:()`?
- Sticking with `ccall` instead of `@ccall` would eliminate the need to manually check the lengths, because `ccall` does that anyway.
- It seems like you get infinite recursion when passing something that’s not a tuple as `argT`.

So my version of this would be

```julia
@generated function Ccall(
	::Val{func}, ::Type{returntype}, ::Type{argtype}, args...
) where {func, returntype, argtype <: Tuple}
	return :(ccall(
		func, returntype, ($((x for x in argtype.parameters)...),),
		$((:(args[$i]) for i in eachindex(args))...)
	))
end
Ccall(func::Val, returntype, argtype::Tuple, args...) =
	Ccall(func, returntype, Tuple{argtype...}, args...)
Ccall(func, returntype, argtype::Tuple, args...) =
	Ccall(Val(func), returntype, argtype, args...)

```

---

<div class="post-metadata">

**Author:** ![uniment](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uniment/32/24532_2.png) [@uniment](https://discourse.julialang.org/u/uniment)\
**Post date:** [March 10, 2023, 5:36pm UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/18 "2023-03-10T17:36:21Z")

</div>

> Would you mind if I turn this into a package?

Be my guest!

> Why did you use `let` – just to avoid having to type `function`? 😃

Yup lol, force of habit

> Similarly, is there any particular reason why you used `quote`/`end` instead of `:()`?

Nope, either is fine

> Sticking with `ccall` instead of `@ccall` would eliminate the need to manually check the lengths, because `ccall` does that anyway.

Yeah but then you’d rely on `ccall`; I wanted to write something that could conceivably replace it because `ccall` seems hacky as all heck to me

> It seems like you get infinite recursion when passing something that’s not a tuple as `argT`.

Oops! Fixed 🙏

---

<div class="post-metadata">

**Author:** ![lmtzx9h4qqnt](https://avatars.discourse-cdn.com/v4/letter/l/74df32/32.png) [@lmtzx9h4qqnt](https://discourse.julialang.org/u/lmtzx9h4qqnt)\
**Post date:** [March 10, 2023, 6:00pm UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/19 "2023-03-10T18:00:23Z")

</div>

OK, cool!

> [@uniment](#):
>
> Yeah but then you’d rely on `ccall`; I wanted to write something that could conceivably replace it because `ccall` seems hacky as all heck to me

Hm, I actually thought `@ccall` was something like a wrapper around `ccall`, but they actually just both turn into `Expr(:foreigncall)`. Makes sense though, since `@ccall` actually has more functionality than `ccall`. In that case, `@ccall` really just seems like the better option in all cases!

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [March 10, 2023, 6:22pm UTC](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761/20 "2023-03-10T18:22:58Z")

</div>

> [@lmtzx9h4qqnt](#):
>
> Hm, I actually thought `@ccall` was something like a wrapper around `ccall`, but they actually just both turn into `Expr(:foreigncall)`.

Yes, they use the same underlying functionality, as I pointed out above:

> [@Sukera](#):
>
> They both use the same `Expr(:foreigncall)` under the hood and both use the same `cconvert`/`unsafe_convert` duality, because they work the same. They are just different interfaces to the same thing. Like I said, `@ccall` is the same as `ccall`, just in macro form.

This is out of necessity, because there’s no other way to communicate with the compiler that you want to do an FFI call. Anything more than generating an `Expr(:foreigncall)` requires modification of codegen and custom compiler pass, which is what I tried to get across in my posts above. Maybe that got lost in translation somewhere.

[Next page](https://discourse.julialang.org/t/is-it-possible-to-splat-into-ccall/95761.md?page=2)