Is it possible to splat into ccall?

General Usage
21

No, that’s all understood. For some reason, I just got the idea at some point that @ccall defers to ccall (instead of making an Expr(:foreigncall) directly). Perhaps because of statements like these:

But they’re not the same! @ccall can call varargs functions with arbitrary (mixed) types, whereas varargs arguments must all be of the same type with ccall. These are equivalent:

julia> ccall(:printf, Cint, (Cstring, Cint...), "%d %d %d\n", 1, 2, 3);
1 2 3

julia> @ccall printf("%d %d %d\n"::Cstring; 1::Cint, 2::Cint, 3::Cint)::Cint;
1 2 3

But there’s no way to express the following using ccall:

julia> @ccall printf("%d %f %s\n"::Cstring; 1::Cint, 2.0::Cdouble, "abc"::Cstring)::Cint;
1 2.000000 abc

(This is also stated in the documentation.)

Edited to add: I just realized there does seem to be one thing that ccall can do and @ccall can’t, which is specifying a calling convention? (Seems to be hard-coded to :ccall, if I understand the code correctly.) Should be straightforward to add, but it’s not in the current implementation.

22

Not sure I follow - which part of that link are you referring to?

julia> ccall(:printf, Cint, (Cstring, Cint, Cdouble, Cstring), "%d %f %s\n", 1, 2.0, "Hello");
1 2.000000 Hello

Maybe that’s illegal and a bug, maybe everything is passed as a *void, I don’t know :person_shrugging: Could also just be working due to printf taking its type information from the format string and not needing the “”“type safety”“” that a same-typed va_list would provide.

Anyway, like I said, they’re the same. They lower to the same Expr(:foreigncall) machinery. The only difference is in who ends up building that Expr, the parser directly or the @ccall macro on its own.

I did find these while looking for a PR that made this work, if that exists. I am on the side of Jeff here though, so… :person_shrugging:

1 Like
23

I’m moderately sure that won’t work on Windows, and it “accidentally” works on Linux because it uses a different calling convention, but it can’t be relied upon. The @ccallmacro has a much nicer interface, on the other hand there are things that you can’t do with that instead you can with ccall (e.g. specifying the calling convention).

2 Likes
24

Indeed, this is what I get:

julia> ccall(:printf, Cint, (Cstring, Cint, Cdouble, Cstring), "%d %f %s\n", 1, 2.0, "Hello");
1 0.000000 Hello

This problem unfortunately extends to my ccaller code above; we’d want a mechanism for specifying where the variadic arguments begin.

My ccaller code also has no provisions for specifying calling convention—although it seems like the place to start is putting it in the @ccall macro. If that is done, afaict @ccall’s functionality would be a strict superset of ccall, without requiring such weird hacks to the language semantics.

25

Yes, we could have added support for other calling conventions in the original @ccall PR. But we skipped that because it was a lot of discussion already to get to the point of merging the current version. And it seemed better to merge it than to get stuck working out notation for calling conventions or other attributes (Ccallmacro by ninjaaron · Pull Request #32748 · JuliaLang/julia · GitHub)

If someone wanted to figure out the nitty gritty detail of calling convention for FFI calls, I’m sure that would be a great PR. Last time we weren’t sure if we needed a general annotations syntax for defining side effects, but now we have @assume_effects ... @ccall. I’m still not sure whether there’s other annotations which might be needed.

1 Like
26

Sorry, it can’t be linked to directly. I meant this part:

System Independent Types

C name Fortran name Standard Julia Alias Julia Base Type
… … … …
va_arg Not supported
... (variadic function specification) T... (where T is one of the above types, when using the ccall function)
... (variadic function specification) ; va_arg1::T, va_arg2::S, etc. (only supported with @ccall macro)

As others have pointed out, some calling conventions differentiate between varargs and “normal” args. The difference is in the nreq field in the Expr(:foreigncall):

julia> @macroexpand @ccall printf("%d %f %s\n"::Cstring; 1::Cint, 2.0::Cdouble, "Hello"::Cstring)::Cint
quote
    local var"#44#arg1root" = Base.cconvert(Cstring, "%d %f %s\n")
    local var"#45#arg1" = Base.unsafe_convert(Cstring, var"#44#arg1root")
    local var"#46#arg2root" = Base.cconvert(Cint, 1)
    local var"#47#arg2" = Base.unsafe_convert(Cint, var"#46#arg2root")
    local var"#48#arg3root" = Base.cconvert(Cdouble, 2.0)
    local var"#49#arg3" = Base.unsafe_convert(Cdouble, var"#48#arg3root")
    local var"#50#arg4root" = Base.cconvert(Cstring, "Hello")
    local var"#51#arg4" = Base.unsafe_convert(Cstring, var"#50#arg4root")
    $(Expr(:foreigncall, :(:printf), :Cint, :(Core.svec(Cstring, Cint, Cdouble, Cstring)), 1, :(:ccall), Symbol("#45#arg1"), Symbol("#47#arg2"), Symbol("#49#arg3"), Symbol("#51#arg4"), Symbol("#44#arg1root"), Symbol("#46#arg2root"), Symbol("#48#arg3root"), Symbol("#50#arg4root")))
end

julia> Meta.@lower ccall(:printf, Cint, (Cstring, Cint, Cdouble, Cstring), "%d %f %s\n", 1, 2.0, "Hello")
:($(Expr(:thunk, CodeInfo(
    @ none within `top-level scope`
1 ─ %1 = Base.cconvert(Cstring, "%d %f %s\n")
│   %2 = Base.cconvert(Cint, 1)
│   %3 = Base.cconvert(Cdouble, 2.0)
│   %4 = Base.cconvert(Cstring, "Hello")
│   %5 = Base.unsafe_convert(Cstring, %1)
│   %6 = Base.unsafe_convert(Cint, %2)
│   %7 = Base.unsafe_convert(Cdouble, %3)
│   %8 = Base.unsafe_convert(Cstring, %4)
│   %9 = $(Expr(:foreigncall, :(:printf), :Cint, :(Core.svec(Cstring, Cint, Cdouble, Cstring)), 0, :(:ccall), :(%5), :(%6), :(%7), :(%8), :(%4), :(%3), :(%2), :(%1)))
└──      return %9
))))

Well, his suggestion doesn’t work when the number of arguments is unknown and can be arbitrary, which kind of defeats the whole point of varargs functions.

27

Yes, that makes sense - good to know that my gut feel about this being dangerous was right :slight_smile: I don’t think there’s a (cross platform) way of checking preemptively whether what we want to call is variadic, right? Aaah, the perils of C ABIs…

Gotcha - I took that part of the docs as a “this is how we support it” and not as a “other ways don’t work”. Maybe the difference in calling conventions should be added here as a warning, to make it clear that just passing the extra vararg arguments as regular arguments does not work?

Yeah, I was more referring to the “can’t be done portably” part :slight_smile:

28

Ah OK – I actually didn’t understand that part. What would the portability issue be? As far as I can tell, the original issue was just asking about turning a Julia tuple into the arguments to a ccall, i. e. exactly the same as my original question, which is all just a matter of syntax.

29

Alright, I’ve now created Ccalls.jl as package implementing the @generated approach!

1 Like
30

A couple notes and ideas, fwiw

  1. For separating args from varargs, I see appeal in using …, but it seems like : would be more ergonomic. Subjective of course. You could use any other operator too, like |, >, etc, or maybe wrap the varargs in their own tuple.
  2. For the method using Pairs like 2=>Cint, it feels a bit backwards; Cint=>2 feels more natural to me. Another idea would be to use Cint: 2, though that’s likely to get pushback as it would entail type piracy, but that could be solved by getting it into Base.
  3. As discussed in this thread, there might be difficulties in getting some programs that use such a generated function to compile statically. That said, ccall already seems to cause trouble for static compilation. I haven’t dug in to understand why.

If we’re willing to diverge from existing semantics a bit more, maybe something like this becomes compelling:

Ccall(Cdouble: :pow, Cdouble: 2, Cdouble: 10)
Ccall(Cint: :printf, Cstring: "%d %f %s\n", (Cint: 1, Cdouble: 2.0, Cstring: "Hello"))

or maybe:

Ccaller(Cdouble, :pow, Cdouble, Cdouble)(2, 10)
Ccaller(Cint, :printf, Cstring, (Cint, Cdouble, Cstring))("%d %f %s\n", (1, 2.0, "Hello"))

Chef’s choice.

1 Like
31

Thanks for the feedback!

I already have .. as an alias for … (apparently forgot to document it, though). : would be another option, but .. seems clearer to me.

I agree that it can seem strange because of the “arrow” syntax, but I found it important to keep the same order as in other Julia syntax, which is value::Type. I think of it as “2 is converted to Cint”, which is true!

I wanted to keep it simple since the whole @generated construction is already somewhat obscure, so I didn’t think about stuff like hijacking :.

I do like this! Hm, so many options. It seems more confusing than anything else to have 4 different ways of writing the same thing, but on the other hand, this is a bit of an experiment, so I guess it wouldn’t hurt trying it?

32

There’s also ∷ (\Colon) which is tempting and wouldn’t be type piracy, but it probably looks too close to :: to avoid confusion.