Designing a C ABI layer for a multi-lingual project

General Usage
1

Hello all,

I’m interested in creating a multi-lingual project with a Julia backend. I’d like to do this through a C ABI layer and would like advice on how to structure it.

Since my usecase is a bit difficult to explain, I’ve likened it to a basic video game. Here’s my current idea of how to structure it:

Julia has a world struct and some way to update the world:

module Game

mutable struct World
    playerx :: Float32
    playery :: Float32
    playerhealth :: Int
end

function step!(w::World, ...)
  ...
end

end #module

It has a function callable from C that creates a World and returns a pointer:

module CLayer

const _registry = Dict{UInt, Base.RefValue{Model}}()

Base.@ccallable function create_world(...) :: Ptr{Nothing}
    w = Game.World(...)
    r = Ref(w)
    p = UInt(pointer_from_objref(r))
    _registry[p] = r  
    return Ptr{Nothing}(p)
end

end #module

Module CLayer also has a function callable from C that updates the world:

Base.@ccallable function step(ptr::Ptr{Nothing}, ...) :: Cvoid
    w = _ref(ptr)
    Game.step!(w[], ...)

    unsafe_copyo!(ptr, pointer(encode(w), length(w))

    nothing
end

I could then compile that with

using PackageCompiler

create_library(
        \"$IN_DIR\",
        \"$OUT_DIR\";
        lib_name        = \"game\",
        incremental     = false,
        filter_stdlibs  = false,
        force           = true,
)

Then, I would have a header file that looked something like this:

#include <stdint.h>

void* create_world(...);
void step(void* world, ...);

I think this should allow for other languages to then implement these C methods. I’ve heard of a similar approach being used for plugins, but I think that would require Julia code also calling C code.

I was wondering what others thought of this approach. Do you think it will scale? Is it performant? Is there a bug somewhere?

Please excuse the many beginner mistakes I’m sure I have made in this post – this is definitely not my forte.

2

// 13. Marshal Arg Op (Julia-aligned pointer → C-packed struct value)

def MarshalArgOp : JLCS_Op<"marshal_arg", [MemoryEffects<[MemRead]>]> {
  let summary = "Pack aligned Julia struct into C-packed layout for FFI call.";
  let description = [{
    Reads an aligned Julia struct from a pointer, loading each field at its
    Julia-aligned byte offset, and assembles them into a packed LLVM struct
    suitable for passing to a C/C++ function.

    The operation takes a pointer to the Julia-side aligned struct and produces
    the packed struct value. The attributes carry the layout mismatch info:
    - memberTypes: the MLIR types of each struct member (for typed loads)
    - juliaOffsets: the byte offset of each member in the Julia-aligned layout

    Example:
    ```mlir
    %packed = jlcs.marshal_arg %ptr
      { memberTypes = [i32, f64], juliaOffsets = [0 : i64, 8 : i64] }
      : (!llvm.ptr) -> !llvm.struct<packed (i32, f64)>
    ```
  }];

Then for strided arrays

```tablegen
%elem = jlcs.load_array_element %view[%i, %j, %k]
      : !jlcs.array_view<f64, 3> -> f64

I been trying to inverse this op so C can call into a thunk packed by julia, which is how i have it setup for julia to call into a thunk compiled by the op

3

Very cool! I’m having trouble finding anything about this besides general info about marshalling. Can you send some link?

4

Heres the tablegen operations but the docs for the project show how I use mlir to marshal the abi between julia and c/cpp then julia uses the thunk with the .so and both sources stay the same, its also dwarf driven and only enums gets parsed from headers, the compilation and builds, linking and bindings are driven by julia, the dialect isnt though, its the cpp coupling for the ffi part and needs to be built with cmake and its only supported by linux if you use it.

You would need to expand these ops for c to julia, but I been focused going from high level to low