Subtleties of type and immutable and memory layout when using ccall

General Usage
1

I have spent considerable time finding a bug in my code related to memory layout of types and immutables when interfacing with structs in c. I want to summarize here to check my understanding and as it may be useful to others.

When to use immutable
Firstly, I have been calling a code that requests and array of structs (memory owned by Julia). The C code is like

typedef struct {
  int size;
  void * data;
} bob_t;

int fill_bob(bob_t * bobs, int nbobs);

which I would call from Julia using

immutable bob_t
    size::Cint
    data::Ptr{Void}
end

function fill_bob(size)
    bobs = Array{bob_t,1}(size)
    err = ccall((:fill_bob, library), Cint, (Ref{bob_t}, Cint), bobs, size)
    bobs
end

In this case one must use an immutable, because C expects the structs to be stored inline inside the array. If I use a type for bob_t then AFAICT the array bobs only holds references to the types, and so the above code doesn’t work. Is that correct? Is there a data structure that stores types inline?

When to use type
On the other hand sometimes one needs a longer term reference to a specific struct that is shared between Julia and C. e.g.

typedef struct {
    int size;
    void * data;
} frank_t;

int do_something(frank_t * frank);

In Julia I often want to store such a struct inside another type.

type frank_t
    size::Cint
    data::Ptr{Void}
end

type Frank
    f::frank_t
    data::Array{Uint8,1}
end

function Frank(size)
    data = Array{Uint8,1)(size)
    Frank(frank_t(size, pointer(data, 1)), data)
end

function do_something(frank::Frank)
    err = ccall((:do_something, library), Cint, (Ref{frank_t},), frank.f)
end

In this case I keep a Julia reference to the array in Frank, which corresponds to the pointer inside of frank_t. Then I can call the c functions as necessary to work on frank_t, but also access the data directly in Julia using the data array field of Frank.

In this case it is essential that frank_t and Frank are type and not immutable, because otherwise the actual address of frank_t changes from call to call. If the C library stores references to frank_t internally (as the one I’m using does) then this can lead to subtle bugs.

Does that make sense? Did I miss something obvious?

Thanks!

2 Likes
2

It seems like the discussion in:

Has ended up with a similar content.

3

Yes, that all seems correct. I would just add that Ref{frank_t} is constructible specifically for the purpose that you can use it to get a stable address pointing to any type frank_t. You can even use it to point to an offset into an array holding frank_t objects, or to C memory declared as Ptr{frank_t}.

4

Regarding inlining, the distinction is not type vs immutable, but whether isbits(T) is true. See this manual section. A type will never be isbits==true; an immutable may be either true or false, depending on the field types.

edit: also, regarding the question of inlined type – i.e. mutable – objects see here for some of the design issues and tradeoffs involved in the choice of demarcation points around mutability.

1 Like
5

Aha, of course Ref{frank_t} is the solution. So in my second case, where I thought we must use a type, I can instead do:

immutable frank_t
    size::Cint
    data::Ptr{Void}
end

immutable Frank
    f::Ref{frank_t}
    data::Array{Uint8,1}
end

function Frank(size)
    data = Array{Uint8,1)(size)
    Frank(Ref{frank_t}(frank_t(size, pointer(data, 1))), data)
end

function do_something(frank::Frank)
    err = ccall((:do_something, library), Cint, (Ref{frank_t},), frank.f)
end

Which is more explicit and also makes me happier as I prefer immutable when dealing with c structs as they are also value types.