Definition of struct written from C and read-only in Julia

General Usage
,
1

I hand over a struct to a C library which periodically will enter values in the struct’s field and then call a Julia callback function. The struct is defined and gets created (memory allocated) in Julia, but only C writes to it and Julia should never write into that memory during regular program execution. The struct is provided to C as a Ref and received from C as a Ptr. The struct’s fields are (at least currently) only primitive types.

I wonder, what the correct definition in Julia is. If my reading of the documentation is correct, the struct must be defined mutable to guarantee a stable memory address, although the values should not be modified in Julia.

As the struct is mutable, from a user’s perspective it seems to be correct to at least declare all fields as const to see if some code accidentally tries to write into the fields in Julia. However, is it correct? Or is Julia then allowed to do some optimizations assuming the field values will never change while C is sneaking in modifications behind Julia’s back?

I other words: What combination of mutable and const is the right choice and are there further options to solve the problem?

2

If you make a field const in Julia it might land on the stack again, which is what you want to avoid. I think it should all be mutable.

You could instead overwrite setproperty! to avoid that generic Julia code accidentally changes some fields. But it is also in pure Julia settings often the case that users should never directly change fields, e.g. maybe just a bit of documentation + giving the users all access via functions in Julia could be sufficient.

2 Likes
3

Even though the struct itself is immutable, the Ref to it is mutable a d has a stable address.

julia> struct Foo
           a::Int
           b::Float64
       end

julia> rfoo = Ref(Foo(2, 3.5))
Base.RefValue{Foo}(Foo(2, 3.5))

julia> function bar(ptr::Ptr{Nothing})::Int8
           a_ptr = Ptr{Int}(ptr)
           b_ptr = Ptr{Float64}(ptr + 8)
           unsafe_store!(a_ptr, 9)
           unsafe_store!(b_ptr, 5.6)
           return 0
       end
bar (generic function with 1 method)

julia> bar_ptr = @cfunction(bar, Int8, (Ptr{Nothing},))
Ptr{Nothing} @0x0000007d5c2ad7a0

julia> @ccall $bar_ptr(rfoo::Ref{Foo})::Int
0

julia> rfoo[]
Foo(9, 5.6)
4

That’s interesting. And, if I may add, calling the self-generated C-compatible function pointer via @ccall is really clever.

However, is it guaranteed to work in general? Your example obviously works, but I read the documentation

in some cases the compiler is able to avoid allocating immutable objects entirely

and

In order to support mutation, such objects [mutable structs] are generally allocated on the heap, and have stable memory addresses.

such that although unmutable structs can have stable addresses (as in your example), depending on (unspecified) circumstances this is not guaranteed. But the documentation might be imprecise or my reading faulty.

5

You are focusing on the immutable struct, which in this case is also a bitstype. Basically, it is a compound primitive. However, the operable type here is the Ref, concretely a Base.RefValue{Foo}.

This could basically be defined as follows.

mutable struct MyRef{T}
    x::T
end
Base.getindex(mr::MyRef) = mr.x

Constructing a MyRef results in an allocation.

julia> @time mr = MyRef(5)
  0.000004 seconds (1 allocation: 16 bytes)
MyRef{Int64}(5)

julia> mr[]
5

MyRef also thus has a defined location on the heap.

julia> @time myref = MyRef(Foo(3, 4.5))
  0.000003 seconds (1 allocation: 32 bytes)
MyRef{Foo}(Foo(3, 4.5))

julia> pointer_from_objref(myref)
Ptr{Nothing} @0x0000007d73bc76b0

julia> pointer_from_objref(myref)
Ptr{Nothing} @0x0000007d73bc76b0

The pointer is thus to the mutable Base.RefValue{Foo}. Because Foo is stored inline as a bitstype, we can access Foo’s fields by dereferencing the reference.

There is not an inconsistency in the documentation. An instance of Foo does not have a defined address, but a Base.RefValue{Foo} does.

6

Ahh, thanks @mkitti that was very enlightening. So as long as the instance of the possibly immutable struct is wrapped into a Ref in a @ccall (which is recommended in the documentation), C will always get the correct address, because the struct’s address is identical to the Ref’s address and the latter one is always defined (as mentioned in Ref’s documentation).

So I understood: The address is not the limiting factor.

Putting that aside: Telling Julia (or the user) that it’s immutable (or const) while values get constantly changed by the C library is still not a good idea or is it?

7

If the Julia user is accessing an “immutable” struct via the Ref then things should be fine. The Ref is mutable and the compiler will not assume that the Ref is always pointing to the same struct instance.

The potential danger of mutating something that should not be mutated is that the compiler may incorrectly assume that the value cannot change. However, if you have it contained within a Ref there is no problem.

1 Like
8

Thanks again @mkitti . As the current interface is to not put everything behind a Ref in Julia (only for C), the struct should be mutable with non-const fields.

But good to know that we could change that in the future if need arises, with both changes (Ref + immutable) done in parallel.