Why this code does a segfault?

General Usage
,
1

Why this code does a segfault ?

foo.c

#include <stdio.h>

struct mystruct
{
    int a;
    int b;
};

typedef struct mystruct mystruct_t;

int recvstruct(mystruct_t* st)
{
    printf("C %u, %u\n", st->a, st->b);
    return 0;
}

mystruct_t* updatestruct(mystruct_t* st)
{
    st->a = 10;
    st->b = 11;
    return st;
}

Compilation

gcc foo.c -fpic -shared -o libfoo.so

Julia

struct mystruct
    a::Int
    b::Int
end

function let_c_print_struct()
    ccall((:recvstruct, "./libfoo.so"), Int, (Ref{mystruct},), mystruct(3, 4))
end

function let_c_update_struct()
    st = mystruct(5, 6)
    ret = ccall((:updatestruct, "./libfoo.so"), Ref{mystruct}, (Ref{mystruct}, ), st)

    @show st
    @show ret[]
end

let_c_update_struct()

The segfault is produced when accessing ret[] ?

2 
julia> struct mystruct
           a::Int32
           b::Int32
       end

julia> st = mystruct(5, 6)
mystruct(5, 6)

julia> ret = ccall((:updatestruct, "./libfoo.so"), Ref{mystruct}, (Ref{mystruct}, ), st)
Base.RefValue{mystruct}(mystruct(10, 11))

Int is Int64 in Julia if you’re on 64-bit system

2 Likes
3

@jling I’m on 64-bit system, initially it was Int32 but the segfault still occurred. Your code is working, but the difference is that st is a global variable. It seems Julia does not like local variables. Why ?

4

not entirely sure, it seems you can just return ret, instead of trying to access it like ret[]

5

Yes, you are right ! When, acceding to ret instead of ret[] no more segfault. Working Julia code

struct mystruct
    a::Int32
    b::Int32
end

function let_c_print_struct2(v::mystruct)
    ccall((:recvstruct, "./libfoo.so"), Int32, (Ref{mystruct},), v)
end

function let_c_update_struct2()
    st = mystruct(5, 6)
    ret = ccall((:updatestruct, "./libfoo.so"), Ref{mystruct}, (Ref{mystruct},), st)
    let_c_print_struct2(ret)
    ret
end

ret = let_c_update_struct2()
let_c_print_struct2(ret)

Seems that Julia is automatically converting struct to Ref{struct}

1 Like
6

PS: I forget to give the original link: Passing a pointer to struct as an argument in Julia - Stack Overflow in where the author said:

  • (C code side) return the pointer, since Julia seems to pass a pointer to a copy of the struct
  • (Julia code side) ccall seems to automatically converts non Ref to Ref (but in my case also does the inverse)
7

May I ask why Ref{mystruct} is used as a return type?

8

It’s Base.cconvert(Ref{mystruct}, st) working behind the scenes.

This is probably unsafe. The return value of Base.cconvert is only preserved from GC during this ccall: ccall((:updatestruct, "./libfoo.so"), Ref{mystruct}, (Ref{mystruct},), st). When the ccall returns, ret will store a pointer to the return value of Base.cconvert(Ref{mystruct}, st) and this value may be GC-ed at any time, so it’s not safe to use it in let_c_print_struct2.

If this code works fine, then it’s probably because 1. when Ref{mystruct} is used as a ccall return type, Julia can be notified to preserve a certain value to make its lifetime expands the returned Ref{{mystruct}; 2. you were so lucky that no GC got triggered between these two calls.

I’d say the de facto way of using Ref in ccall is:

function let_c_print_struct2(v_ref::Ref{mystruct})
    ccall((:recvstruct, "./libfoo.so"), Int32, (Ptr{mystruct},), v_ref)
end

function let_c_update_struct2()
    st_ref = Ref(mystruct(5, 6))
    ccall((:updatestruct, "./libfoo.so"), Ptr{mystruct}, (Ptr{mystruct},), st_ref)
    let_c_print_struct2(st_ref)
    st_ref
end

Now, there is no tmp variable that needs to be created when doing the conversion, so no GC-related segfaults.

5 Likes
9

@Gnimuc This code is not mine, I’m trying to deal with similar code which is still segfaulting. But it seems that Julia made a copy and therefore the content is not modifiable. My hypothesis is maybe to use Ptr instead of Ref or set the struct mutable. The official documentation is not very easy to understand for me and missed of some ultra-basic examples.

10

@Gnimuc Ok your code seem working :slight_smile: and I can return st_ref[] without segfault

11

If mystruct is an immutable isbitstype, then mystruct(5, 6) doesn’t need to have a valid memory address(you cannot use pointer_from_objref on these types), in some cases, it might be optimized out by the compiler.

To pass this value to a C function argument that expects a pointer, the first thing we need to do on the Julia side is to give it a valid pointer. That’s where the Ref types come in.

I’d say it’s better to explicitly create this pointer by using st_ref = Ref{mystruct}(mystruct(5, 6)), rather than use (Ref{mystruct},) as the ccall argument type for implicitly doing the conversion.

3 Likes
12

Would have you mean instead :
I’d say it’s better to explicitly doing this by writing Ref{mystruct(5, 6)} instead of using mystruct(5, 6) in the ccall argument type

?

13

edited

14

In the beginning, In my personal code I was doing st_ref = Ref(mystruct(5, 6)) but with a struct having fields with pointers (referring to a static C memory) but since I had segfault I found the code I posted on stackoverflow but which is also segfaulting, so that is why I asked this question. With these fresh information, I’ll keep investigating why my code is not working.

Thanks for your help !

15

For this type of usage where a type has to match a C type, there are many Julia aliases for C types like Cint in this case, which will match the size of int on your system. These should be used in cases like this. That doesn’t seem to be the main issue here, however.

6 Likes
16

@StefanKarpinski thanks, the issue does not come here. My real code is the following (minified to follow the initial code):

foo.c

// gcc foo.c -fpic -shared -o libfoo.so

#include <stdio.h>
#include <assert.h>

struct mystruct
{
    double* data;
    size_t size;
};

typedef struct mystruct mystruct_t;

void recvstruct(mystruct_t const* st)
{
    assert(st != NULL);
    printf("mystruct=%g, %g, %g (%zu)\n", 
           st->data[0], st->data[1], st->data[2], st->size);
}

void updatestruct(mystruct_t* st)
{
    static double values[16] = { 1.0, 2.0, 3.0 };

    assert(st != NULL);
    st->data = values;
    st->size = 3u;
}

julia

struct Mystruct
    data::Ptr{Float64}
    size::Csize_t
    Mystruct() = new(Ptr{Float64}(), 0)
end

function print_struct(v::Ref{Mystruct})
    ccall((:recvstruct, "./libfoo.so"), Cvoid, (Ref{Mystruct},), v)
end

function update_struct()
    st = Ref(Mystruct())
    ccall((:updatestruct, "./libfoo.so"), Cvoid, (Ref{Mystruct},), st)

    # This will work everytimes
    print_struct(st)

    # But on a bigger application this will crash (even st[] will crash)
    V = Vector{Float64}(undef, st[].size)
    for i in 1:st[].size
        V[i] = unsafe_load(st[].data, i)
    end
    return V
end

V = update_struct()

Which in this case works well

mystruct=1, 2, 3 (3)
3-element Vector{Float64}:
 1.0
 2.0
 3.0

But this idea, within a bigger C++ application, will crash:

  • where static double values[16] is in fact static std::vector<double> values
  • st->data = values comes in fact from std::vector::data()
  • st->size = 3u comes in fact from std::vector::size()
  • the Mystruct is a just a C structure to make the transition between the C++ std::vector and the Julia Vector without making copies (thanks to static which allows the local variable to lives like a scoped global variable).

Do you see something obviously odd ? Is possibly the GC doing this ?

Allocations: 753506 (Pool: 753256; Big: 250); GC: 1

I hope to find it quickly by myself, this is probably something obvious.

And when doing show(st) I have this segfault:

CSparseMatrix_t:3 = 1, 2, 3
Base.RefValue{CSparse}(CSparse(
signal (11): Erreur de segmentation
in expression starting at /home/qq/MyGitHub/xxx.jl:4
jl_object_id_ at /buildworker/worker/package_linux64/build/src/builtins.c:371
type_hash at /buildworker/worker/package_linux64/build/src/jltypes.c:981
typekey_hash at /buildworker/worker/package_linux64/build/src/jltypes.c:993 [inlined]
lookup_type at /buildworker/worker/package_linux64/build/src/jltypes.c:585
inst_datatype_inner at /buildworker/worker/package_linux64/build/src/jltypes.c:1157
jl_inst_arg_tuple_type at /buildworker/worker/package_linux64/build/src/jltypes.c:1429
arg_type_tuple at /buildworker/worker/package_linux64/build/src/gf.c:1836 [inlined]
jl_lookup_generic_ at /buildworker/worker/package_linux64/build/src/gf.c:2363 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2415
_show_default at ./show.jl:412
show_default at ./show.jl:395 [inlined]
show at ./show.jl:390
unknown function (ip: 0x7fcee49197d5)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
_show_default at ./show.jl:412
show_default at ./show.jl:395 [inlined]
show at ./show.jl:390
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
show at ./show.jl:392
update_struct at /home/qq/MyGitHub/xxx.jl:567
unknown function (ip: 0x7fcee4914e3a)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:115
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:204
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:155 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:561
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:669
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:877
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:825
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:929
eval at ./boot.jl:360 [inlined]
include_string at ./loading.jl:1094
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
_include at ./loading.jl:1148
include at ./client.jl:444
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:115
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:204
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:155 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:561
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:669
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:877
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:825
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:929
eval at ./boot.jl:360 [inlined]
eval_user_input at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:139
repl_backend_loop at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:200
start_repl_backend at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:185
#run_repl#42 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:317
run_repl at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:305
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
#874 at ./client.jl:387
jfptr_YY.874_46600.clone_1 at /home/qq/julia-1.6.0/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
jl_f__call_latest at /buildworker/worker/package_linux64/build/src/builtins.c:714
#invokelatest#2 at ./essentials.jl:708 [inlined]
invokelatest at ./essentials.jl:706 [inlined]
run_main_repl at ./client.jl:372
exec_options at ./client.jl:302
_start at ./client.jl:485
jfptr__start_41020.clone_1 at /home/qq/julia-1.6.0/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
true_main at /buildworker/worker/package_linux64/build/src/jlapi.c:560
repl_entrypoint at /buildworker/worker/package_linux64/build/src/jlapi.c:702
main at julia (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4007d8)
Allocations: 747206 (Pool: 746931; Big: 275); GC: 1
Erreur de segmentation
17

You probably need to use GC.@preserve to make sure that st is not garbage collected before you access its fields. Almost by definition if Julia’s GC collecting something causes a crash when using ccall, it’s your fault. The only exception is if it collects something it shouldn’t but that’s extremely rare these days. It’s on you to tell it what it cannot collect, which is what GC.@preserve does.

2 Likes
18

The segfault happened within the show function and it looks like you were using a custom show method. However, you did not give us enough info for diagnosing. An MWE should be self-contained, not an oversimplified one.

The correct way to use Ref always works regardless of the application size. There must be another root cause that triggered the issue.

19

@Gnimuc it’s not the fault to show(), since:

// C struct
typedef struct mystruct
{
    double* data;
    int size;
} mystruct_t;

...

# Julia
struct Mystruct
    data::Ref{Float64}
    size::Csize_t
    Mystruct() = new(Ref{Float64}(), 0)
end

On 64-bits archi C and Julia sizeof are returning 16. Before calling ccall() the show() is returning me the good value and I can display good addresses.

julia> st = Ref(Mystruct())

julia> println(st)
Base.RefValue{Mystruct}(Mystruct(Base.RefValue{Float64}(0.0), 0x0000000000000000))

julia> println("Julia Before S=", repr(UInt64(pointer_from_objref(st))), ", data=", repr(UInt64(pointer_from_objref(st[].data))))
Julia Before S=0x00007f30b6d76110, data=0x00007f30b6d7c290

Which match the C addresses: C Before S=0x7f30b6d76110 data=0x7f30b6d7c290 (S address of the struct and data the address of the data).

But after calling ccall which makes mystruct_t::data refer to a valid C block of memory (static C array), things becomes bad:

  • C After: S=0x7f30b6d76110 data=0x7f30677ca880 here data now refers to correct block of C memory
  • print_struct(N) which using ccall() is working well since the result is good mystruct=1, 2, 3 (3)
  • Julia address of the struct is still good but segfault when accessing to address of data: Julia After S=0x00007f30b6d76110, data= signal (11): Erreur de segmentation`

@StefanKarpinski If I do

    st = Ref(Mystruct())

    GC.@preserve st begin
       ccall((:updatestruct, "./libfoo.so"), Cvoid, (Ref{Mystruct},), st)
       st[] # this still segfault
   end

This still segfault. What is the syntax to protect data ? Looks like Julia cannot access to the C memory.

Afaik I cannot call valgrind julia since there is too many of Conditional jump or move depends on uninitialised value before calling my script

20

Is this a typo?

To match typedef struct mystruct, the correct version is:

struct Mystruct
    data::Ptr{Cdouble}
    size::Cint
end

If you don’t know how to correctly map a C struct to Julia, you can use Clang.jl’s binding generator:

pkg> dev Clang

julia> using Clang.Generators

shell> vim test.h

shell> cat test.h
typedef struct mystruct
{
    double* data;
    int size;
} mystruct_t;

julia> ctx = create_context("test.h")
[ Info: Parsing headers...
Context(...)

julia> build!(ctx)
[ Info: Processing header: test.h
[ Info: Building the DAG...
┌ Warning: default libname: ":libxxx" is being used, did you forget to set `library_name` in the toml file?
└ @ Clang.Generators ~/.julia/dev/Clang/src/generator/audit.jl:16
[ Info: Emit Julia expressions...
struct mystruct
    data::Ptr{Cdouble}
    size::Cint
end

const mystruct_t = mystruct

[ Info: Done!
Context(...)
2 Likes