C struct garbage collection not run frequently enough

[mattsignorelli]

Also I tried just including and linking with libjulia, but it doesn’t have a symbol for jl_free?

nm $HOME/.julia/juliaup/julia-1.10.3+0.aarch64.apple.darwin14/lib/libjulia.dylib | grep 'jl_malloc\|jl_calloc\|jl_realloc\|jl_free'
00000000000029b8 T _jl_calloc
000000000000c718 s _jl_calloc_addr
0000000000003480 T _jl_malloc
000000000000ce48 s _jl_malloc_addr
000000000000348c T _jl_malloc_stack
000000000000ce50 s _jl_malloc_stack_addr
0000000000003888 T _jl_realloc
000000000000d0f8 s _jl_realloc_addr

[Sukera]

@mattsignorelli

If I understand correctly, with this pointer I then unsafe_load into my RTPSA struct.

“into” is not really correct - unsafe_load(p, i) just “dereferences” the pointer. It’s the same as if you’d do p[i-1] in C, which is the same as doing *(p + sizeof(<type-p-points-to>)*(i-1)) . For i == 1, this is the same as *p. You get the object at that exact memory location, just in regular julia semantics.

@mattsignorelli

Now when the finalizer is eventually called, I pass it pointer_from_objref for that RTPSA. This pointer is different from the pointer originally returned by the C code during construction.

Different in what sense? They should be egal, i.e. have the same address. The one thing that might be different is provenance, but since this is FFI, that’s gone anyway.

@mattsignorelli

I cannot compile the C to a shared library with these undefined symbols jl_free and jl_malloc. So I’m not exactly sure what you mean by it will “just work”?

@mattsignorelli

Also I tried just including and linking with libjulia, but it doesn’t have a symbol for jl_free?

It’s in libjulia-internal, since it’s part of the runtime and not codegen. I don’t think it’s in the h-file, but you should be able to define it on your end with extern. At that point, the dynamic linker should find everything when you load the .so in julia, no need for manual linking.

[mattsignorelli]

@Sukera

“into” is not really correct - unsafe_load(p, i) just “dereferences” the pointer. It’s the same as if you’d do p[i-1] in C, which is the same as doing *(p + sizeof(<type-p-points-to>)*(i-1)) . For i == 1, this is the same as *p. You get the object at that exact memory location, just in regular julia semantics.

@Sukera

Different in what sense? They should be egal, i.e. have the same address. The one thing that might be different is provenance, but since this is FFI, that’s gone anyway.

The C code constructor spits out a pointer to the struct that the C allocated (not Julia). So given some Ptr{RTPSA}, how exactly do I construct a new Julia RTPSA object?

e.g. I modified the code to make RTPSA mutable. I can’t just return unsafe_load because now that allocates a new object:

julia> tpsa = GTPSA.mad_tpsa_newd(d.desc, GTPSA.MAD_TPSA_SAME) # this is C ctor
Ptr{GTPSA.RTPSA} @0x00006000004f4b08

julia> pointer_from_objref(unsafe_load(tpsa)) 
Ptr{Nothing} @0x000000010bdc67d0

julia> @allocations unsafe_load(tpsa)
1

julia> unsafe_pointer_to_objref(tpsa)

[70941] signal (11.2): Segmentation fault: 11

[Sukera]

@mattsignorelli

The C code constructor spits out a pointer to the struct that the C allocated (not Julia)

It doesn’t matter to unsafe_load who allocated the memory behind the pointer. It simply loads the value behind the pointer, i.e. it dereferences the pointer.

@mattsignorelli

So given some Ptr{RTPSA}, how exactly do I construct a new Julia RTPSA object?

You don’t construct anything; unsafe_load gives you the object at the address of that pointer. The object already exists right there. Does the object not exist at that address?

@mattsignorelli

e.g. I modified the code to make RTPSA mutable. I can’t just return unsafe_load because now that allocates a new object:

@mattsignorelli

julia> @allocations unsafe_load(tpsa)
1

I am pretty sure that this is an artifact of accessing the non-const global tpsa. The allocation should be from the resulting dynamic dispatch.

[Sukera]

@mattsignorelli

julia> tpsa = GTPSA.mad_tpsa_newd(d.desc, GTPSA.MAD_TPSA_SAME) # this is C ctor
Ptr{GTPSA.RTPSA} @0x00006000004f4b08

julia> pointer_from_objref(unsafe_load(tpsa)) 
Ptr{Nothing} @0x000000010bdc67d0

Do you have this version of your package available at some branch? I can’t access the documentation of the GTPSA C library, seems like the link at CERN is dead?

https://mad.web.cern.ch/mad/releases/madng/html/mad_mod_diffalg.html

[mattsignorelli]

@Sukera

I am pretty sure that this is an artifact of accessing the non-const global tpsa. The allocation should be from the resulting dynamic dispatch.

julia> using BenchmarkTools

julia> @ballocated unsafe_load($tpsa)
48

I guess I could have an alias for Ptr{RTPSA} and then dispatch on those (does the GC know that RTPSA is still in use if there are instances of Ptr{RTPSA}?)

@Sukera

Do you have this version of your package available at some branch? I can’t access the documentation of the GTPSA C library, seems like the link at CERN is dead?

I’ve created a branch and PR with mutable structs here where we can continue the discussion since it’s getting a bit off topic from the original point now: mutable rtpsa by mattsignorelli · Pull Request #121 · bmad-sim/GTPSA.jl · GitHub

[Sukera]

For those keeping track: unsafe_load indeed ends up creating a copy of that mutable struct, which is very unfortunate The approach I outlined above doesn’t work.

[mattsignorelli]

I’ve got the jl_malloc et al in the C code, but I’ve found that memset of that memory in C to 0 causes a segfault in Julia, any ideas?

[mattsignorelli]

For anyone following, the solution I went with was to instead define the constructor for these special C structs straight in Julia, instead of using those in C, and allocate in Julia via @ccall jl_malloc. This now doesn’t have Julia’s memory usage blow up.