# How to keep a reference for C structure to avoid GC?
**URL:** https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310
**Category:** General Usage
**Created:** [February 25, 2018, 7:10am UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310 "2018-02-25T07:10:06Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@tk3369](https://discourse.julialang.org/u/tk3369)
#### Post date: [February 25, 2018, 7:10am UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/1 "2018-02-25T07:10:06Z")
I need to pass a struct that may contain an array of its own type to a C library. To achieve that, I use the `pointer` function to obtain an address and encode that in an `UInt64` field (as expected by the C library).
I’ve seen random crashes or corrupted data, so I’m guessing that I must be doing something wrong here. I wonder if it’s because the array `ar` got GC’ed even though the object is still around. If that’s the case, what would be the correct way to maintain the reference or avoid GC?
Here’s the MWE to illustrate the situation:
```julia
julia> struct Bar
val::UInt64
ty::UInt8
Bar(v::Float64) = new(reinterpret(UInt64, v), 0x00)
Bar(vv::Vector{Float64}) = begin
ar = Bar.(vv)
new(reinterpret(UInt64, pointer(ar)), 0x01)
end
end
```
So the value of `val` could either be a `Float64` value or the address of an object of type `Vector{Bar}`. Here’s how to construct such objects:
```julia
julia> b1 = Bar(1.0)
Bar(0x3ff0000000000000, 0x00)
julia> b2 = Bar([2.0, 3.0])
Bar(0x0000000123aea430, 0x01)
```
To get the array back, I could do this:
```julia
julia> p2 = reinterpret(Ptr{Bar}, b2.val)
Ptr{Bar} @0x000000012b38b700
julia> q2 = unsafe_load.(p2, 1:2)
2-element Array{Bar,1}:
Bar(0x4000000000000000, 0x00)
Bar(0x4008000000000000, 0x00)
julia> [reinterpret(Float64, x.val) for x in q2]
2-element Array{Float64,1}:
2.0
3.0
```
---
### Author:  [@barche](https://discourse.julialang.org/u/barche)
#### Post date: [February 25, 2018, 1:27pm UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/2 "2018-02-25T13:27:39Z")
You will need to keep a reference to `ar` somewhere, e.g. in a global Dict, maybe using the pointer as a key. That way, you can remove the arrays from the dict when you free the C struct by removing the element corresponding to the pointer.
---
### Author:  [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [February 25, 2018, 1:31pm UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/3 "2018-02-25T13:31:39Z")
Don’t use a global dict, that’s not generic and is the slowest possible solution.
What you can use depends on how you are calling the C code. A possible solution that should work in most cases is [https://github.com/JuliaPy/PyCall.jl/blob/314ac274326e78f12fbcb73ae0f17f63a3f4bba9/src/pytype.jl#L314](https://github.com/JuliaPy/PyCall.jl/blob/314ac274326e78f12fbcb73ae0f17f63a3f4bba9/src/pytype.jl#L314)
---
### Author:  [@tk3369](https://discourse.julialang.org/u/tk3369)
#### Post date: [February 25, 2018, 6:17pm UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/4 "2018-02-25T18:17:43Z")
So here are my use cases:
1. Create the object and pass it into the C function. After the call is finished, the object is usually discarded.
2. The C function may return the struct. In which case, an `unpack` function is created to extract the data into Julia objects. It would do the same thing as listed above - a one dimensional array of native Julia values is returned (`[2.0, 3.0]`)
So the risk lies with #1 - GC may kick in and overwrite what `ar` was pointing to with some garbage and the C call could fail.
#2 isn’t really a problem because the memory is allocated by the C library not by Julia, hence it’s not going to be picked up by GC.
I’m not sure how to maintain the pointer though. I went through the PyCall code but don’t really understand what it does. I understand a `Dict` isn’t performant but if that’s just a data structure thing then I will adjust according to my usage pattern and use a different data structure as necessary. Also, if maintain this pointer then I would have to register a `finalizer` function when the parent object is freed. Is that right?
Another idea is to create another wrapper struct that contains the julia data the underlying C struct. More concretely:
```julia
struct Foo
ar::Vector{Bar}
obj::Bar
end
```
and then the binding function would look like:
```julia
function callme(x)
ccall((:func, LIB), Ptr{Bar}, (Ptr{Bar},), x)
end
```
and I would have to deference to the object before calling the binding function:
```julia
callme(Ref(foo.obj))
```
---
### Author:  [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [February 25, 2018, 6:43pm UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/5 "2018-02-25T18:43:00Z")
> [@tk3369](#):
>
> if that’s just a data structure thing then I will adjust according to my usage pattern and use a different data structure as necessary.
> [@tk3369](#):
>
> Another idea is to create another wrapper struct that contains the julia data the underlying C struct. More concretely:
Yes, this is the pycall approach. If this is an struct that you need to operate on directly, you can make a struct with an extra field that holds the reference to the julia object (i.e. give it three field with the third one being an Any).
> [@tk3369](#):
>
> and I would have to deference to the object before calling the binding function:
No this is wrong. This will not keep the julia array live. Julia is free to disgard the `ar` field.
It seems that you only need to use this struct when passing stuff to C so in that case, you can simply create that struct at `ccall` callsite within cconvert/unsafe\_convert. You can put all allocations into `cconvert(::Type{Ptr{Bar}}, ::Vector{Float64})`, and `cconvert(::Type{Ptr{Bar}}, ::Float64)` including allocation of a `Ref{Bar}` and return the `Ref{Bar}` in `unsafe_convert`. The `ccall` lowering will guarantee that the unsafe operatioins in `unsafe_convert` are safe assuming all the objects that needs to be alive are in the return value of `cconvert`.
```julia
Base.cconvert(::Type{Ptr{Bar}}, v::Float64) = Ref(Bar(v)) # This use the `::Float64` constructor
function Base.cconvert(::Type{Ptr{Bar}}, vv::Vector{Float64}) # This replaces the `::Vector{Float64}` constructor
ar = Bar.(vv)
bar = Bar(reinterpret(UInt64, pointer(ar)), 0x01)
return (Ref(bar), ar)
end
function Base.unsafe_convert(::Type{Ptr{Bar}}, t::Tuple{Ref{Bar},Vector{Bar}})
return pointer_from_objref(t[1])
end
```
---
### Author:  [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [February 25, 2018, 6:55pm UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/6 "2018-02-25T18:55:34Z")
And as a rule of thumb,
1. If you are not using any julia object from their pointer values, you should never need to worry about object lifetime.
2. If the invisible object lifetime is bound to something visible in julia, which can be a function local scope or a julia object (the latter being the usecase in PyCall), you should not use any external state to keep the object live and should use the related julia object/scope to do so. This can mean using `@gc_preserve` or writing the correct `unsafe_convert`/`cconvert` since these are the only two local mechanism to keep object alive.
3. Only if the object is kept alive in a way that’s totally invisible to any local julia state (like if you store the julia address to a C global object) it is then needed to use an external/global state to manage/extend the lifetime of the julia object.
---
### Author:  [@tk3369](https://discourse.julialang.org/u/tk3369)
#### Post date: [February 25, 2018, 7:15pm UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/7 "2018-02-25T19:15:23Z")
> [@yuyichao](#):
>
> No this is wrong. This will not keep the julia array live. Julia is free to disgard the ar field.
Mmm… I don’t understand how `ar` could be freed. The `Foo` object is holding a reference to `ar` until the `ccall` is finished. What am I missing?
Just so we’re on the same page, let’s say the struct now looks like this:
```julia
julia> struct Bar
val::UInt64
ty::UInt8
Bar(v::Float64) = new(reinterpret(UInt64, v), 0x00)
Bar(v::Vector{Bar}) = new(reinterpret(UInt64, pointer(v)), 0x01) # v could be gc'ed unless it's referenced elsewhere
end
julia> struct Foo
ar::Vector{Bar}
obj::Bar
end
```
Then, the `ccall` would like:
```julia
# binding function created by Clang.jl
function callme(x)
ccall((:func, LIB), Ptr{Bar}, (Ptr{Bar},), x)
end
# a useful business function...
function reachthesky()
ar = Bar.([2.0, 3.0])
obj = Bar(ar)
foo = Foo(ar, obj)
unpack(callme(Ref(foo.obj)))
end
```
---
### Author:  [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [February 25, 2018, 7:27pm UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/8 "2018-02-25T19:27:12Z")
> [@tk3369](#):
>
> The Foo object is holding a reference to ar until the ccall is finished. What am I missing?
No, the `Foo` object can be free’d.
---
### Author:  [@tk3369](https://discourse.julialang.org/u/tk3369)
#### Post date: [February 25, 2018, 8:41pm UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/9 "2018-02-25T20:41:56Z")
Wow… is it because the compiler sees that I don’t use the `foo` object anymore? Would it be “saved” if I continue using `foo` after the call?
```julia
function reachthesky()
ar = Bar.([2.0, 3.0])
obj = Bar(ar)
foo = Foo(ar, obj)
unpack(callme(Ref(foo.obj)))
println(foo)
end
```
Is `@gc_preserve` a 0.7 thing? Maybe that would be useful in this case?
```julia
function reachthesky()
ar = Bar.([2.0, 3.0])
obj = Bar(ar)
@gc_preserve ar unpack(callme(Ref(obj)))
end
```
---
### Author:  [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)
#### Post date: [February 25, 2018, 8:48pm UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/10 "2018-02-25T20:48:19Z")
> [@tk3369](#):
>
> Would it be “saved” if I continue using foo after the call?
The compiler could reorder stuff and replace your aggregate with a bunch of scalars such that `Foo` doesn’t even get constructed anymore etc etc. Perhaps it doesn’t do that now but in general, using the official way ( `@gc_preserve`) is a good idea (yes 0.7).
---
### Author:  [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [February 25, 2018, 8:52pm UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/11 "2018-02-25T20:52:35Z")
> [@tk3369](#):
>
> Would it be “saved” if I continue using foo after the call?
No. As mentioned above, the only two ways you can guarantee something is live locally are `@gc_preserve` and `ccall`.
> [@tk3369](#):
>
> Maybe that would be useful in this case?
If you only have one or two of these calls yes that’s fine. If you are calling this from different places using the `unsafe_convert` and `cconvert` would be better.
---
### Author:  [@tk3369](https://discourse.julialang.org/u/tk3369)
#### Post date: [February 25, 2018, 9:45pm UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/12 "2018-02-25T21:45:20Z")
Thanks. I’ve learned a lot from this.
---
### Author:  [@tk3369](https://discourse.julialang.org/u/tk3369)
#### Post date: [February 25, 2018, 11:08pm UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/13 "2018-02-25T23:08:37Z")
Sharing my [sample notebook that illustrates the use of cconvert/unsafe\_convert](https://github.com/tk3369/julia-notebooks/blob/master/ccall%20-%20using%20cconvert%20and%20unsafe_convert.ipynb) for those who are interested.
---
### Author:  [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [February 25, 2018, 11:26pm UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/14 "2018-02-25T23:26:03Z")
Yes, that’s exactly right!
---
### Author:  [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [February 25, 2018, 11:27pm UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/15 "2018-02-25T23:27:42Z")
And just as a minor note that you should probably free the returned pointer after you’ve loaded everything you want from it…
---
### Author:  [@jameson](https://discourse.julialang.org/u/jameson)
#### Post date: [February 26, 2018, 1:21am UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/16 "2018-02-26T01:21:57Z")
```julia
return convert(Ptr{Bar2}, pointer_from_objref(t[1]))
```
should be
```julia
return unsafe_convert(Ptr{Bar2}, t)
```
for more generality / more correct usage of the Ref interface
---
### Author:  [@tk3369](https://discourse.julialang.org/u/tk3369)
#### Post date: [February 26, 2018, 3:08am UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/17 "2018-02-26T03:08:46Z")
Thanks, updated notebook. Made a small change since we need the first element of the tuple.
```julia
return unsafe_convert(Ptr{Bar2}, t[1])
```
---
### Author:  [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)
#### Post date: [February 26, 2018, 10:58am UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/18 "2018-02-26T10:58:55Z")
Very nice that you took the time to make a good example out of this.
---
### Author:  [@pedrohnv](https://discourse.julialang.org/u/pedrohnv)
#### Post date: [April 23, 2018, 6:13pm UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/19 "2018-04-23T18:13:42Z")
I could not make sense of this discussion nor the [sample notebook](https://github.com/tk3369/julia-notebooks/blob/master/ccall%20-%20using%20cconvert%20and%20unsafe_convert.ipynb) to adapt to my specific use case.
Any help would be appreciated:
I have a Struct with a fixed size array field. I want to use Julia to build that struct and then call the C code which will do the heavy computation accessing the elements in the struct. The Julia struct does not need to stay in memory after the ccall.
Here is an example:
```julia
//C code
typedef struct {
double array[3];
} Mystruct;
int access_struct_array(Mystruct* mystruct)
{
for (int i = 0; i < 3; i++)
{
printf("%f\n", mystruct->array[i]);
}
return 0;
}
```
Then I use Julia to build the struct and call the C code
```julia
type Mystruct
array::Array{Float64, 1}
end
mystruct = Mystruct([1.1, 2.1, 3.1]);
ccall(("access_struct", "./shared.so"), Int, (Ref{Mystruct},), mystruct); #undefined behavior
```
Why is the behavior of `ccall` undefined?
I would say it is related to the fact that [Julia arrays are not just pointers](https://discourse.julialang.org/t/pointer-julia-not-modified-after-ccall-to-c-function-which-uses-another-c-function/10455/13). Then, what is the Julia equivalent of a C array?
---
### Author:  [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [April 23, 2018, 6:51pm UTC](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/20 "2018-04-23T18:51:49Z")
You do not need to use this thread at all if all what you have is a C array instead of a C pointer. C arrays members are just, well, arrays (i.e. consecutive objects) so anything with a similar memory layout should work. You just need to construct such an object and pass it as reference. `struct ::Float64; ::Float64; ::Float64 end`, `NTuple{3,Float64}` or even just the array direction should work.
[Next page](https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310.md?page=2)