# How to use ccall, cconvert, and unsafe\_convert in a convenient and memory-safe way?

**URL:** <https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932>\
**Category:** General Usage\
**Tags:** ccall, garbage-collection\
**Created:** [June 23, 2020, 12:34pm UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932 "2020-06-23T12:34:12Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [June 23, 2020, 12:34pm UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932/1 "2020-06-23T12:34:12Z")

</div>

I am writing Julia code to conveniently interface a C library, which in turn is just an interface to a C++ library. Therefore, many times I have to store pointers to `void` as the C interface does not know anything about the actual types either.

For this purpose, I have several structs like this

```julia
mutable struct CxxObjectWrapper
  handle::Ptr{Cvoid}
end

```

which have no other purpose than to hold a `Ptr` to the actual memory (which is managed through the C library), and to be able to use it conveniently with multiple dispatch.

My questions are regarding the proper use of `CxxObjectWrapper` when used in `ccall`. Many C functions look like

```nohighlight
long foo(void *cxx_object_wrapper);

```

to which I write the following Julia function:

```julia
function foo(obj::CxxObjectWrapper)
  ccall((:call_foo, libname), Clong, (Ptr{CVoid},), obj.handle)
end

```

Now my questions are:

1. Is this a problem in terms of garbage collection (specifically, the `obj.handle` part)? E.g., should I wrap the `ccall` in something like `GC.@preserve obj ccall(...)`?
2. What would be the safe and canonical way to do this conversion automatically? Should I define something like

```julia
cconvert(::Type{Ptr{Cvoid}}, x::CxxObjectWrapper) = x.handle

```

or do I need to use `unsafe_convert`? I am a little at a loss here, since the docstrings imply that `cconvert` should not return a `Ptr` but I am not sure if this applies in this case (or if it just means it should not return a `Ptr` to the object itself).

---

<div class="post-metadata">

**Author:** ![pixel27](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pixel27/32/8902_2.png) [@pixel27](https://discourse.julialang.org/u/pixel27)\
**Post date:** [June 23, 2020, 12:56pm UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932/2 "2020-06-23T12:56:03Z")

</div>

I used cconvert when doing this in my code:

```julia
cconvert(::Type{Ptr{Cvoid}}, x::CxxObjectWrapper) = x.handle

```

According to my read of the documentation Julia ensures that the object is not garbage collected during the C call if it gets the pointer via the cconvert call.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [June 23, 2020, 3:08pm UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932/3 "2020-06-23T15:08:50Z")

</div>

No that’s the job for unsafe\_convert.

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [June 23, 2020, 6:06pm UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932/4 "2020-06-23T18:06:06Z")

</div>

> [@yuyichao](#):
>
> No that’s the job for unsafe\_convert.

Does this answer refer to the suggestion of @pixel27 of using `cconvert` for my purpose, or is it an objection against the statement that

> [@pixel27](#):
>
> […] Julia ensures that the object is not garbage collected during the C call if it gets the pointer via the cconvert call.

?

If I combine the information of the docstrings of `cconvert` (which says _Neither `convert` nor `cconvert` should take a Julia object and turn it into a `Ptr.`_) and `unsafe_convert` (_Convert `x` to a C argument of type `T` where the input `x` must be the return value of `cconvert(T, ...)`._), does it mean I should define both

```julia
cconvert(::Type{Ptr{Cvoid}}, x::CxxObjectWrapper) = x

```

and

```julia
unsafe_convert(::Type{Ptr{Cvoid}}, x::CxxObjectWrapper) = x.handle

```

? To me, that seems like overkill…

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [June 23, 2020, 6:42pm UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932/5 "2020-06-23T18:42:03Z")

</div>

That was a directly reply to @pixel27’s comment but discourse automatically removed reply reference to the previous comment again… Sign…

OTOH,

> [@pixel27](#):
>
> According to my read of the documentation Julia ensures that the object is not garbage collected during the C call if it gets the pointer via the cconvert call.

is indeed wrong. The guarantee is that the returned object from `cconvert` will be valid, so you can derive pointers from them.

> [@sloede](#):
>
> ```julia
> cconvert(::Type{Ptr{Cvoid}}, x::CxxObjectWrapper) = x
> 
> ```

I think this is the default.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [June 23, 2020, 6:46pm UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932/6 "2020-06-23T18:46:53Z")

</div>

Also, if it’s used many times in the code, then defining two more functions instead of one shouldn’t be too much harder. If it’s just a one off call, you can also use `GC.@preserve`…

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [June 23, 2020, 6:50pm UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932/7 "2020-06-23T18:50:17Z")

</div>

You are [right about the default](https://github.com/JuliaLang/julia/blob/44fa15b1502a45eac76c9017af94332d4557b251/base/essentials.jl#L391):

```julia
cconvert(::Type{<:Ptr}, x) = x

```

Thus I will just define the appropriate `unsafe_convert` then. Thanks for the clarification!

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [June 23, 2020, 6:51pm UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932/8 "2020-06-23T18:51:00Z")

</div>

Ah, so in my example above, I _have_ to use `GC.@preserve` or else I am not memory safe?

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [June 23, 2020, 6:53pm UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932/9 "2020-06-23T18:53:16Z")

</div>

No, I’m saying if this is the only place you need the `unsafe_covert`, you can replace that with `GC.@preserve` at the `ccall`site instead. Both are valid options and they are roughly equivalent here…

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [June 23, 2020, 6:57pm UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932/10 "2020-06-23T18:57:12Z")

</div>

Sorry, I was not very clear: In my example above, if I don’t do anything, ie, neither `unsafe_convert` nor `GC.@preserve`, then I can get into trouble? Or put differently, the example above is broken if left as-is?

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [June 23, 2020, 6:58pm UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932/11 "2020-06-23T18:58:27Z")

</div>

> [@sloede](#):
>
> if I don’t do anything, ie, neither `unsafe_convert` nor `GC.@preserve` , then I can get into trouble?

If you are talking about

> [@sloede](#):
>
> ```julia
> function foo(obj::CxxObjectWrapper)
> ccall((:call_foo, libname), Clong, (Ptr{CVoid},), obj.handle)
> end
> 
> ```

then yes. (note that even if you define `unsafe_convert` it won’t fix this automatically. You’ll need to replace the `obj.handle` with `obj` of course…)

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [June 23, 2020, 7:00pm UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932/12 "2020-06-23T19:00:33Z")

</div>

And also to clarify, it’s only a problem if you have finalizers to manage the `handle` based on the lifetime of the parent struct (You didn’t say this explicitly but given it’s mutable I assume that’s the case). If you aree doing full manual management of the C memory then you don’t need anything like this. The pointer **value** will never be wrong. It’s only freeing of the underlying memory by julia (through finalizer) that’s of concern here.

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [June 23, 2020, 8:42pm UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932/13 "2020-06-23T20:42:19Z")

</div>

You are right about the handling through `finalizer`. The actual definition of `CxxObjectWrapper` looks something like this:

```julia
mutable struct CxxObjectWrapper
  handle::Ptr{Cvoid}

  function CxxObjectWrapper()
    # Create C++ instance through C library and retrieve pointer
    handleref = Ref{Ptr{Cvoid}}(C_NULL)
    ccall((:CxxObject_new, libname), Clong, (Ref{Ptr{Cvoid}},), handleref)

    # Store pointer to C++ object in new Julia CxxObjectWrapper instance
    x = new(handleref)
    finalizer(x) do x
      ccall((:CxxObject_delete, libname), Clong, (Ptr{Cvoid},), x.handle)
    end

    return x
  end
end

```

However, even though I feel like I read almost everything on the topic of GC and C memory management either here, on SO, and in the docs and docstrings, I still can’t seem to wrap my head around the fact why I would need something like `unsafe_convert` to avoid memory issues in functions like this:

```julia
function foo(obj::CxxObjectWrapper)
  ccall((:call_foo, libname), Clong, (Ptr{CVoid},), obj.handle)
end

```

As far as I can tell, there is no way that instance `obj` is not referenced anymore, since `obj` is just another name for the same instance that was passed in at the call site, e.g., `bar` in case it is called as `foo(bar)`.

The only possible issue I could see is when `foo()` is called with a temporary instance of `CxxObjectWrapper`, e.g., `foo(CxxObjectWrapper())`. But in this case, I still don’t see a need for a “special” GC-aware function: A simple

```julia
handle(x::CxxObjectWrapper) = x.handle

```

should do the trick as well, since in this case,

```julia
function foo2(obj::CxxObjectWrapper)
  ccall((:call_foo, libname), Clong, (Ptr{CVoid},), handle(obj))
end

```

`obj` is always referenced somewhere. Or am I completely off the track here? Sorry for asking so persistently, but from your many other contributions on Discourse I feel like you are exactly the right person to ask these questions and get a reliable and well-founded answer 😬

---

<div class="post-metadata">

**Author:** ![abulak](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/abulak/32/28314_2.png) [@abulak](https://discourse.julialang.org/u/abulak)\
**Post date:** [June 23, 2020, 10:47pm UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932/14 "2020-06-23T22:47:54Z")

</div>

> [@sloede](#):
>
> As far as I can tell, there is no way that instance `obj` is not referenced anymore, since `obj` is just another name for the same instance that was passed in at the call site, e.g., `bar` in case it is called as `foo(bar)` .

`obj` may be GCed before `ccall` and `obj.handle` (julia will rewrite `handle(obj)` to it in this case) will be passed by value; the compiler has license to reorder and even elide creation of objects. We recently had a bug like this and it was very hard to track 😉

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [June 24, 2020, 12:20am UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932/15 "2020-06-24T00:20:19Z")

</div>

> [@sloede](#):
>
> The only possible issue I could see is when `foo()` is called with a temporary instance of `CxxObjectWrapper` , e.g., `foo(CxxObjectWrapper())` .

No that’s not the case.

> [@sloede](#):
>
> ```julia
> handle(x::CxxObjectWrapper) = x.handle
> 
> ```

Well I don’t see how this can help even if `foo` get passed a copy. It’ll at most make things worse since your `x` could be a copy too.

> [@sloede](#):
>
> `obj` is always referenced somewhere.

Now what does **that** mean. If what you are saying is that the `obj` is always either

1. referenced by a global variable
2. appears in a `GC.@preserve` argument
3. return value of `cconvert` **during** a ccall (i.e. when the control is in the C code and have not returned yet).

Then yes, you are right. You would not need to do anything to make the use of `obj.handle` valid. Otherwise, not, there’s basically no other well defined reference of `obj`. Just because you have a local variaible has absolutely nothing to do with the lifetime of the object.

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [June 24, 2020, 4:46am UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932/16 "2020-06-24T04:46:57Z")

</div>

> [@yuyichao](#):
>
> > [@sloede](#):
> >
> > ````julia
> > > handle(x::CxxObjectWrapper) = x.handle
> > > ```
> > 
> > ````
> 
> Well I don’t see how this can help even if `foo` get passed a copy. It’ll at most make things worse since your `x` could be a copy too.

Ah, OK, now I am getting closer. What you are essentially saying is that the compiler may decide to create a copy of `bar`/`x` to pass it into `foo`/`handle`? Or where could this copy come from? I always thought (but don’t ask me where I read it) that in Julia function arguments are always passed by sharing/reference, unless the argument is `isbits`, in which case a copy can be made. However, any struct with a `finalizer` can never be `isbits`, since it has to be `mutable` to have a `finalizer`. Or where am I missing something?

> [@yuyichao](#):
>
> 1. referenced by a global variable
> 2. appears in a `GC.@preserve` argument
> 3. return value of `cconvert` **during** a ccall (i.e. when the control is in the C code and have not returned yet).
> 
> […]  
> Just because you have a local variaible has absolutely nothing to do with the lifetime of the object.

I think these two statements, in their simplicity and clarity, are absolutely vital to keep in mind when working with `ccall`, and should be in the official documentation! Even though the topic is briefly touched [here](https://docs.julialang.org/en/v1/manual/calling-c-and-fortran-code/#Garbage-Collection-Safety-1), I think your answer plus a counter example (e.g., “do not use `ccall(..., a.b)` if `b`’s validity depends on `a`’s validity, instead use `ccall(..., a)` and an appropriately defined `cconvert`/`unsafe_convert`)”) would help the unsuspecting user (like me) who has just enough knowledge to be in danger of shooting themselves in the foot 😉 Do you think it would be advisable to create a PR to the manual for this?

To sum it up, if I get everything correctly, with my definition of

```julia
unsafe_convert(::Type{Ptr{Cvoid}}, x::CxxObjectWrapper) = x.handle

```

I am able to pass `obj` directly to `ccall`, which in turn will call automatically call `cconvert` (as per the [docs](https://github.com/JuliaLang/julia/blob/44fa15b1502a45eac76c9017af94332d4557b251/base/docs/basedocs.jl#L862-L865)), which in the absence of other definitions will select the method in [`Base`](https://github.com/JuliaLang/julia/blob/44fa15b1502a45eac76c9017af94332d4557b251/base/essentials.jl#L391),

```julia
cconvert(::Type{<:Ptr}, x) = x

```

and thus `obj` is always properly referenced (as per your third condition above). Thank you very much for these answers and your patience, this helps a lot!

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [June 24, 2020, 10:14am UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932/17 "2020-06-24T10:14:00Z")

</div>

> [@sloede](#):
>
> What you are essentially saying is that the compiler may decide to create a copy of `bar` / `x` to pass it into `foo` / `handle` ?

No! I was following your logic that `obj` might have been a logic. No such copying is happening anywhere.

I’m asking because if you think adding such a function would help, there’s a major misconception somewhere so I want to know why you think it helps.

> [@sloede](#):
>
> unless the argument is `isbits` , in which case a copy can be made

No. That’s not true either. It’s not true semantically, a reference is always passed, and it’s not true performance-wise, a pointer is passed for large structures.

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [June 24, 2020, 10:25am UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932/18 "2020-06-24T10:25:15Z")

</div>

> [@yuyichao](#):
>
> > [@sloede](#):
> >
> > What you are essentially saying is that the compiler may decide to create a copy of `bar` / `x` to pass it into `foo` / `handle` ?
> 
> No! I was following your logic that `obj` might have been a logic. No such copying is happening anywhere.

Thanks for the clarification. I thought `handle(x) = x.handle` could prevent the GC from cleaning up `x`, since it still “sees” it as an argument to `handle(x)`. However, I clearly still have a considerable way to go before I fully understand when and where the GC can (and will act), but for now I am happy enough with my half-knowledge plus your valuable input.

> [@yuyichao](#):
>
> > [@sloede](#):
> >
> > unless the argument is `isbits` , in which case a copy can be made
> 
> No. That’s not true either. It’s not true semantically, a reference is always passed, and it’s not true performance-wise, a pointer is passed for large structures.

That’s good to know for at least two reasons: One, it is interesting from a technical point of view, and two, it tells me that I still have a very naive concept of what goes on behind the scenes in Julia (apparently much, much more than I thought). Coming from the C++ world (where it was _usually_ very easy to reason what goes on by doing a static analysis of the code), I still have a lot of catching up to do with respect to Julia’s internals. For now, however, I think I am happy enough with the information I gathered from our discussion here, but I might revisit this in the future when I encounter similar issues again.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [June 24, 2020, 11:50am UTC](https://discourse.julialang.org/t/how-to-use-ccall-cconvert-and-unsafe-convert-in-a-convenient-and-memory-safe-way/41932/19 "2020-06-24T11:50:08Z")

</div>

> [@sloede](#):
>
> Coming from the C++ world (where it was _usually_ very easy to reason what goes on by doing a static analysis of the code)

Note that value passing in C++ can be passing a pointer as well. The compiler may not have enough information to elide the copy though.

Also that is what I was suspecting. In julia, what you write in one expression has absolutely nothing to do with anything. `ptr = handle(obj); ccall(....., ptr)` is absolutely equivalent to `ccall(...., handle(obj))`. Intermediate result in a expression get no special treatment in any regard.