# Properly using \`finalizer\`, \`ccall\`, \`cconvert\` and \`unsafe\_convert\`

**URL:** <https://discourse.julialang.org/t/properly-using-finalizer-ccall-cconvert-and-unsafe-convert/6183>\
**Category:** General Usage\
**Tags:** question, ccall\
**Created:** [October 1, 2017, 6:08pm UTC](https://discourse.julialang.org/t/properly-using-finalizer-ccall-cconvert-and-unsafe-convert/6183 "2017-10-01T18:08:34Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon61610682](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@anon61610682](https://discourse.julialang.org/u/anon61610682)\
**Post date:** [October 1, 2017, 6:08pm UTC](https://discourse.julialang.org/t/properly-using-finalizer-ccall-cconvert-and-unsafe-convert/6183/1 "2017-10-01T18:08:34Z")

</div>

I have problems when using `finalizer`s with my Julia `v0.6`. I am using Atom when writing/running the code if this makes any difference.

As for the minimal working example, consider having the following C API:

```nohighlight
extern "C" {
double d_somefunc(const double, void **);
double f_somefunc(const float, void **);
void delhandle(void *);
}

```

Above, I use `void **` to change the pointer value, _i.e._, the second arguments to `x_somefunc` are in/out arguments. Moreover, in this small example, `void *` is simply an opaque pointer in the `C++` library that is exposed to users for the C API.

After compiling the library as a shared object, I run the following Julia code to test the functioning of the library:

```julia
libhandle = Libdl.dlopen(joinpath(pwd(), "libexternal.so"))
funhandle = Libdl.dlsym(libhandle, :d_somefunc)
delhandle = Libdl.dlsym(libhandle, :delhandle)

mutable struct MyType{T<:Real}
  val::T
  handle::Ref{Ptr{Void}}
end

handle(m::MyType) = m.handle[]
value(m::MyType) = m.val
value!(m::MyType, val::Real) = (m.val = val)

function (::Type{MyType})(val::Real)
  obj = MyType{typeof(val)}(val, Ref{Ptr{Void}}(0))
  finalizer(obj, destruct!)
  return obj
end

function destruct!(m::MyType)
  println("Destructor called")
  ccall(delhandle, Void, (Ptr{Void},), m.handle[])
end

function (m::MyType)()
  return ccall(funhandle, Cdouble, (Cdouble, Ref{Ptr{Void}}), m.val, m.handle)
end

# Run below for 4 times
# for k in 1:4
m = MyType(1)
m()
m()
value!(m, 7)
# end

gc()

```

Behind the scenes, `x_somefunc` functions check

1. if `handle` (the second argument) points to an already allocated (by the C++ library) object of some type `MyClass`,
2. if not, they create a new object on the heap and change `handle`’s value to point to the corresponding object’s address, and,
3. they call the object’s `operator()` on the `val`.

In other words, this is simply a stateful function call in the C++ library, which is hidden behind the opaque pointer `void *`.

When I run the above commands until I get the garbage collector in Julia to do its job, I receive the following output (upon stopping Julia):

```julia
Destructor called
Destructor called
Destructor called
error in running finalizer: ErrorException("task switch not allowed from inside gc finalizer")
error in running finalizer: ErrorException("task switch not allowed from inside gc finalizer")
error in running finalizer: ErrorException("task switch not allowed from inside gc finalizer")
error in running finalizer: Base.ArgumentError(msg="stream is closed or unusable")
myptrs.size(): 1
myptrs.size(): 1
myptrs.size(): 2
myptrs.size(): 2
myptrs.size(): 3
myptrs.size(): 3
myptrs.size(): 4
myptrs.size(): 4
dtor called.
dtor called.
dtor called.
dtor called.

```

Apparently, the C++ library is successfully cleaning its mess upon exit (`dlclose`). But I would like to know why I am getting the errors above in Julia.

Thank you for your time and help 🙂

**EDIT.** For those who would like to have a complete set of files to reproduce the error, below are the header and implementation files:

```nohighlight
/* external.hpp file */
#ifndef EXTERNAL_HPP_
#define EXTERNAL_HPP_

template <class T> class MyClass {
private:
  T val_;

public:
  MyClass();
  MyClass(const T &);
  ~MyClass();
  T operator()(const T &);
};

// Tell compiler that MyClass has been instantiated somewhere for float and
// double
extern template class MyClass<double>;
extern template class MyClass<float>;

// Tell compiler that somefunc has been instantiated somewhere for float and
// double
template <class T> T somefunc(const T &, void *&);
extern template double somefunc(const double &, void *&);
extern template float somefunc(const float &, void *&);

// Tell compiler that deletehandle is available somewhere
extern void deletehandle(void *);

// Tell compiler that below function names are available in C-symbols
extern "C" {
double d_somefunc(const double, void **);
double f_somefunc(const float, void **);
void delhandle(void *);
}

#endif

```

```nohighlight
/* external.cpp file */
#include "external.hpp"

#include <iostream>
#include <map>
#include <memory>

template <class T> MyClass<T>::MyClass() = default;
template <class T> MyClass<T>::MyClass(const T &val) : val_{val} {}
template <class T> MyClass<T>::~MyClass() { std::cout << "dtor called.\n"; }
template <class T> T MyClass<T>::operator()(const T &val) {
  T res{val_ * val};
  val_ = val;
  return res;
}

// Tell compiler to instantiate MyClass here for float and double
template class MyClass<double>;
template class MyClass<float>;

template <class T> void myclassdeleter(void *ptr) {
  if (ptr != nullptr)
    delete static_cast<T *>(ptr);
}

std::map<void *, std::unique_ptr<void, void (*)(void *)>> myptrs;

template <class T> T somefunc(const T &input, void *&handle) {
  auto search = myptrs.find(handle);
  if (search == myptrs.end()) {
    std::unique_ptr<void, void (*)(void *)> up{new MyClass<T>{T{0}},
                                               &myclassdeleter<MyClass<T>>};
    handle = up.get();
    myptrs.emplace(handle, std::move(up));
  }
  std::cout << "myptrs.size(): " << myptrs.size() << '\n';
  auto objptr = static_cast<MyClass<T> *>(handle);
  auto res = objptr->operator()(input);
  return res;
}

// Tell compiler to instantiate somefunc here for float and double
template double somefunc(const double &, void *&);
template float somefunc(const float &, void *&);

void deletehandle(void *handle) {
  myptrs.erase(handle);
  std::cout << "myptrs.size(): " << myptrs.size() << '\n';
}

// Export below functions in C-symbols
extern "C" {
double d_somefunc(const double input, void **handle) {
  return somefunc(input, *handle);
}
double f_somefunc(const float input, void **handle) {
  return somefunc(input, *handle);
}
void delhandle(void *handle) { return deletehandle(handle); }
}

```

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [October 1, 2017, 6:17pm UTC](https://discourse.julialang.org/t/properly-using-finalizer-ccall-cconvert-and-unsafe-convert/6183/2 "2017-10-01T18:17:18Z")

</div>

> [@aytekinar](#):
>
> handle::Ref{Ptr{Void}}

I see no reason to use a `Ref` here.

> [@aytekinar](#):
>
> println(“Destructor called”)

This is not allowed

---

<div class="post-metadata">

**Author:** ![anon61610682](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@anon61610682](https://discourse.julialang.org/u/anon61610682)\
**Post date:** [October 1, 2017, 6:20pm UTC](https://discourse.julialang.org/t/properly-using-finalizer-ccall-cconvert-and-unsafe-convert/6183/3 "2017-10-01T18:20:21Z")

</div>

Hey @yuyichao. Thank you for your prompt answer.

> [@yuyichao](#):
>
> I see no reason to use a Ref here.

I think I need to use some sort of `Ref` there, please check the external.cpp file above in the edited version.

> [@yuyichao](#):
>
> This is not allowed

So, you think the `println` statement was the problem, then? I will try without them. What are the allowed operations, then?

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [October 1, 2017, 6:23pm UTC](https://discourse.julialang.org/t/properly-using-finalizer-ccall-cconvert-and-unsafe-convert/6183/4 "2017-10-01T18:23:56Z")

</div>

> [@aytekinar](#):
>
> I think I need to use some sort of Ref there, please check the external.cpp file above in the edited version.

No, you only need to mutate the pointer, which doesn’t need a `Ref`. It’s also an abstract type FWIW.

> [@aytekinar](#):
>
> What are the allowed operations, then?

Anything that does not violate `task switch not allowed from inside gc finalizer`.

---

<div class="post-metadata">

**Author:** ![anon61610682](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@anon61610682](https://discourse.julialang.org/u/anon61610682)\
**Post date:** [October 1, 2017, 6:28pm UTC](https://discourse.julialang.org/t/properly-using-finalizer-ccall-cconvert-and-unsafe-convert/6183/5 "2017-10-01T18:28:48Z")

</div>

> [@yuyichao](#):
>
> No, you only need to mutate the pointer, which doesn’t need a Ref. It’s also an abstract type FWIW.

Could you provide a small example/hint on how to achieve this? I mean, should I need a change in the C API or in Julia? I have tried before `Ptr{Ptr{Void}}` and my Julia crashed. Which one is also an abstract type? I am sorry, I cannot follow since these are not covered in docs, or simply I could not understand from what I read there.

> [@yuyichao](#):
>
> Anything that does not violate task switch not allowed from inside gc finalizer.

Again, I could not find any list of functions/operations that result in task switching.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [October 1, 2017, 7:29pm UTC](https://discourse.julialang.org/t/properly-using-finalizer-ccall-cconvert-and-unsafe-convert/6183/6 "2017-10-01T19:29:35Z")

</div>

> [@aytekinar](#):
>
> Could you provide a small example/hint on how to achieve this?

Just use a `::Ptr{Void}` as the field type. You should just figure out how you would write such an interface code in C and just map that to julia.  
The use of `ccall` here is also invalid since julia is allowed to free `m` before any of the ccall. So you have to define `unsafe_convert` and `cconvert` methods that does the correct convertion with defined GC behavior. You can do any pointer arithmetic you want in the `unsafe_convert` after taking the address of the object with `pointer_to_objref`. See the doc for `cconvert` and `unsafe_convert` which are mentioned [here](https://docs.julialang.org/en/latest/manual/calling-c-and-fortran-code/#Auto-conversion:-1)

> [@aytekinar](#):
>
> Which one is also an abstract type?

`Ref{Ptr{Void}}` Ref [https://docs.julialang.org/en/latest/manual/types/#Abstract-Types-1](https://docs.julialang.org/en/latest/manual/types/#Abstract-Types-1). In generl, your code seems to be having a lot of type instability, which isn’t necessarily an issue if you don’t care about performance.

> [@aytekinar](#):
>
> Again, I could not find any list of functions/operations that result in task switching.

There isn’t such a list, in general though any [IO function](https://docs.julialang.org/en/latest/manual/networking-and-streams/#Networking-and-Streams-1) will have such an effect.

---

<div class="post-metadata">

**Author:** ![anon61610682](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@anon61610682](https://discourse.julialang.org/u/anon61610682)\
**Post date:** [October 1, 2017, 8:35pm UTC](https://discourse.julialang.org/t/properly-using-finalizer-ccall-cconvert-and-unsafe-convert/6183/7 "2017-10-01T20:35:51Z")

</div>

Thank you, again, for your answer.

> [@yuyichao](#):
>
> Just use a ::Ptr{Void} as the field type. You should just figure out how you would write such an interface code in C and just map that to julia.

> [@yuyichao](#):
>
> Ref{Ptr{Void}} Ref [https://docs.julialang.org/en/latest/manual/types/#Abstract-Types-1](https://docs.julialang.org/en/latest/manual/types/#Abstract-Types-1). In generl, your code seems to be having a lot of type instability, which isn’t necessarily an issue if you don’t care about performance.

OK. Your concern with `Ref` inside my Julia struct was due to the struct’s having an abstract field, right? I mean, the type instability, and hence the performance degradation, was due to the abstract field, or?

In C, I cannot pass `void * x` to my function if I were to change `x` to point to a different address. In the end, `x` gets copied, right? At least I couldn’t achieve it unless I was using `void **` and passing `x`’s address in the C code (similarly, `void *& x` in the C++ version, to indicate pass-by-ref).

> [@yuyichao](#):
>
> The use of ccall here is also invalid since julia is allowed to free m before any of the ccall. So you have to define unsafe\_convert and cconvert methods that does the correct convertion with defined GC behavior. You can do any pointer arithmetic you want in the unsafe\_convert after taking the address of the object with pointer\_to\_objref. See the doc for cconvert and unsafe\_convert which are mentioned here

Which of `ccall` is invalid? The one in the finalizer, _i.e._, `destruct!`? Should I put `gc_enable(false); gc_enable(true)` guards inside `destruct!` to protect `ccall`?

I have also tried using the below code, which did not succeed, either:

```julia
mutable struct MyType{T<:Real}
  val::T
  handle::Ptr{Void}
end

handle(m::MyType) = m.handle
value(m::MyType) = m.val
value!(m::MyType, val::Real) = (m.val = val)

function (::Type{MyType})(val::Real)
  obj = MyType{typeof(val)}(val, Ptr{Void}(0))
  finalizer(obj, destruct!)
  return obj
end

function destruct!(m::MyType)
  ccall(delhandle, Void, (Ptr{Void},), handle(m))
end

function (m::MyType)()
  return ccall(funhandle, Cdouble, (Cdouble, Ptr{Ptr{Void}}), value(m),
    pointer_from_objref(m.handle))
end

```

And, by the way, `pointer_from_objref(m.handle)` in this case seems to return different addresses every time I run the command.

> [@yuyichao](#):
>
> There isn’t such a list, in general though any IO function will have such an effect.

Thank you for the information.

**EDIT.** The below code for the object call seems to solve my problem, though I am not sure if it is OK/valid to write like that:

```julia
# struct definition and others same as above...

function (m::MyType)()
  objref = Ref{Ptr{Void}}(handle(m))
  res = ccall(funhandle, Cdouble, (Cdouble, Ref{Ptr{Void}}), value(m), objref)
  m.handle = objref[]
  return res
end

```

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [October 1, 2017, 8:50pm UTC](https://discourse.julialang.org/t/properly-using-finalizer-ccall-cconvert-and-unsafe-convert/6183/8 "2017-10-01T20:50:30Z")

</div>

> [@aytekinar](#):
>
> I mean, the type instability, and hence the performance degradation, was due to the abstract field, or?

This is the issue with abstract field, yes.

> [@aytekinar](#):
>
> In C, I cannot pass void \* x to my function if I were to change x to point to a different address.

In C you’ll of course have a `void *x;` field, and you’ll pass it as `&obj->x`. The same is easily doable in julia with properly defined `unsafe_convert`.

> [@aytekinar](#):
>
> Which of ccall is invalid?

All of them, actually the finalizer one might be the only valid one if it is only used as a finalizer since the GC will root the argument for you.

> [@aytekinar](#):
>
> Should I put gc\_enable(false); gc\_enable(true) guards inside destruct! to protect ccall?

Never, ever do that.

> [@aytekinar](#):
>
> I have also tried using the below code, which did not succeed, either:

You should put the conversion in `unsafe_convert` instead. Your ccall **MUST** reads `ccall(......, m)`, not `m.handle` , not `handle(m)`, not anything other than `m`.

---

<div class="post-metadata">

**Author:** ![anon61610682](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@anon61610682](https://discourse.julialang.org/u/anon61610682)\
**Post date:** [October 1, 2017, 10:10pm UTC](https://discourse.julialang.org/t/properly-using-finalizer-ccall-cconvert-and-unsafe-convert/6183/9 "2017-10-01T22:10:46Z")

</div>

Thank you for your message. I think I have improved upon the first version by taking into account your warnings:

```julia
libhandle = Libdl.dlopen(joinpath(pwd(), "libexternal.so"))
funhandle = Libdl.dlsym(libhandle, :d_somefunc)
delhandle = Libdl.dlsym(libhandle, :delhandle)

import Base: cconvert, unsafe_convert

mutable struct MyType{T<:Real}
  val::T
  handle::Ptr{Void}
end

value(m::MyType) = m.val
value!(m::MyType, val::Real) = (m.val = val)

cconvert(t::Type{Cdouble}, m::MyType) = cconvert(t, value(m))
unsafe_convert(t::Type{Ptr{Void}}, m::MyType) = m.handle

function destruct!(m::MyType)
  ccall(delhandle, Void, (Ptr{Void},), m)
end

function (::Type{MyType})(val::Real)
  obj = MyType{typeof(val)}(val, Ptr{Void}(0))
  finalizer(obj, destruct!)
  return obj
end

function (m::MyType)()
  handleref = Ref{Ptr{Void}}(m.handle)
  res = ccall(funhandle, Cdouble, (Cdouble, Ref{Ptr{Void}}), m, handleref)
  m.handle = handleref[]
  return res
end

m = MyType(1)
m()
value!(m, 5)
m()
value!(m, 7)
m()

# Run below for 4 times
for k in 1:4
  m = MyType(1)
  m()
  value!(m, 7)
  m()
end

gc()

```

Now, `ccall` does not seem to have `obj.field` parts, nor does my `unsafe_convert` have any expression, as is forbidden in the documentation.

The only problem, apparently, is that there is no difference between `Ref{Ptr{Void}}` and `Ptr{Void}` in Julia. `Ref{Ptr{Void}}` is also calling `convert(::Type{Ptr{Void}}, x)`. For this reason, I needed to encapsulate `m.handle` inside `Base.RefValue` inside the functor.

> [@yuyichao](#):
>
> In C you’ll of course have a void \*x; field, and you’ll pass it as &obj-\>x.

Still I cannot understand your point above — forgive me. I am having the problem:

```c
double somefunc(const double val, void *x) {
  double result;
  // do something with x->func(val) and set result to something
  // change x to point to a different object
  return result;
}

```

Since I cannot change the argument `x`’s value, but only its copy inside the function, I need `void **` here. Then, I can simply

```c
double somefunc(const double val, void **x) {
  double result;
  // do something with (*x)->func(val) and set result to something
  // change x to point to a different object by
  // *x = new_address;
  return result;
}

```

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [October 2, 2017, 2:47am UTC](https://discourse.julialang.org/t/properly-using-finalizer-ccall-cconvert-and-unsafe-convert/6183/10 "2017-10-02T02:47:04Z")

</div>

> [@aytekinar](#):
>
> nor does my unsafe\_convert have any expression, as is forbidden in the documentation.

This is not forbidden, the forbidden thing is returning address of anything you allocated in the function.

> [@aytekinar](#):
>
> no difference between Ref{Ptr{Void}} and Ptr{Void} in Julia.

That’s not true, see below.

> [@aytekinar](#):
>
> Still I cannot understand your point above

Yes, your function will take a `void **` that’s fine, but if you want to mutate a `void*` in a C struct, you don’t define that field as `void** `, you’ll just define it as `void*`. And you’ll use it as

```c++
struct mytype {
    double v;
    void *ptr;
};

// and some code in a function
mytype *obj;
// ...
somefunction(obj->v, &obj->ptr);

```

And you can write a exact equivalent in julia.

> [@aytekinar](#):
>
> ```julia
> handleref = Ref{Ptr{Void}}(m.handle)
> res = ccall(funhandle, Cdouble, (Cdouble, Ref{Ptr{Void}}), m, handleref)
> m.handle = handleref[]
> 
> ```

No this is not necessary.

Also

> [@aytekinar](#):
>
> cconvert(t::Type{Cdouble}, m::MyType) = cconvert(t, value(m))

is fine but also not necessary. The validity of the `::Cdouble` doesn’t rely on the liveness of the `::MyType`. It’s certainly valid code though, just a little over-complicated.

The direct (and **invalid**!) way to implement the c++ code above in julia would simply be

```julia
ccall(somefunction, Cdouble, (Cdouble, Ptr{Ptr{Void}}), obj.v, pointer_to_objref(obj) + fieldoffset(typeof(obj), 2))

```

The only invalid part of the above code is that `pointer_to_objref(obj) + fieldoffset(typeof(obj), 2)` will not keep `obj` alive during the call. And the fix would be to somehow keep it alive. This can be acomplished either with `@gc_preserve` on 0.7 like

```julia
Base.@gc_preserve obj ccall(somefunction, Cdouble, (Cdouble, Ptr{Ptr{Void}}), obj.v, pointer_to_objref(obj) + fieldoffset(typeof(obj), 2))

```

which will be the preferred way **if** the convertion is callsite specific. On 0.6 where there’s no `@gc_preserve` or if this call convertion is used multiple times, it is preferred to use the `unsafe_convert`/`cconvert` and you can do this simply by putting that unsafe convert in, well, `unsafe_convert`, sth like.

```julia
Base.unsafe_convert(::Type{Ptr{Ptr{Void}}}, obj::MyType) = Ptr{Ptr{Void}}(pointer_to_objref(obj) + fieldoffset(typeof(obj), 2))

```

And simply do

```julia
ccall(somefunction, Cdouble, (Cdouble, Ptr{Ptr{Void}}), obj.v, obj)

```

The use of either `Ref{Ptr{Void}}` or `Ptr{Ptr{Void}}` is valid and doesn’t really matter. In fact, any `Ref` or `Ptr` type should work. I usually prefer `Ptr` though since `Ref` has special meaning and it is better to not mess with it.

> [@aytekinar](#):
>
> ```julia
> return ccall(funhandle, Cdouble, (Cdouble, Ptr{Ptr{Void}}), value(m),
> pointer_from_objref(m.handle))
> 
> ```

And I’ll also add that I can see how this is a failed attempt at translating the C code I translated above. The important thing to note here about the difference between C and julia is that there’s no C reference in julia. Even though it is not called so in C, `obj.field` produces a lvalue reference which is why `&` on it gives you the address **inside** `obj`. OTOH, `obj.field` always produce a rvalue in julia, or in another word, local variable assignment is never significant, you can insert as many of them as you want in the code and they’ll never make any difference in semantics. This means that `pointer_from_objref(m.handle)` corresponds to

```c++
{
    void *handle = m->handle;
    return &handle;
}

```

and it should be easy to see why it doesn’t give you the result you want. That’s also why you need to explicitly take the address of `m` in order to get the address of `handle` field in `m`.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [October 2, 2017, 2:59am UTC](https://discourse.julialang.org/t/properly-using-finalizer-ccall-cconvert-and-unsafe-convert/6183/11 "2017-10-02T02:59:26Z")

</div>

> [@yuyichao](#):
>
> This is not forbidden, the forbidden thing is returning address of anything you allocated in the function.

Also note that this is not forbidden just for `unsafe_convert` but forbidden in any function (ok, well, it’s not really forbidden but it is just undefined behavior if the caller want to use the pointer to load/store anything. It’s obviously fine to return any pointer you feel like as long as you promise not to use that pointer for anything bad).

In fact, there is **less** restriction on `unsafe_convert` than any other functions. It is the only function you can return an address derived from the argument since the caller must keep the arguments valid if it want to use the returned pointer. You can of course require your own functions to have a similar semantic (i.e. restriction on the caller) but `unsafe_convert` is the only Base API that does.

---

<div class="post-metadata">

**Author:** ![anon61610682](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@anon61610682](https://discourse.julialang.org/u/anon61610682)\
**Post date:** [October 2, 2017, 7:23am UTC](https://discourse.julialang.org/t/properly-using-finalizer-ccall-cconvert-and-unsafe-convert/6183/12 "2017-10-02T07:23:16Z")

</div>

Wow! Thank you for all the explanations — this was a bit trickier than I had expected.

I would like to clarify a couple of things for myself, by repeating what you had written, if you don’t mind.

> [@yuyichao](#):
>
> This is not forbidden, the forbidden thing is returning address of anything you allocated in the function.

> [@yuyichao](#):
>
> Also note that this is not forbidden just for unsafe\_convert but forbidden in any function (ok, well, it’s not really forbidden but it is just undefined behavior if the caller want to use the pointer to load/store anything. It’s obviously fine to return any pointer you feel like as long as you promise not to use that pointer for anything bad).
> 
> In fact, there is less restriction on unsafe\_convert than any other functions. It is the only function you can return an address derived from the argument since the caller must keep the arguments valid if it want to use the returned pointer. You can of course require your own functions to have a similar semantic (i.e. restriction on the caller) but unsafe\_convert is the only Base API that does.

Sorry for my misuse of the word _forbidden_. I meant to refer to the following paragraph in the [documentation](https://docs.julialang.org/en/latest/stdlib/c/#Base.unsafe_convert) for `Base.unsafe_convert`:

> …  
> Be careful to ensure that a Julia reference to x exists as long as the result of this function will be used. Accordingly, the argument x to this function should never be an expression, only a variable name or field reference. For example, x=a.b.c is acceptable, but x=[a,b,c] is not.  
> …

This piece of information was the reason I was trying to avoid using any expression but the object itself or its field names via `obj.fieldname` above. But now, I see that even if `Ptr{Ptr{Void}}(pointer_to_objref(obj) + fieldoffset(typeof(obj), 2))` creates a temporary, the temporary is simply the address of something which will stay alive during the call.

Thank you once more for clarification.

> [@yuyichao](#):
>
> That’s not true, see below.

Regarding `Ref{Ptr{Void}}`, I simply wanted to say that I could not use the method overload `cconvert(::Type{Ref{Ptr{Void}}}, x)` when there was `cconvert(::Type{Ptr{Void}}, x)` present. But, yet again, what I _should have_ done was to use `Ptr{Ptr{Void}}` as you have pointed out already.

> [@yuyichao](#):
>
> The only invalid part of the above code is that pointer\_to\_objref(obj) + fieldoffset(typeof(obj), 2) will not keep obj alive during the call.  
> …

And this is because I am simply creating a temporary, directly inside `ccall`, which does not guarantee anything related to the actual object’s staying live or not, right? Basically, indirection **must** be **avoided**.

> [@yuyichao](#):
>
> And simply do
> 
> `ccall(somefunction, Cdouble, (Cdouble, Ptr{Ptr{Void}}), obj.v, obj)`

The above code will be valid, provided that I have _at least one_ `obj` passed _directly_ inside `ccall`, right?

> [@yuyichao](#):
>
> The use of either Ref{Ptr{Void}} or Ptr{Ptr{Void}} is valid and doesn’t really matter. In fact, any Ref or Ptr type should work. I usually prefer Ptr though since Ref has special meaning and it is better to not mess with it.

Uhm… Again, as I said above, `Ref{Ptr{Void}}` does not resolve to `Ref{Ptr{Void}}` overload of `unsafe_convert`. For this reason, I needed to use `Ptr{Ptr{Void}}` in my overload (see below).

Moreover, what do you mean by _special meaning_ of `Ref`? Is it only that `Ref` is used to indicate the variable is in gc’s responsibility as opposed to `Ptr`s being the C library’s, or is there something else? To avoid any misuse of wording, as I had already done with “_forbidden_”, I refer to the following [statements](https://docs.julialang.org/en/latest/manual/calling-c-and-fortran-code/#When-to-use-T,-Ptr%7BT%7D-and-Ref%7BT%7D-1):

> For C code accepting pointers, Ref{T} should generally be used for the types of input arguments, allowing the use of pointers to memory managed by either Julia or C through the implicit call to Base.cconvert. In contrast, pointers returned by the C function called should be declared to be of output type Ptr{T}, reflecting that the memory pointed to is managed by C only.

> [@yuyichao](#):
>
> Even though it is not called so in C, obj.field produces a lvalue reference which is why & on it gives you the address inside obj. OTOH, obj.field always produce a rvalue in julia, or in another word, local variable assignment is never significant, you can insert as many of them as you want in the code and they’ll never make any difference in semantics. This means that pointer\_from\_objref(m.handle) corresponds to

I slightly disagree with your comment on C and lvalue reference, but your explanation makes perfect sense. I think the dot operator in Julia has a different meaning from that in Julia. But thinking, _roughly_, that `pointer_from_objref(m.handle)` actually means `pointer_from_objref(getfield(m, :handle))` clarifies the observation I had. Again, I was reaching for the address of a temporary.

Thank you very much for your time and explanations. Below is the correct version, I hope, for the last time:

```julia
libhandle = Libdl.dlopen(joinpath(pwd(), "libexternal.so"))
funhandle = Libdl.dlsym(libhandle, :d_somefunc)
delhandle = Libdl.dlsym(libhandle, :delhandle)

import Base: cconvert, unsafe_convert

mutable struct MyType{T<:Real}
  val::T
  handle::Ptr{Void}
end

value(m::MyType) = m.val
value!(m::MyType, val::Real) = (m.val = val)

unsafe_convert(t::Type{Ptr{Void}}, m::MyType) = m.handle
unsafe_convert(t::Type{Ptr{Ptr{Void}}}, m::MyType) =
    Ptr{Ptr{Void}}(pointer_from_objref(m) + fieldoffset(typeof(m), 2))

function destruct!(m::MyType)
  ccall(delhandle, Void, (Ptr{Void},), m)
end

function (::Type{MyType})(val::Real)
  obj = MyType{typeof(val)}(val, Ptr{Void}(0))
  finalizer(obj, destruct!)
  return obj
end

function (m::MyType)()
  res = ccall(funhandle, Cdouble, (Cdouble, Ptr{Ptr{Void}}), value(m), m)
end

m = MyType(1)
m()
value!(m, 5)
m()
value!(m, 7)
m()

# Run below for 4 times
for k in 1:4
  m = MyType(1)
  m()
  value!(m, 7)
  m()
end

gc()

```

I mean, it works as expected, finally! It is also easier to parse if one knows how to write the counterpart in the C version.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [October 2, 2017, 1:21pm UTC](https://discourse.julialang.org/t/properly-using-finalizer-ccall-cconvert-and-unsafe-convert/6183/13 "2017-10-02T13:21:51Z")

</div>

> [@aytekinar](#):
>
> this was a bit trickier than I had expected.

As I said, if you just figure out how you would write the whole thing in C, it shouldn’t be too hard to translate it to julia. And with a few tweaks you can make it valid julia.

> [@aytekinar](#):
>
> Sorry for my misuse of the word forbidden. I meant to refer to the following paragraph in the documentation for Base.unsafe\_convert:

Forbidden isn’t misused afaik. Though that doc is misleading. It’s trying to say that `unsafe_convert(T, a.b.c)` is allowed while `unsafe_convert(T, [a, b, c])` isn’t though really none is allowed on it’s own…

> [@aytekinar](#):
>
> `Ptr{Ptr{Void}}(pointer_to_objref(obj) + fieldoffset(typeof(obj), 2))` creates a temporary, the temporary is simply the address of something which will stay alive during the call.

Not sure what you mean by temporary here. Every expression creates “temporary” and that’s not a problem. It’s not even a issue if the expression allocates. The only issue is if the expression takes the address of an object and uses the address without making sure the object is always alive, whether or not that object is a “temporary”. Semantically even this expression may allocate, it’s just that the optimizer should be able to do a good job almost all of the cases. And even if this expression allocates the pointer on the heap, it is actually not necessary to keep that alive since the value will be used, not the heap allocated `Ptr` object.

> [@aytekinar](#):
>
> I could not use the method overload cconvert(::Type{Ref{Ptr{Void}}}, x) when there was cconvert(::Type{Ptr{Void}}, x)

That’s what I thought you mean and that shouldn’t be the case…

> [@aytekinar](#):
>
> And this is because I am simply creating a temporary, directly inside ccall, which does not guarantee anything related to the actual object’s staying live or not, right?

That’s not the issue. The issue is that with a few exceptions, for any expression `f(a, b, g(c, h(d)))` (the function call is just for illustration can be replaced with any other expression like operators and getfield (which are really special function calls…)) they are semantically equivalent to

```julia
t1 = h(d)
t2 = g(c, t1)
t3 = f(a, b, t2)

```

And there’s no guarantee that any of the variable is alive other than where it appears in the code. E.g. `d` may be dead during the second or third expression and `a` may be dead during the first two expressions!!! (The latter can happen due to compiler optimization especially dead code elimination). This is also very different from C++ where the lifetime of `t1` and `t2` are guaranteed beforee the whole expression finishes. So it’s invalid because it’s using the address of `obj` but `ccall(somefunction, Cdouble, (Cdouble, Ptr{Ptr{Void}}), obj.v, pointer_to_objref(obj) + fieldoffset(typeof(obj), 2))` doesn’t tell the compiler that it should preserve the `obj` during the ccall, which is also why that’s the only thing that need fixing. In another word, `pointer_to_objref(obj) + fieldoffset(typeof(obj), 2)` by itself does not create any problematic temporaries (i.e. it doesn’t take address of any temporaries). It’s just that it returns a value whose validity is tied to the liveness of `obj` and the user need to be aware of that.

> [@aytekinar](#):
>
> And simply do
> 
> ccall(somefunction, Cdouble, (Cdouble, Ptr{Ptr{Void}}), obj.v, obj)
> 
> The above code will be valid, provided that I have at least one obj passed directly inside ccall, right?

The above code should already be valid and the reason for that is `obj` is used as the return value of `cconvert` for one of the argument. What you said is kind of true since breaking that requires a invalid `cconvert` and with a valid `cconvert` what you said should be true.

> [@aytekinar](#):
>
> Is it only that Ref is used to indicate the variable is in gc’s responsibility as opposed to Ptrs being the C library’s,

No, not at all, absolutely not. The mention of that in the doc (what you linked and quoted below) seems to be confusing a large number of people…  
See [Use Ref to denote GC-managed memory. by maleadt · Pull Request #56 · JuliaGPU/CUDAdrv.jl · GitHub](https://github.com/JuliaGPU/CUDAdrv.jl/pull/56). Basically it doesn’t have any special meaning to the GC, not at all. It’s treated exactly the same as `Ptr`, just with a few methods defined differently to give it the special meaning mentioned below.

The special meaning is the special `cconvert` rule it has to pass a value by reference “implicitly” like `ccall(..., (..., Ref{Int}, ...), ..., 1, ...)` You shouldn’t overload `unsafe_convert/cconvert(::Type{Ref{T}}, ::T2)` where `T2` can be reasonably converted to `T`. Anything else should be fine though I see no reason to do so… Note that you do need to overwrite both `cconvert` and `unsafe_convert` though since the special `cconvert` behavior means that it doesn’t have a no-op default `cconvert`. I don’t see a reason to use `Ref` here though and since whether `T2` can be reasonably converted to `T` is pretty fuzzy it’s better not to mess with it.

> [@aytekinar](#):
>
> I slightly disagree with your comment on C and lvalue reference

Sure, I’m just using C++ terminology which generalizes the semantic of these sub expression results. It’s definately a lvalue reference in c++, in which case even `&handle(m)` can be valid.

> [@aytekinar](#):
>
> Below is the correct version, I hope, for the last time:

Yep, looks correct. The other type instability I mentioned is

> [@aytekinar](#):
>
> libhandle = Libdl.dlopen(joinpath(pwd(), “libexternal.so”))  
> funhandle = Libdl.dlsym(libhandle, :d\_somefunc)  
> delhandle = Libdl.dlsym(libhandle, :delhandle)

You should just use `(:d_somefunc, libpath)` in ccall directly or make them const or `Ref{Ptr{Void}}`. What you have is perfectly correct though, just slightly inefficient.

---

<div class="post-metadata">

**Author:** ![anon61610682](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@anon61610682](https://discourse.julialang.org/u/anon61610682)\
**Post date:** [October 3, 2017, 7:56am UTC](https://discourse.julialang.org/t/properly-using-finalizer-ccall-cconvert-and-unsafe-convert/6183/14 "2017-10-03T07:56:33Z")

</div>

> [@yuyichao](#):
>
> That’s what I thought you mean and that shouldn’t be the case…

I can confirm that this is **not** working. Maybe I should open an issue then on GitHub. I might be wrongly implementing this, but I have tried **(not valid anymore, see below)**

```julia
unsafe_convert(t::Type{Ptr{Void}}, m::MyType) = m.handle
unsafe_convert(t::Type{Ref{Ptr{Void}}}, m::MyType) =
    Ref{Ptr{Void}}(pointer_from_objref(m) + fieldoffset(typeof(m), 2))

```

as well as the `Ptr{Ptr{Void}}` casted return type for the second overload above (with the proper `ccall` change), and I have always got

```julia
MethodError: Cannot `convert` an object of type MyType{Int64} to an object of type Ptr{Void}
This may have arisen from a call to the constructor Ptr{Void}(...),
since type constructors fall back to convert methods.

```

Anyways… For me, this is not a problem anymore, and I do think that the `Ptr{Ptr{Void}}` solution _should_ be the preferred way. That’s just more reasonable from its C counterpart.

> [@yuyichao](#):
>
> The special meaning is the special cconvert rule it has to pass a value by reference “implicitly” like ccall(…, (…, Ref{Int}, …), …, 1, …) You shouldn’t overload unsafe\_convert/cconvert(::Type{Ref{T}}, ::T2) where T2 can be reasonably converted to T. Anything else should be fine though I see no reason to do so… Note that you do need to overwrite both cconvert and unsafe\_convert though since the special cconvert behavior means that it doesn’t have a no-op default cconvert. I don’t see a reason to use Ref here though and since whether T2 can be reasonably converted to T is pretty fuzzy it’s better not to mess with it.

**EDIT & CORRECTION.** Actually, following this comment, I _think_ I have found the bug in my above code snippet. Since, as you pointed out, I _have to_ overload both of them, I need the below code snippet (assuming `T = Ptr{Void}` and `T2 = MyType`) to make things work correctly:

```julia
unsafe_convert(t::Type{Ptr{Void}}, m::MyType) = m.handle
cconvert(t::Type{Ref{Ptr{Void}}}, m::MyType) = m
# no reasonable conversion above. hence, define he conversion to be
# the object itself. then, let the unsafe convert do its job below
unsafe_convert(t::Type{Ref{Ptr{Void}}}, m::MyType) =
    Ptr{Ptr{Void}}(pointer_from_objref(m) + fieldoffset(typeof(m), 2))

```

Prior to your comment above, I was getting the method error I have mentioned in this message. Now I understand the reason: _lack of no-op cconvert_.

For all the help, thank you very much, once more!