Properly using `finalizer`, `ccall`, `cconvert` and `unsafe_convert`

[yuyichao]

@aytekinar

this was a bit trickier than I had expected.

As I said, if you just figure out how you would write the whole thing in C, it shouldn’t be too hard to translate it to julia. And with a few tweaks you can make it valid julia.

@aytekinar

Sorry for my misuse of the word forbidden. I meant to refer to the following paragraph in the documentation for Base.unsafe_convert:

Forbidden isn’t misused afaik. Though that doc is misleading. It’s trying to say that unsafe_convert(T, a.b.c) is allowed while unsafe_convert(T, [a, b, c]) isn’t though really none is allowed on it’s own…

@aytekinar

Ptr{Ptr{Void}}(pointer_to_objref(obj) + fieldoffset(typeof(obj), 2)) creates a temporary, the temporary is simply the address of something which will stay alive during the call.

Not sure what you mean by temporary here. Every expression creates “temporary” and that’s not a problem. It’s not even a issue if the expression allocates. The only issue is if the expression takes the address of an object and uses the address without making sure the object is always alive, whether or not that object is a “temporary”. Semantically even this expression may allocate, it’s just that the optimizer should be able to do a good job almost all of the cases. And even if this expression allocates the pointer on the heap, it is actually not necessary to keep that alive since the value will be used, not the heap allocated Ptr object.

@aytekinar

I could not use the method overload cconvert(::Type{Ref{Ptr{Void}}}, x) when there was cconvert(::Type{Ptr{Void}}, x)

That’s what I thought you mean and that shouldn’t be the case…

@aytekinar

And this is because I am simply creating a temporary, directly inside ccall, which does not guarantee anything related to the actual object’s staying live or not, right?

That’s not the issue. The issue is that with a few exceptions, for any expression f(a, b, g(c, h(d))) (the function call is just for illustration can be replaced with any other expression like operators and getfield (which are really special function calls…)) they are semantically equivalent to

t1 = h(d)
t2 = g(c, t1)
t3 = f(a, b, t2)

And there’s no guarantee that any of the variable is alive other than where it appears in the code. E.g. d may be dead during the second or third expression and a may be dead during the first two expressions!!! (The latter can happen due to compiler optimization especially dead code elimination). This is also very different from C++ where the lifetime of t1 and t2 are guaranteed beforee the whole expression finishes. So it’s invalid because it’s using the address of obj but ccall(somefunction, Cdouble, (Cdouble, Ptr{Ptr{Void}}), obj.v, pointer_to_objref(obj) + fieldoffset(typeof(obj), 2)) doesn’t tell the compiler that it should preserve the obj during the ccall, which is also why that’s the only thing that need fixing. In another word, pointer_to_objref(obj) + fieldoffset(typeof(obj), 2) by itself does not create any problematic temporaries (i.e. it doesn’t take address of any temporaries). It’s just that it returns a value whose validity is tied to the liveness of obj and the user need to be aware of that.

@aytekinar

And simply do

ccall(somefunction, Cdouble, (Cdouble, Ptr{Ptr{Void}}), obj.v, obj)

The above code will be valid, provided that I have at least one obj passed directly inside ccall, right?

The above code should already be valid and the reason for that is obj is used as the return value of cconvert for one of the argument. What you said is kind of true since breaking that requires a invalid cconvert and with a valid cconvert what you said should be true.

@aytekinar

Is it only that Ref is used to indicate the variable is in gc’s responsibility as opposed to Ptrs being the C library’s,

No, not at all, absolutely not. The mention of that in the doc (what you linked and quoted below) seems to be confusing a large number of people…
See Use Ref to denote GC-managed memory. by maleadt · Pull Request #56 · JuliaGPU/CUDAdrv.jl · GitHub. Basically it doesn’t have any special meaning to the GC, not at all. It’s treated exactly the same as Ptr, just with a few methods defined differently to give it the special meaning mentioned below.

The special meaning is the special cconvert rule it has to pass a value by reference “implicitly” like ccall(..., (..., Ref{Int}, ...), ..., 1, ...) You shouldn’t overload unsafe_convert/cconvert(::Type{Ref{T}}, ::T2) where T2 can be reasonably converted to T. Anything else should be fine though I see no reason to do so… Note that you do need to overwrite both cconvert and unsafe_convert though since the special cconvert behavior means that it doesn’t have a no-op default cconvert. I don’t see a reason to use Ref here though and since whether T2 can be reasonably converted to T is pretty fuzzy it’s better not to mess with it.

@aytekinar

I slightly disagree with your comment on C and lvalue reference

Sure, I’m just using C++ terminology which generalizes the semantic of these sub expression results. It’s definately a lvalue reference in c++, in which case even &handle(m) can be valid.

@aytekinar

Below is the correct version, I hope, for the last time:

Yep, looks correct. The other type instability I mentioned is

@aytekinar

libhandle = Libdl.dlopen(joinpath(pwd(), “libexternal.so”))
funhandle = Libdl.dlsym(libhandle, :d_somefunc)
delhandle = Libdl.dlsym(libhandle, :delhandle)

You should just use (:d_somefunc, libpath) in ccall directly or make them const or Ref{Ptr{Void}}. What you have is perfectly correct though, just slightly inefficient.