Cfunction from 0.6 to 1.0

General Usage
1

I am upgrading a package that wraps a C library from julia 0.6 to 1.0. In principle this should be very simple, but in practice I am running into some problems (naturally). In particular, migrating from 0.6 to 0.7 as an intermediate, I go from all tests passing to encountering a segfault early on from a ccall.

The following code is the segfault culprit in 0.7, but works in 0.6

# `functions` contains symbol pairs
for (name, C_name) in functions
    # define julia function with ccall
    @eval $name(mgr::Ptr{Manager}, f_pp::Ptr{Ptr{Node}}, g_pp::Ptr{Ptr{Node}}) =
                ccall(($(QuoteNode(C_name)), _LIB_CUDD),
                      Ptr{Node}, (Ptr{Manager}, Ptr{Ptr{Node}}, Ptr{Ptr{Node}}), mgr, f_pp, g_pp)
    # create c function pointer
    @eval $(Symbol(name, "_c")) = Compat.@cfunction($name, Ptr{Node}, (Ptr{Manager}, Ptr{Ptr{Node}}, Ptr{Ptr{Node}}))
end

What I have noticed in manually inspecting the pointers generated by the second @eval is that in 0.6 they are each unique, while in 0.7 they are all Ptr{Nothing} @0x0000000000000000. I assumed using Compat.@cfunction would be all that was necessary to migrate this code, but it appears not. If anyone knows what is missing / what is going on, I’d appreciate the input!

2

For reference (and ease of reading), here is one function in functions written out explicitly instead of using @eval.

function add_plus(mgr::Ptr{Manager}, f_pp::Ptr{Ptr{Node}}, g_pp::Ptr{Ptr{Node}})
    func = ccall((:Cudd_addPlus, _LIB_CUDD),
        Ptr{Node}, (Ptr{Manager}, Ptr{Ptr{Node}}, Ptr{Ptr{Node}}), mgr, f_pp, g_pp)
    return func
end
add_plus_c = Compat.@cfunction(add_plus, Ptr{Node}, (Ptr{Manager}, Ptr{Ptr{Node}}, Ptr{Ptr{Node}}))
3

Seems to work fine for me in both Julia 0.7 and Julia 1.2 (using dummy definitions struct Manager; end and struct Node; end) and removing the Compat. in Julia 1.2. At least, I’m getting a non-NULL pointer back.

4

Seems to work fine for me

Fair enough! Turns out the null pointers were from when I was looking at 1.0, and in 0.7 they are indeed valid. That said, I see the segfault in both 0.7 and 1.0. I haven’t tried in 1.2 (or 1.1 for that matter) but I’ll give that a go as well.

I also forgot a potentially crucial piece of information. All of the functions defined in @eval are actually called inside yet another ccall:

function add_apply(mgr::Ptr{Manager}, op, f::Ptr{Node}, g::Ptr{Node})
    res = ccall((:Cudd_addApply, _LIB_CUDD),
        Ptr{Node}, (Ptr{Manager}, Ptr{Cvoid}, Ptr{Node}, Ptr{Node}), mgr, op, f, g)
    if res == C_NULL # Fails to compute a result
        throw(OutOfMemoryError())
    end
    return res
end

So the call that segfaults ultimately looks like add_apply(manager, add_plus_c, f, g)

EDIT:
Segfaulted also in 1.2.
Also, I’m noticing now that the pointers in all versions of julia are unique if the code above is run at the terminal (and also if using @show at the point they are defined). Inspecting the value of the pointer in the module after using it, show it is null however. Any ideas @stevengj?

5

Precompilation of modules is off by default in Julia v0.6 and on by default in v0.7 and above. In addition, any pointers in global variables are set to null during precompilation to avoid baking in pointers to arbitrary memory addresses (which will be completely different when the precompiled module is loaded later on).

It seems like your module is not safe for precompilation, which is causing the problem you’re seeing. Rather than trying to construct cfunctions as global variables in your module, you should be able to use the const Ref trick, e.g.:

module Foo

const add_plus_c = Ref{<fill this in>}()

function __init__()
  add_plus_c[] = @cfunction(...)
end

end

For more on precompilation, see: Modules · The Julia Language

3 Likes
6

Thanks @rdeits! I didn’t know that about precompilation, but that seems like a likely culprit. I’ll try the Ref trick.

Edit: setting __precompile__(false) gets past the tests, so I’d say this may well be solved!

7

Yeah, turning off precompilation will avoid the issue, but note that it means that no package which depends on your package can be precompiled either, which may be annoying. Using a const Ref, on the other hand, will let you and any downstream users benefit from precompilation.

8

Indeed, I’ll go through with the const Ref change as well. Thanks again @rdeits.

9

Just to clarify what goes in Ref{<fill this in>}, should it be Ptr{Nothing} since that’s what the @cfunction returns?

10

Yup, exactly

11

I hate to return bearing bad news, but using Refs leads to a different issue. The pointers are now all valid, but running the tests still fails, this time with “Bus error: 10” rather than a segfault.

Not sure if it matters, but I’m defining and filling the Refs in a loop as before, with __init__() looking as follows (could probably be written much simpler, but oh well)


function __init__()
    for (name, _) in functions
        @eval $(Symbol(name, "_c"))[] = Compat.@cfunction($name, Ptr{Node}, (Ptr{Manager}, Ptr{Ptr{Node}}, Ptr{Ptr{Node}}))
    end
end

I also tried defining a few the explicit way and saw the same result.

12

I’m not sure what’s going on there. One thing to note is that if you are using a Ref foo to hold your cfunction, then you need to be sure to replace all the usages of foo with foo[]. Have you already done that?

1 Like
13

I hadn’t! I remembered reading something about how the Ref was converted into a pointer automatically in the ccall…?
Not sure why I thought that, and in any case I guess that wouldn’t even be what I want.

Anyway, everything works now. Thanks a lot for all the help!

14

You’re right–a Ref will get converted into a pointer, but in this case it’s going to be a pointer to the function pointer, not the function pointer itself (the Ref essentially adds one more layer of indirection).