Conversion of nothing to void pointer

[projekter]

It is quite common in C interfaces to have functions that accept pointers to something, but they can also be given NULL instead, which means that for example the part that is related with this parameter is not of interested and does not have to be calculated. In a Julia interface, the most natural way to achieve this is to make the corresponding parameter a Union{Nothing,X}, if X is, say, an array type.
I was a bit astonished to find that there is by default no conversion defined that casts nothing to a C_NULL, so the “simple” ccall with a Ptr{eltype(X)} will fail and explicit conversion has to be done.
Is there a (good) reason why Base does not contain

Base.convert(t::Type{<:Ptr}, ::Nothing) = t(C_NULL)

(and the same for missing probably)?

[jpsamaroo]

It’s an interesting idea for sure, and does seem sensible as it doesn’t otherwise have a sensible behavior w.r.t ccall. The only concern is that it’s easy to unintentionally generate nothing in Julia (because some other part of your code isn’t working correctly) and then suddenly end up with a segfaults when the C API you call into isn’t prepared for a null in that argument position. Still, I’m personally in favor of this, as it seems very convenient.

[Sukera]

I’m more in the camp of “be explicit about what you mean & what you expect”; if you don’t know whether the object you’re going to pass to C is a Nothing or a T, not having the conversion defined by default will alert you to something fishy going on. As such, I think such a conversion is more appropriately placed in a function wrapping that specific C call, which can then distinguish between Nothing and T through an if or even dispatch.

There shouldn’t really be a raw ccall in regular code; it should always be wrapped, to ensure the expectations of that ccall are met (as well as possible error return values be checked).

[jakobnissen]

Null pointers and their related segfaults are such an awful part about C. We are very fortunate in Julia to have nothing, which allows us to express optionality in the type system, and which will produce a debuggable error when used erroneously.
Wanting an implicit convert from nothing to null pointers seems to me like introducing C-level recklessness into Julia, at a time when everyone is working to introduce safety in C-like languages. It’s a terrible idea.

[joa-quim]

@projekter

Union{Nothing,X}, if X is, say, an array type.

Interfacing this with C when X is an array of floats is a real pain. Need to make a copy and replace the (ghrrr) Nothings by NaNs.

[projekter]

I agree that if there’s an easy way in the language to prevent accidental mistakes without making it harder to write something that should be simple, then it should be enforced in the language. But I would argue that precisely would still be there even with auto-conversion:

• In my Julia wrapper for the C function, I have to explicitly allow Nothing as a type*, so I am already aware of the fact that something “nully” can go on.
• There are two ways to pass pointers via ccall: by saying that the type is a Ref or a Ptr. By choosing Ref, I explictly make sure that this is non-null data. I only choose Ptr if null can happen.
  So this is already a lot of safeguarding. Making this still more verbatim by forcing the developer to write isnothing(x) ? C_NULL : x is not a big deal, but I also don’t see how it helps avoiding pitfalls.

(*) You might of course say, what if my parameter is untyped. True. But then, you either have an interface contract (such as stride) that doesn’t work on nothing and will therefore already fail before or require special attention. Or you decided on purpose not to make use of Julia’s type safety. But even then, this won’t bite you in the neck even with automatic conversion, provided you disallow null pointers by using Ref instead of Ptr. This is the beauty in having two - from a C point of view equivalent - pointer types.

Let’s just do a quick comparison, which is of course a rather pointless example (and I even violate my first point, but still null-safety is guaranteed easily):

testfn1(x) = @ccall mylib.test(x::Ptr{Cdouble})::Cvoid
testfn2(x) = @ccall mylib.test((isnothing(x) ? C_NULL : x)::Ptr{Cdouble})::Cvoid
testfn3(x) = @ccall mylib.test(x::Ref{Cdouble})::Cvoid
testfn4(x) = @ccall mylib.test((isnothing(x) ? C_NULL : x)::Ref{Cdouble})::Cvoid

As I said, very crude. Let’s just assume that mylib.test does some operation on one double and nothing if it gets a null pointer. Then:

• testfn1 works if it gets actual data. It will currently fail with nothing, but would work if the conversion was added.
• testfn2 works in all cases.
• testfn3 works if it gets actual data. Even if the conversion is added, it will fail with nothing (on the Julia side), as it can’t be converted to a Ref.
• testfn4 makes no sense for the same reason.

@joa-quim: This is not what I’m talking about (I guess, as I’m not sure what you are talking about). I don’t want to have an array with eltypes that might be nothing or some actual values. This will clearly be quite painful, as it also depends on how Julia internally represents this - which might even change from version to version. I’m only talking about the parameter (i.e., the whole vector) itself being optional. With Fortran interfaces, this is even natively integrated in the language by the optional keyword.

[joa-quim]

@projekter

I don’t want to have an array with eltypes that might be nothing

Me neither, and that’s what I was talking about. My only solution is make a copy of the array (of the float type, either 32 or 64), copy all non-Nothihng elements and replace the Nothings by NaN. If anything more clever exists I would love to know it.

[nalimilan]

@joa-quim

Me neither, and that’s what I was talking about. My only solution is make a copy of the array (of the float type, either 32 or 64), copy all non-Nothihng elements and replace the Nothings by NaN. If anything more clever exists I would love to know it.

See Feature request: convert between Array{Union{T, Missing}, N} and Array{T, N} without copying · Issue #26681 · JuliaLang/julia · GitHub. Technically it is possible to grab the underlying array of values, reinterpret that memory as an array of Float64, set entries corresponding to nothing or missing to NaN, and pass a pointer to that memory to C. This should be safe as Julia does not care about the memory contents of those entries (the fact that they are nothing or missing is known from the separate type tag array). But in practice this hasn’t been implemented in a clean way so you’d have to hack this manually.

[Sukera]

@projekter

Let’s just do a quick comparison, which is of course a rather pointless example (and I even violate my first point, but still null-safety is guaranteed easily):

What about this approach using dispatch?

testfn(x::Nothing)         = @ccall mylib.test(C_Null::Ptr{Cdouble})::Cvoid
testfn(x::Vector{Float64}) = @ccall mylib.test(x::Ptr{Cdouble})::Cvoid

DRYness aside, this seems much more preferrable to me, taking advantage of the existing type system - especially because Julia just doesn’t have a global NULL value like C or Java has. Trying to make Nothing that NULL value seems to me like it’s missing the point of having nothing be its own type in the first place.

[projekter]

This is indeed probably the best possibility in the case of the small example. But now imagine a function that takes ten parameters, does some preprocessing, possibly postprocessing, etc… And one or two of the parameters are optional in the sense of null-passing. Relying on dispatch in this case is of course also possible, but would then involve lots of code duplication.
Let me give two concrete examples. One is a bug report that I just filed with Mosek, where they relied on the proposed behavior that nothing is implicitly converted to C_NULL (which I would argue is quite a natural assumption). The C-Interface looks like

MSKrescodee (MSKAPI MSK_appendacc) (
  MSKtask_t task,
  MSKint64t domidx,
  MSKint64t numafeidx,
  const MSKint64t * afeidxlist,
  const MSKrealt * b)

The length of the last two arrays is given by numafeidx, and b can be NULL if it is not present. The wrapper function does some preprocessing on the indices (convert one-based to zero-based) and retrieves the value for numafeidx from the length of the vectors. The Julia signature is

function appendacc(task::MSKtask,domidx::Int64,afeidxlist::Vector{Int64},b::Union{Nothing,Vector{Float64}})

This was a “small” example. I recently wrote a Julia wrapper for the Fortran function LANCELOT_simple from the GALAHAD library. The Fortran interface looks like this:

SUBROUTINE LANCELOT_simple( n, X, MY_FUN, fx, exit_code,                &
                                   MY_GRAD, MY_HESS,                           &
                                   BL, BU, VNAMES, CNAMES, neq, nin, CX, Y,    &
                                   iters, maxit, gradtol, feastol, print_level )
       INTEGER ( KIND = ip_ ), INTENT( IN )    :: n
       INTEGER ( KIND = ip_ ), INTENT( OUT )   :: exit_code
       REAL ( KIND = rp_ ), INTENT( INOUT ) :: X( : )
       REAL ( KIND = rp_ ), INTENT( OUT )   :: fx
       CHARACTER ( LEN = 10 ), OPTIONAL :: VNAMES( : ), CNAMES( : )
       INTEGER ( KIND = ip_ ), OPTIONAL :: maxit, print_level
       INTEGER ( KIND = ip_ ), OPTIONAL :: nin, neq, iters
       REAL ( KIND = rp_ ), OPTIONAL :: gradtol, feastol
       REAL ( KIND = rp_ ), OPTIONAL :: BL( : ), BU( : ), CX( : ), Y( : )
                            OPTIONAL :: MY_GRAD, MY_HESS

There are tons of optional arrays in here (to make it worse, assumed-shape, but that’s a different matter entirely). There are also a couple of callbacks and references which make preprocessing necessary. My Julia function signature looks like

function LANCELOT_simple(n::Integer, X::AbstractVector{Cdouble}, MY_FUN::Function; MY_GRAD::Union{Function,Missing}=missing,
    MY_HESS::Union{Function,Missing}=missing, BL::Union{<:AbstractVector{Cdouble},Nothing}=nothing,
    BU::Union{<:AbstractVector{Cdouble},Nothing}=nothing, neq::Integer=0, nin::Integer=0,
    CX::Union{<:AbstractVector{Cdouble},Nothing}=nothing, Y::Union{<:AbstractVector{Cdouble},Nothing}=nothing,
    maxit::Integer=1000, gradtol::Real=1e-5, feastol::Real=1e-5, print_level::Integer=1)

and indeed, there’s a lot of (unnecessary) clutter due to isnothing/ismissing checks in my ccall (I used nothing in the semantic sense that there is no boundary, and missing in the sense that while these things for sure exist, they need to be constructed manually or we are not interested in their outputs).

[aplavin]

@joa-quim

Need to make a copy and replace the (ghrrr) Nothings by NaNs.

Try SentinelArrays — they represent AbstractArray{Union{T, Nothing}} as an Array{T} with a sentinel value:

parent = zeros(UInt8, 10)
B = SentinelArray(parent, typemax(UInt8), nothing)

[joa-quim]

Thanks. I’ll give a look on how they do it. (I’d rather not to add another dependency for something that I don’t need very often).