Calling a C function from within a Julia program

Hi! I’m using Julia 1.6.1 on a MacBook Pro and trying to use the MISR-Toolkit library of C functions within Julia programs. This library is maintained by NASA/JPL and available from GitHub at

I was able to access function MtkVersion from this library (which requires no argument and retrieves the version number of the Toolkit), so the library is correctly installed:

julia> mtklib = "/Applications/Mtk-1.5.0/lib/libMisrToolkit.so"
"/Applications/Mtk-1.5.0/lib/libMisrToolkit.so"

julia> v = ccall((:MtkVersion, mtklib), Cstring, ())
Cstring(0x000000014487a6f0)

julia> unsafe_string(v)
"1.5.0"

I then wanted to use the C function that converts an ISO 8601 date and time specification (“yyyy-mm-ddThh:mm:ssZ”) into a Julian day number, which features the following header:

MTKt_status MtkDateTimeToJulian(
  const char *datetime, /**< [IN] Date and time (ISO 8601) */
  double *jd /**< [OUT] Julian date */ )
{
    ...
}

However, the following call generates an error message (3) indicating an incorrect argument:

julia> dt = "2021-06-27T10:53:29Z"
"2021-06-27T10:53:29Z"

julia> status = ccall((:MtkDateTimeToJulian, mtklib), Cint, (AbstractString, Float64), dt, jd)
3

Reading through the documentation at

https://docs.julialang.org/en/v1/manual/calling-c-and-fortran-code/

and in particular the paragraph entitled “When to use T, Ptr{T} and Ref{T}”, I tried to refer to the output argument jd as a pointer type, but got the following error messages:

julia> status = ccall((:MtkDateTimeToJulian, mtklib), Cint, (AbstractString, Ptr{Float64}), dt, jd)
ERROR: MethodError: no method matching unsafe_convert(::Type{Ptr{Float64}}, ::Float64)
Closest candidates are:
  unsafe_convert(::Type{Ptr{Tv}}, ::SuiteSparse.CHOLMOD.Sparse{Tv}) where Tv at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/SuiteSparse/src/cholmod.jl:296
  unsafe_convert(::Type{Ptr{T}}, ::SharedArrays.SharedArray{T, N} where N) where T at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/SharedArrays/src/SharedArrays.jl:361
  unsafe_convert(::Type{Ptr{T}}, ::SharedArrays.SharedArray) where T at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/SharedArrays/src/SharedArrays.jl:362
  ...
Stacktrace:
 [1] top-level scope
   @ ./REPL[32]:1

I suspect the problem is specifically related to the characterization of the output argument jd. Thanks in advance for any help in figuring out how to successfully call that C function from a Julia program.

This is a pointer to double, so

julia> jd_ref = Ref{Float64}(0.0)

julia> status = ccall((:MtkDateTimeToJulian, mtklib), Cint, (AbstractString, Ptr{Float64}), dt, jd_ref)

julia> jd = jd_ref[]

@Gnimuc: Thanks for the quick answer. Here is the outcome:

julia> jd_ref = Ref{Float64}(0.0)
Base.RefValue{Float64}(0.0)

julia> status = ccall((:MtkDateTimeToJulian, mtklib), Cint, (AbstractString, Ptr{Float64}), dt, jd_ref)
3

julia> julia> jd = jd_ref[]
> (generic function with 1 method)

julia> jd = jd_ref[]
0.0

So the good point is that Julia does not complain about the ccall command, but I still get an error message (3, instead of 0), and the Julian day number is of course not 0 either. Any other thought?

AbstractString should be Cstring, I think.

@Gnimuc: I had tried this too before, but it does not seem to make any difference:

julia> status = ccall((:MtkDateTimeToJulian, mtklib), Cint, (String, Ptr{Float64}), dt, jd_ref)
3

Cstring and String are different types.

1 Like

@Gnimuc: Indeed, I just realized after pressing the “Reply” button… Now I get a 0 status (no error) and everything works fine:

julia> status = ccall((:MtkDateTimeToJulian, mtklib), Cint, (Cstring, Ptr{Float64}), dt, jd_ref)
0

julia> jd_ref[]
2.4593929538078704e6

Just one last question: is this syntax Ref{Float64}(0.0) the generic way to declare an output argument before calling a C function? Should I do this with all output variables in general? Thanks again for your prompt and efficient help.

is this syntax Ref{Float64}(0.0) the generic way to declare an output argument before calling a C function?

Yes. If you need to use the content(jd) of this Ref variable(jd_ref), it’s necessary to init it in this way(i.e. bind Ref(jd) to a variable name(jd_ref)). If not, you could use a tmp variable like:

jd = 0
ccall((:MtkDateTimeToJulian, mtklib), Cint, (Cstring, Ptr{Float64}), dt, Ref(jd))
# this time, the content of the tmp variable `Ref(jd)` will be populated by the C function,
# but there is no way to get that content on the Julia side.
# jd is still 0. 

BTW, you might be interested in CBinding.jl and Clang.jl which are two packages in the Julia ecosystem that can make life easier when calling C libs in Julia.

3 Likes

@Gnimuc: Great! I’ll look at those two packages. Thanks again: your help is greatly appreciated.

3 Likes

@Gnimuc: Sorry to bother you again, but I still have additional questions about using that C library… When I try to use the C function that converts a Float64 Julian day number into a string instead, I always get a MethodError: no method matching unsafe_convert... error, no matter what I use to initially declare the output variable and to define the type of argument in the ccall itself. For reference, the header of that other C function is as follows:

MTKt_status MtkJulianToDateTime(
  double jd, /**< [IN] Julian date */
  char datetime[MTKd_DATETIME_LEN] /**< [OUT] Date and time (ISO 8601) */ )
{
...
}

where the identifier MTKd_DATETIME_LEN is initialized to the constant 28 in an included .h file. Now, using a syntax similar to the cases above:

julia> mtklib = "/Applications/Mtk-1.5.0/lib/libMisrToolkit.so"
"/Applications/Mtk-1.5.0/lib/libMisrToolkit.so"

julia> jd = 2.4593929538078704e6
2.4593929538078704e6

julia> julia> dt_ref = Ref{String}("")
Base.RefValue{String}("")

julia> status = ccall((:MtkJulianToDateTime, mtklib), Cint, (Float64, Cstring), jd, dt_ref)
ERROR: MethodError: Cannot `convert` an object of type Base.RefValue{String} to an object of type Cstring
Closest candidates are:
  convert(::Type{Cstring}, ::Union{Ptr{Int8}, Ptr{Nothing}, Ptr{UInt8}}) at c.jl:167
  convert(::Type{T}, ::T) where T at essentials.jl:205
Stacktrace:
 [1] cconvert(T::Type, x::Base.RefValue{String})
   @ Base ./essentials.jl:396
 [2] top-level scope
   @ ./REPL[5]:1

or

julia> status = ccall((:MtkJulianToDateTime, mtklib), Cint, (Float64, String), jd, dt_ref)
ERROR: MethodError: Cannot `convert` an object of type Base.RefValue{String} to an object of type String
Closest candidates are:
  convert(::Type{String}, ::String) at essentials.jl:210
  convert(::Type{T}, ::T) where T<:AbstractString at strings/basic.jl:231
  convert(::Type{T}, ::AbstractString) where T<:AbstractString at strings/basic.jl:232
  ...
Stacktrace:
 [1] cconvert(T::Type, x::Base.RefValue{String})
   @ Base ./essentials.jl:396
 [2] top-level scope
   @ ./REPL[6]:1

How should output character strings be defined BEFORE the ccall command, as well as WITHIN the arguments of that command?

Lastly, I have already noted that other C functions in that library declare output arguments as char *arg or char **arg (instead of simply char arg): how should those arguments be referred to, initially and within the ccall?

ccall((:MtkJulianToDateTime, mtklib), Cint, (Float64, Cstring), jd, dt_ref)

The ccall macro implicitly calls Base.cconvert for converting the types of argvalues(jd, dt_ref) to corresponding argtypes(Float64, Cstring).

The error ERROR: MethodError: Cannot convert an object of type... means Julia fails to find a pre-defined cconvert method. In the case above, it’s Base.cconvert(Cstring, Ref{String}("")). Now we know the reason, but how to fix this?

IMO, the first step is to check whether argtypes are matched to those types used in C. Here is the function prototype of MtkJulianToDateTime:

MTKt_status MtkJulianToDateTime( double jd, char datetime[MTKd_DATETIME_LEN] );

The type of datetime is a constant char array, but it’s actually not. When it is used as the argument type in a function prototype, it actually decays to a char pointer char *, so the corresponding Julia type could be Cstring if it’s NUL-terminated or Ptr{Cuchar} if not. In practice, it’s OK to use Ptr{Cuchar} if we don’t know which is the case. You could refer to the Type Correspondences section for further details.

Another thing the C function prototype tells us is the size(MTKd_DATETIME_LEN) of datetime, so the C function MtkJulianToDateTime expects a pointer to a container that can be filled with MTKd_DATETIME_LEN chars. One way to do this in Julia is binding dt_ref to a Vector like dt_ref = Vector{Cuchar}(undef, MTKd_DATETIME_LEN). A Ref with NTuple{MTKd_DATETIME_LEN, Cuchar} as its field type can also work, but it’s not easy to work with this type in Julia.

The final version:

jd = 2.4593929538078704e6
dt_chars = Vector{Cuchar}(undef, MTKd_DATETIME_LEN)
status = ccall((:MtkJulianToDateTime, mtklib), Cint, (Float64, Ptr{Cuchar}), jd, dt_chars)
dt = String(dt_chars)

For char **arg, it’s a container that is populated with char *s, so if it is an input argument, Ptr{UInt8}[pointer("strings1"), pointer("strings2")] with type Vector{Ptr{UInt8}} would be fine.

@Gnimuc: Thanks for your continuing support. Here is the outcome:

julia> dt_chars = Vector{Cuchar}(undef, 28)
28-element Vector{UInt8}:
 0xe0
 0x68
 0x3c
 0x1b
 0x01
 0x00
 0x00
 0x00
 0x30
 0x53
    ⋮
 0x12
 0x01
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00

julia> status = ccall((:MtkJulianToDateTime, mtklib), Cint, (Float64, Ptr{Cuchar}), jd, dt_chars)
0

julia> dt = String(dt_chars)
"2021-06-27T10:53:29Z\0\0\0\0\0\0\0\0"

which is the correct outcome. Is it possible to generalize this? I will generally not know in advance how long the output string will be, so what should I specify instead of 28 in this particular case?

There are several ways to do this: read null terminated string from byte vector in julia - Stack Overflow

I added the generator script and generated the bindings using Clang.jl here:

You could pick up one that matches your local platform or just copy-paste the code you need.

Note that, Cchar is used everywhere instead of Cuchar. When converting a Vector{Cchar} to a String, you need to convert Vector{Cchar} to Vector{Cuchar} firstly. But I guess something like this can also work:

jd = 2.4593929538078704e6
dt_chars = Vector{Cuchar}(undef, MTKd_DATETIME_LEN)  # let `Base.cconvert` do the conversion for us   
status = ccall((:MtkJulianToDateTime, mtklib), Cint, (Float64, Ptr{Cchar}), jd, dt_chars)
dt = String(dt_chars)
1 Like

@Gnimuc: Wow! How can you generate so much code for various platforms in so little time? Amazing…

I’ll definitely explore this solution, and thanks for the link to the very useful Stack Overflow page showing how to remove the null characters at the end of a string.

In the meantime, my earlier question was somewhat different: when I call an arbitrary C function that outputs a string variable, I generally do not know what length to expect. For instance, a function may return the path (directory) and filename of a data file, which might be 150 characters long, while another could return a paragraph of text. Should I declare the output variable as dt_chars = Vector{Cuchar}(undef, 10000) just to be on the safe side (but that seems wasteful of memory), or is there a way to tell Julia and/or C that the length of the output variable will only be known at run time?

We don’t know unless the C API tells us.

In the case of this library, it uses constant array types in the function prototype instead of pointers. I think the purpose here is to use those fixed-size array types to inform users that the size of the array can be safely set to certain compiler time constant macro values.

Different C libraries have different API styles for this problem. For example, some libraries may add another variable to the function prototype and users need to query the size of the container in advance.