Undefined behaviour of ccall

General Usage
1

I have been using my julia wrapper for some c-library called fasterac (for use with digital electronics (https://faster.in2p3.fr/). It has been working fine, and I processed many TB of data with it. With Julia 1.11 (and as I tested, also 1.10) I came across a weird undefined behavior. It seems that library call (faster_file_reader_open) cannot access file, and instead of file pointer returns null. This happens randomly.

Here is an example in plain C

#include <stdio.h>
#include <stdlib.h>

#include "fasterac/fasterac.h"
#include "fasterac/utils.h"

int inspect(char* filename) {
    faster_file_reader_p reader; 
    faster_data_p data;

    reader = faster_file_reader_open (filename);  
    if (reader == NULL) {
        return -1;
    }

    int n = 0;
    while ((data = faster_file_reader_next (reader)) != NULL) {  
        n = n + 1;                                                
    }                                                         
    faster_file_reader_close(reader);                      
    return n;
}

int main (int argc, char** argv) {
  int good = 0;
  int bad = 0;
  for (int i = 0; i < 10000; i++) {
    int n = inspect(argv[1]);
    if (n > 0)
        good += 1;
    else
        bad += 1;
  }
  printf("Good: %d, Bad: %d \n", good, bad);
  return 0;
}

And an equivalent minimal example in Julia.

using Printf

function file_reader_open(filename::String)
    datafile = Vector{UInt8}(filename)
    reader = ccall((:faster_file_reader_open, "libfasterac.so"),
                   Ptr{Cvoid}, (Ptr{UInt8}, ), 
                   datafile)
    return reader
end

function file_reader_close(reader)
    void = ccall((:faster_file_reader_close, "libfasterac.so"),
                   Cvoid, (Ptr{Cvoid}, ), 
                   reader)
    return void
end

function file_reader_next(reader)
    data = ccall((:faster_file_reader_next, "libfasterac.so"),
                 Ptr{Cvoid}, (Ptr{Cvoid}, ), 
                   reader)
end

function inspect(filename)
    reader = file_reader_open(filename)
    if reader == C_NULL
        return -1
    end
    n = 0
    while ((data = file_reader_next(reader)) !== C_NULL)
            n += 1
    end
    file_reader_close(reader)
    return n
end

function main()
    bad = 0
    good = 0
    for i in 1:10000
        n = inspect("test_data.fast")
        if n < 0
            bad += 1
        else
            good += 1
        end
    end

    println("Good: $good, Bad: $bad")
end

While C code runs without any problem, Julia’s version has success rate about 95%. With a larger input file, the success rate goes down (to about 40%). Problem is in the line

    reader = ccall((:faster_file_reader_open, "libfasterac.so"),
                   Ptr{Cvoid}, (Ptr{UInt8}, ), 
                   datafile)

which returns 0 (NULL) interpreted as “error opening file”. A non-zero value is a proper pointer to the file.

In Julia 1.10 I have similar results. In Julia 1.9 everything works fine (this, and earlier versions I have been using with this wrapper extensively). What may have cause that problem?

2 Likes
2

I bet what’s happening here is that your faster_file_reader_open needs a null-terminated string, but passing a Ptr{UInt8} from a vector of UInt8s doesn’t necessarily guarantee one. Instead, just simplify this to pass the string itself as a Cstring:

function file_reader_open(filename::String)
    reader = ccall((:faster_file_reader_open, "libfasterac.so"),
                   Ptr{Cvoid}, (Cstring, ), 
                   filename)
    return reader
end
2 Likes
3

Also, don’t you need GC.@preserve for the array argument that’s being passed to the ccall?

4

No, GC.@preserve isn’t needed if you’re explicitly passing the object itself directly to ccall.

It’s only required if you’re working with raw pointers or the like that implicitly reference objects in a way Julia can’t track.

3 Likes
5

Oh wow, I had no idea about that. This’d be good to document somewhere.

6

Of course, that was it. And now that’s obvious. Was Julia before 1.10 aligning strings in memory with extra null at the end? Interestingly, that bug never came to the light until recently.

7

IIRC, String did (and still does) have a trailing NULL, but converting the string to a Vector{UInt8} doesn’t copy that into the new vector. It might be that the conversion previously aliased the memory of the String, which probably doesn’t happen anymore since Memory has become the backing store of Vector. That aliasing would then just happen to also be NULL-terminated.

1 Like