Ccall works on global scope but not within function

General Usage
,
1

I’m trying to create bindings to a C library. I generated some Julia code using Clang.jl, and I was able to successfully make some function calls reproducing some C-code program. It’s not very reliable, though. Right now I have a piece of code that seems to work often if I run it in the global scope, but if I create a function with that code inside, and call that function instead, it does not work. The only kind of problem I imagine might happen would be related to invalid memory access, but I never get a segfault or something like that, it only hangs, or the function fails to work as it should (detecting some hardware). What’s the best way to even approach debugging something like that?

Something I suspect I could be doing wrong is that the function expects to receive an array where it can fill in some data. It does not allocate this memory, it only writes there. I think in C this is supposed to be just some array of a struct, and the function returns the number of positions used.

I’m creating a Vector{MyStruct} in Julia, and passing that to ccall. Could this be a problem? Are there situations where one should call malloc from within Julia to allocate memory instead?

2

Could be a rooting problem, where your Julia array is getting garbage-collected before the C routine accesses it?

In particular, given a v::Vector{MyStruct}, you should be doing ccall with a Ptr{MyStruct} argument and passing v, not pointer(v) (which is not rooted unless you explicitly use GC.@preserve).

Hard to say more without example code.

3

This looks like a GC issue. You need to take a closer look to the C API to guarantee that the lifetime of the resources allocated on the Julia side extends that on the C side. Sometimes it’s evitable to use global variables and reference counting.

This is probably due to you didn’t correctly use the C API to release the resources or reset the device.

For example, in one Julia session, you created a handle devicePtr to detect the hardware, then somehow you leaked the devicePtr. Then, the next time you try to detect the hardware, it just hangs.

To debug this kinda problem, you may need to do a “hard reset” by plug-unplug the device.

1 Like
4

Thanks, I did make the mistake with pointer before, but I fixed that after I started using Clang.jl to generate the bindings.

Here’s a slightly involved example that supposedly reproduces how this library works. I’m not sure I reproduced all the bugs, but it does illustrate one of the issues I’m having. I’m using this way to transform NTuple{N, UInt8} to strings, and that fails when I move the code to within a function. There seems to be some gotcha related to references. What’s the proper way to encapsulate that code, creating a handy getstr or something similar?

Here’s my C code

#include<stdint.h>
#include<stdio.h>

struct MyStruct {
  char name[16];
  uint8_t ipaddress[4];
  double xyz;
};


int writedata(struct MyStruct * buffer, int * buffersize) {
  *buffersize = *buffersize >> 1;
  for (int i = 0 ; i< *buffersize; i++) {
    int j = 0;
    for (; j<3; j++)
      buffer[i].name[j] = 'a' + j + (j+1)*i;
    buffer[i].name[j] = 0;
    buffer[i].ipaddress[0] = 192;
    buffer[i].ipaddress[1] = 168;
    buffer[i].ipaddress[2] = 0;
    buffer[i].ipaddress[3] = 128 + i;
    buffer[i].xyz = i*i*3.1415;
  }
  return (*buffersize > 0) ? 0 : 1;
}

int main(int argc, char** argv) {
  struct MyStruct mydata[8];
  int buffersize = 8;
  writedata(mydata, &buffersize);
  for (int i; i < buffersize; i++) {
    printf("%s - %d.%d.%d.%d - %f\n",
	   mydata[i].name,
	   mydata[i].ipaddress[0], mydata[i].ipaddress[1], mydata[i].ipaddress[2], mydata[i].ipaddress[3],
	   mydata[i].xyz);
  }
}

/* gcc testapi.c -o testapi.so -shared -fPIC */

And the Julia code

struct MyStruct
    name ::NTuple{16, UInt8}
    ipaddress ::NTuple{4, UInt8}
    xyz ::Float64
end

# getstr(x) = unsafe_string(Ptr{UInt8}(pointer_from_objref(Ref(x))))
getstr(x) = unsafe_string(Ptr{UInt8}(pointer_from_objref(x)))
# getstr(x) = Ptr{UInt8}(pointer_from_objref(x))
# getstr(x) = x

function api_writedata(buffer, buffersize)
    ccall(
        (:writedata, "testapi.so"),
        Cint,
        (Ptr{MyStruct}, Ptr{UInt32}),
        buffer, buffersize
    )
end 

mydata = Vector{MyStruct}(undef, 8)
buffersize = Ref(0x00000008)

api_writedata(mydata, buffersize)
for data in mydata[1:buffersize[]]
    # println("$(data.name) - $(join(data.ipaddress,'.')) - $(data.xyz)")
    println("$(getstr(Ref(data.name))) - $(join(data.ipaddress,'.')) - $(data.xyz)")
end
println("***")

function process()
    mydata = Vector{MyStruct}(undef, 8)
    buffersize = Ref(0x00000008)

    api_writedata(mydata, buffersize)
    for data in mydata[1:buffersize[]]
        # println("$(data.name) - $(join(data.ipaddress,'.')) - $(data.xyz)")
        println("$(getstr(Ref(data.name))) - $(join(data.ipaddress,'.')) - $(data.xyz)")
    end
    
end

process()

The resulting output:

abc - 192.168.0.128 - 0.0
bdf - 192.168.0.129 - 3.1415
cfi - 192.168.0.130 - 12.566
dhl - 192.168.0.131 - 28.273500000000002
***
 - 192.168.0.128 - 0.0
 - 192.168.0.129 - 3.1415
 - 192.168.0.130 - 12.566
 - 192.168.0.131 - 28.273500000000002
5

This isn’t safe, because x isn’t rooted — it could get garbage-collected while unsafe_string is running. You need getstr(x) = GC.@preserve x unsafe_string(Ptr{UInt8}(pointer_from_objref(x)))

However, I would tend to avoid pointer_from_objref entirely, and just use something like String(collect(data.name)).

6

Great, thanks! I think the point of unsafe_string is that it expects zero-terminated data. I suppose an alternative could be String(collect(Iterators.takewhile(x->x!= '\0', Char.(aa))))?

7

Don’t convert to Char, which is a 32-bit quantity (representing a Unicode codepoint) — you want to just look at bytes for a NUL-terminated C string.

You could do String(collect(Iterators.takewhile(!iszero, data.name))).

8

Alright, but creating a String will still implicitly convert it al to Char, right?

9

Nope. A String is stored as a UTF-8 encoded sequence of bytes, which is not the same as an array of Char. (When you iterate over a String, it decodes these bytes into a sequence of Char values for you, but that’s not the format in which it is stored.)

1 Like
10

TIL! Still, converting something to Char and then into a String, or give the array straight to the String constructor, we should expect the UTF-8 encoding process to finish with the same result, no? Sorry for prolonging this topic, you’re saying something very insightful and I’m trying to tap into this sea of knowledge :slight_smile:

11

OK everyone, I think I got it working now. Other than the string issue, it seems the problem I had was only related to how the C function works. There’s a second array I was treating as an output, but apparently it’s an input, so I was feeding it garbage and causing havoc. I just followed the C examples, with a C_NULL instead of an array, and now the bindings seem reliable. Nice start for a Monday. Thanks so much for the help!

12

That will make a redundant copy of the created array in the call to String. You need to allocate the array with Base.StringVector(n) (which allocates space for a null terminator) to ensure that a no-copy version of String is used.

13

Yes, but it’s not clear that the use-case here is performance-sensitive enough for the extra code (and reliance on undocumented internals) to be worth the trouble.

14

Yes, there is a String(::AbstractVector{Char}) constructor (which seems to be undocumented?) that will construct the string by UTF-8 encoding the characters.

1 Like