Direct manipulation of Julia arrays in C code

[jose]

Hi community!

This is my first question here, so if I’m doing something wrong, please advice me.

Well, in order to make this question I have to briefly introduce my case. I’m developing a NLP library in C++. It’s nothing very original, fancy or revolutionary, but more like an amateur-made-to-learn-copy-a-lot-of-work-of-other-libraries project. Well, a NLP library without a dynamic interface is a little worthless, so I thought about doing it in Julia, a language I have no experience (so I can learn Julia and NLP with the same shot).

For now I’m using Eigen as a primary backend to the CPU part of the library in the C++ side. But I don’t want to wrap and expose Eigen matrixes or tensors to Julia, I want the data to be owned by Julia’s arrays, so users can have all their nice and familiar syntax. So, what I want is to manipulate the Julia arrays directly through the C api. I have looked at the documentation, and that seems totally possible.

What makes me doubt is how to pass from C to Julia:

→ I have seen that I can exclude from Julia memory ownership a Julia array created from C if I pass a 0 to the last boolean argument of jl_ptr_to_array_Xd. Is there a way to switch off the memory ownership of an array that is already owned by the Julia gc system (for example, created with jl_alloc_array_Xd in C or directly in Julia)?

→ How can I construct in Julia code an array that has been allocated directly in C (I imagine I have to return a jl_array_t* pointer from a ccall, but what do I do with this pointer to have a complete Julia array)? Does this array to have the gc option activated? Ccall where only pointers are passed are costly?

→ When I pass a Julia Array to a ccall, the C code receives a jl_array_t* pointer or a pointer to the underlying data structure (similar to calling jl_array_data)?

Thank you in advance for any help

[yuyichao]

@jose

Is there a way to switch off the memory ownership of an array that is already owned by the Julia gc system (for example, created with jl_alloc_array_Xd in C or directly in Julia)?

No.

@jose

I imagine I have to return a jl_array_t* pointer from a ccall, but what do I do with this pointer to have a complete Julia array

Use the array type as the return type in ccall. The call won’t have GC issue assuming the C code itself doesn’t. The call also won’t have any performance issue other than any performance impact of allocating an array in the first place.

@jose

When I pass a Julia Array to a ccall, the C code receives a jl_array_t* pointer or a pointer to the underlying data structure (similar to calling jl_array_data)?

Either. You can pass in an object pointer with Any or Ref{<actual_type>} as parameter type like you can do with any other julia object. Using the actual array type here also work as a special case for arrays. You can pass the pointer with pointer or ref of element type as parameter type. Please read the doc of calling C/fortran function on how to pass parameters to C.

[jose]

@yuyichao ok, thank you very much, you have answered all my doubts, I think I can build from here.

[fasiha]

@yuyichao

Use the array type as the return type in ccall. The call won’t have GC issue assuming the C code itself doesn’t.

Could you elaborate on this @yuyichao? If a C function malloc’s some memory and returns a pointer, and then ccall returns that pointer as a Julia array type, will Julia take over ownership of the memory? I.e., will Julia free the underlying pointer (which was originally allocated in C) when the array goes out of scope?

Or does this require allocating the array in C using Julia-specific functions (and not malloc)?

[yuyichao]

@fasiha

If a C function malloc 's some memory and returns a pointer, and then ccall returns that pointer as a Julia array type, will Julia take over ownership of the memory?

That’s illegal.

@fasiha

Or does this require allocating the array in C using Julia-specific functions (and not malloc )?

Yes. (That’s the question this is answering, which is why I quoted the question…)

The convertion from pointer to Array is always explicit, in both julia and C.