I’ve been using Julia 1.3.1 in a kind of hacky way to integrate it into C#. I have no troubles passing C# arrays of say, float’s to Julia and changing them in place. Which is fantastic, but when I try to return an array of a C Type back to C# its in some sort structure I can’t exactly suss out.
What are the options here?
Is there a way to convert a Julia Array to a native C array before passing it back to C#?
Alternatively, how can I figure out the structure of a Julia array from say a pointer? I looked at it, and it’s not boxed in a way I’m really familiar with. There seems to be some number of bytes(64?) prior to the actual stored data?
Someone mentioned 1.4 will have native C layouts of array for isbits types? Can anyone confirm if that is true?
Any help here would be appreciated. I can provide code, but I think this is more of a need for information/where to find the information.
I’ll ignore all C# part since I assume all C<->C# is your own thing.
There’s not really much conversion. You can just pass an array to a C function that accept a pointer and the function will get the pointer to the data. You cannot resize this way, obviously, since a pointer does not carry that information and I’ll assume that you don’t need it. (you basically have to call into julia in order to resize arrays)
You can’t, if you are talking about the array to the data anyway. If you get the pointer from the method above (ccall(..., (Ptr{Cvoid}, ...), array, ...)) it’s just the pointer to the array and you can’t get any array metadata from there. You can get the pointer to jl_array_t as well but I don’t think you need to do that.
Yes/no, see above.
Who said that? No the array layout has always been C-like. And the pointer you get in the method above has always been giving you the pointer to the data.
2 Likes
Hm, I’m not sure I understand a lot of what you are saying - consider me not a soft-ware engineer because I’m not! But I’ll try to understand if you can bare with me a bit.
So basically what I am doing is using PackageCompiler to make my code into a *.DLL. So it’s living somewhere in C. Now I have a julia function called from C# that returns a pointer to a Julia array. For simplicity lets say the type and length of the array are known to C#. But, C# cannot marshal the array basically no matter what I try once it is created in Julia to a native type.
So I tried using an array of Int8’s for debugging. What I found was the pointer returned from Julia has some sort of header or is boxed or something, such that I had to read roughly 64 bytes to see the array data from the pointer location. For floats this probably is different…
From what you are saying it sounds like this is not true for Julia to C? So is this is something in C#?
Am I missing some key concept here? It’s a little underdocumented once you start going into these waters.
Correct. If you post your interface between Julia and c I I’ll be able to tell you if you are doing that correctly. I won’t be able to help with the c# part and I can at most guess in that case.
3 Likes
basically I have a function that makes a dumby array
a = randn(CFloat, 10)
then I return a pointer to that; there is no additional C code written. I am calling this function directly from C#. but if you can confirm that in C the pointer to a is not a problem then it’s something I am missing with C#.
As for documentation, the one for array is here (scroll down a bit to Array{T,N}). It says
and the address of the first element is passed.
which I think is about as direct as it can be…
2 Likes
Well, if you have just return pointer(a) then that’s invalid code since a may not be valid anymore. Julia, like C# IIUC, is a managed language and you must make sure the objects’ use is visible to the compiler. Doing this across language barrier is not trivial and the solution will depend on the exact call order.
1 Like
(But also note that this may or may not be the issue. I expect that if you have GC disabled and if that is exactly what you have, it should refturn the correct value given the current implementation of the compiler even though it’s still invalid code. You can disable the GC to see if you have anything more reasonable. If not, then something else is messing it up.)
well to make this more thought out, I did use an IDDict to store this array in julia so it would stay put, but maybe it’s not actually staying at the same address? Are there common patterns/techniques to do this stuff in an address safe way?
Yes C# is managed and it’s painful for me…
That’s fine. Storing a reference to a mutable type is safe enough. That’s basically the correct way. (In theory you do need to make sure the IDDict is either a global or is locally guaranteed rooted with GC.@preserve but I don’t think the current implemenation will give you too much trouble on that.)
2 Likes
I also recommend you to do the operation in C if you could and figure out if things are correct that way. It’ll be more easy to debug that way (at least I’ll be able to understand it more easily). If/once that’s fine, you can go to your C# friend to figure out if that part is done correctly.
And just a thought, julia has 1-based indexing but it does not mean the pointer you got in C has to be indexed using base 1. The pointer you get in C should be the “address to the first element” as shown in the document I linked above. I assume you got this right but I can also see why one could make a mistake here.
1 Like
I was hoping to avoid writing C because I am pretty bad at it, but it would be the proper tool for debugging.
This gave me some food for thought and a taste of sanity. I was debugging C# and Julia simultaneously that’s just a mistake in general. I did not know about the GC.@preserve call that sounds like a helpful thing to try.
I’ll also try going from Julia straight to C first. It does seem like this is either a GC issue, or a C# being … C# issue. In the absolute worst case I can standup ZMQ, but I’d love to do this as properly as possible - if possible - and it should be very possible. People do this all the time with C dll’s in C#, so I can figure it out.
Thank you, I’ll report back once I suss out the bug. May be a few days (need a break from the problem and the weekend is nice for that).
If you are already storing a in IdDict you should be fine.
You don’t need to/should’t go too complicated, just do a simple
float f(float *p)
{
return p[0];
}
and do your ccall(:f, Cfloat, (Ptr{Cfloat},), a) or ccall(:f, Cfloat, (Ptr{Cfloat},), pointer(a)) (making sure a is in the IdDict for the second case) and see if you get the answer you expect by comparing it to a[1].
I’m having a hard time picturing what you are doing. If you created a method in Julia that returns a Vector{Float64} then that is a “Julia” type. And the PackageCompiler would have to do some sort of conversion to a pointer of Float64s…but then that conversion would lose the length, since C really doesn’t have an array object.
Are you returning a Ptr{Float64}…because that might work. As long as your C# code knows the length of the array.
But in either case you run the risk of the memory freeing…since Julia allocated the array, Julia needs to free the array, so it needs someway of knowing it’s “still being used” by C#. Hopefully you have considered that.
It might be better to create the array in C# pass it to Julia to populate it. That way C# is responsible for the memory allocation (and freeing).
I did find:
https://stackoverflow.com/questions/3776485/marshal-c-int-array-to-c-sharp
But that would require some C code to call the Marshal.Copy to move the array into a C# object.
2 Likes
So this is the entire issue and its surprisingly easy to run in circles trying to figure it out. But first, I don’t think you can make a pointer to a Float64 in Julia. Anyways.
Yes I am passing the array length around, its clunky, but … what else can be done other then streaming? I have ensured that the information still stays in memory via a global IDDict, so thankfully it is still “somewhere” but “where” is now a good question…
Actually doing only “inplace” functions is possibly the best solution for me. It’s not ideal… But I don’t believe ideal is possible here. Ideal is C# can just marshal the Julia/C array types as if it knows all about them. In reality this becomes a hacky passing of several parameters (dimensions, type, and pointer).
But yes I’ve found that SO post and the general idea seems okay but when I get the array from Julia it’s not as anticipated even if I know the length… So it seems something else is at play…
@yuyichao
Ah that’s right! you can call C from inside julia with ccall that does make debugging way way easier… These Julia dev’s think of everything. I was going to build the dll, and cmake and yadda yadda. I could try this tomorrow morning :).
Yep everything works fine as you said it should.
Instead of using C, I just used Cxx and C++ (much easier then make files and stuff on windows). The global IdDict works, but - without it, often times the first run through the function it spit out random numbers. On later compilations, the memory address appeared stable(for a dumby application). So global IdDict is the way to go.
And yes I can confirm julia’s native type arrays, share the same memory layout as C/C++ I’m going to mark this as solved. But I am now struggling with PackageCompiler(1.1.0) and getting C# to integrate with it - but that’s solvable.