Passing a C struct into Julia

[Vangelis]

Dear experts,
I am trying to find a way to pass a C struct into a Julia model.
Please note that the C struct has at least one array. For example:
on the C side:

typedef struct
{
   int* the_integers;
   int* the_integers_2;
   double* the_doubles;
}the_numbers;

and on Julia side:

julia> struct the_numbers
       the_integers::Array{Cint,1}
       the_integers_2::Array{Cint,1}
       the_doubles::Array{Cdouble,1}
       end

So, the struct is defined in both sides (C and Julia) but I cannot find which API should be used.
The intention is to pass the C struct into Julia.
Could you please help me?

[anon37204545]

Looks like something similar (although more complex) was already answered in Dealing with complex C structure - #2 by tk3369. Does that help you?

[Vangelis]

Hi Zlatan,
thanks for your reply.
Unfortunately the article refers to the use of ccall while I am trying to do the other way around.
Assume that I have a C code which calls a Julia module and provides a struct as argument for this model.

Would that be possible?

[anon37204545]

So you want to embed Julia inside C? There is a section for that, too: High-level embedding.

[Vangelis]

Yes I have been through that but it doesn’t say anything about passing structs.
It explains nicely the arrays but not the structs

[yuyichao]

@Vangelis

Unfortunately the article refers to the use of ccall while I am trying to do the other way around.

There’s no “other way around”. You are talking about defining compatible structure, and there’s no direction on that. (Unless you are talking about defining the C struct to be compatible with the julia one). That said, the linked thread is unrelated.

Document is here https://docs.julialang.org/en/latest/manual/calling-c-and-fortran-code/#Struct-Type-Correspondences-1

Your C pointers are, well, pointers and not arrays and not julia arrays. And you do need to make sure the memory is managed correctly.

[Vangelis]

Hi Yuyichao,
providing the same definition in Julia is not a problem.
Looking into the link you provided to me (thanks) it seems that it is pretty straightforward to define the structs.
Although, calling from C, it is accessing a different memory.

Is there an example on how to pass a pointer from C to Julia?
Actually, how does it have to be defined in Julia?

---

Thanks in advance!

[yuyichao]

@Vangelis

providing the same definition in Julia is not a problem.

Well, that is the problem in that your definition of struct the_numbers in julia is incorrect. You need,

@yuyichao

Your C pointers are, well, pointers and not arrays and not julia arrays. And you do need to make sure the memory is managed correctly.

[Vangelis]

Hi Yuyichao,
changing the the_numbers to have NTuple instead of Array, it worked.
But in order to make it work, I made my module as ccallable.
And by having it as ccallable, we are “losing” the multiple dispatch feature of Julia, right?

[yuyichao]

@Vangelis

But in order to make it work, I made my module as ccallable.

No that doesn’t make sense. As in it doesn’t make sense you need this to share data.
If you need this to call the julia function from C and that’ totally fine and you aren’t losing anything sincne the caller in C doesn’t have any of that anyway.

@Vangelis

changing the the_numbers to have NTuple instead of Array, it worked.

That isn’t right unless you’ve also changed the C code. As I said, you have a pointer in C, not an array.

[Vangelis]

Hi Yuyichao,
thanks for your comments.
If I may post my code here so that you can advise me better on what I should change.
For simplicity reasons, I reduced the size of the arrays to 3.
Here comes my module:

module MyModule

struct the_numbers
       the_integers::NTuple{3,Cint}
       the_integers_2::NTuple{3, Cint}
       the_doubles::NTuple{3,Cdouble}
end

struct test_struct_1
       a::Cint
       my_nums::the_numbers
end

Base.@ccallable function print_struct(my_struct::test_struct_1)::Cdouble
	println(my_struct.a)
	println(my_struct.my_nums.the_doubles[1])
	return my_struct.my_nums.the_doubles[1] + my_struct.my_nums.the_doubles[2];

end

if abspath(PROGRAM_FILE) == @__FILE__
	my_numbers = the_numbers(ntuple(i -> i*1 ,3),ntuple(i -> (i+4)*2,3),ntuple(i -> (i + 7.7)*3,3))

	my_n_struct = test_struct_1(3, my_numbers)

	res = print_struct(my_n_struct)
	println("Res is: ",  res)
end

end

Then I create the precompile by doing:
julia --startup-file=no --trace-compile=MyModule_precompile.jl MyModule.jl

Now it is time to create the custom system image. First I create the file:

Base.init_depot_path()
Base.init_load_path()

@eval Module() begin
    Base.include(@__MODULE__, "MyModule.jl")
    for (pkgid, mod) in Base.loaded_modules
        if !(pkgid.name in ("Main", "Core", "Base"))
            eval(@__MODULE__, :(const $(Symbol(mod)) = $mod))
        end
    end
    for statement in readlines("MyModule_precompile.jl")
        try
            Base.include_string(@__MODULE__, statement)
        catch
            # See julia issue #28808
            Core.println("failed to compile statement: ", statement)
        end
    end
end # module

empty!(LOAD_PATH)
empty!(DEPOT_PATH)

and then:
julia --startup-file=no --cpu-target="generic" --output-o sys.o -J"lib/julia/sys.so" create_sysimage.jl
and
gcc -shared -o libsys.so -Wl,--whole-archive sys.o -Wl,--no-whole-archive -L"lib/" -ljulia

then I wrote a small C++ code:

#include <iostream>
#include <string>


#include <julia.h>

JULIA_DEFINE_FAST_TLS()
#ifdef __cplusplus
extern "C" {

#endif


typedef struct
{
   int the_integers[3];
   int the_integers_2[3];
   double the_doubles[3];
}the_numbers;

typedef struct 
{
	int a;
	the_numbers my_nums;
}test_struct_1;

extern double print_struct(test_struct_1 my_struct);



int main()
{

	// jl_init();
	jl_options.image_file = JULIAC_PROGRAM_LIBNAME;
    julia_init(JL_IMAGE_JULIA_HOME);

	test_struct_1 my_struct;
	my_struct.a = 4;

	my_struct.my_nums.the_integers[0] = 1;
	my_struct.my_nums.the_integers[1] = 2;
	my_struct.my_nums.the_integers[2] = 3;

	my_struct.my_nums.the_integers_2[0] = 4;
	my_struct.my_nums.the_integers_2[1] = 5;
	my_struct.my_nums.the_integers_2[2] = 6;

	my_struct.my_nums.the_doubles[0] = 7.7;
	my_struct.my_nums.the_doubles[1] = 8.8;
	my_struct.my_nums.the_doubles[2] = 9.9;

	double res = print_struct(my_struct);

	std::cout << "From C++: " << res << std::endl;

	jl_atexit_hook(0);

return 0;
}

#ifdef __cplusplus
}
#endif

and compile as:

g++ -DJULIAC_PROGRAM_LIBNAME=\"libsys.so\" \
        -o call_MyModule call_MyModule.cpp \
        -L'./' -lsys -O2 -fPIE -I'include/julia' \
        -L'lib' -ljulia lib/julia/libstdc++.so.6

[yuyichao]

@Vangelis

For simplicity reasons, I reduced the size of the arrays to 3.

@Vangelis

typedef struct
{
   int the_integers[3];
   int the_integers_2[3];
   double the_doubles[3];
}the_numbers;

This is not a simplification from what you had in,

@Vangelis

typedef struct
{
   int* the_integers;
   int* the_integers_2;
   double* the_doubles;
}the_numbers;

These are completely incompatible c code, i.e.

@yuyichao

That isn’t right unless you’ve also changed the C code. As I said, you have a pointer in C, not an array.

I can’t tell ou what to do here since only you know what you really want to do. Which version of the c code works for your real use case? Are the sizes always the same?

And for the other point I was talking about,

@yuyichao

As in it doesn’t make sense you need this to share data.
If you need this to call the julia function from C and that’ totally fine and you aren’t losing anything sincne the caller in C doesn’t have any of that anyway.

Given how you are currently calling your function in c, you certainly needs ccallable. However, that is unrelated to anything you mentioned in the original post - you don’t need it to pass your struct. And you also only lost multiple dispatch due to wanting to call the function from c and not because you use ccallable.

[Vangelis]

As I mentioned, the ccallable was the only way for me to make it work.
Assuming that what I posted lately, is the usecase, could you please describe another way where you don’t go through the ccallable?

[yuyichao]

Sure, but I’m saying that suddenly mentioning a solution to a completely different problem is very misleading, especially for other readers that doesn’t know much about these so it needs clarification at least for future readers. It’s otherwise perfectly fine if this is the only way you can get it work.

As for alternative way for calling Julia function from C. There are a lot of them. You can either call Julia code using c calling convention or with generic Julia calling convention.

For c calling convention you just need a function pointer which you can get via @cfunction in Julia. You are now facing the problem of passing a value from Julia to c.

For Julia calling convention, you just need to get the function object, and then you can call it with a variety of methods like jl_call* after allocating Julia objects for the arguments.

Both approaches requires accessing some Julia value from c code and there are also many ways to do it. It can be initiated from Julia side, by letting Julia code call a c function and pass in whatever you want as an argument. It can be done purely from c code without running any Julia code by accessing Julia global variables. It can be somewhere in between by using jl eval string to get some julia object from c, and in face all the Julia code mentioned above can either be written in the Julia file, or as string in c file.

[Vangelis]

thanks for the swift answer Yuyichao!

You nicely mentioned:

@yuyichao

For Julia calling convention, you just need to get the function object, and then you can call it with a variety of methods like jl_call* after allocating Julia objects for the arguments.

how would you do the call in order to pass a struct?
Would the jl_new_struct be useful here?

[yuyichao]

You don’t need to do the call any differently, you need “allocating Julia object for the argument”.

There are also basically two ways to do this following the generic Julia abi vs using the c abi. jl_new_struct can be used here, as a generic Julia abi, but,

1. Much like any generic call like this, it’s inefficient.
2. You still need to allocate the field so at some point you need to get a jl_value_t* without using this function.

[Vangelis]

Finally,
it worked with the jl_new_struct.
I am pasting here code.

The code in Julia is:

module MyModule

struct the_numbers
       the_integers::Vector{Cint}
       the_integers_2::Vector{Cint}
       the_doubles::Vector{Cdouble}
end

struct test_struct_1
       a::Cint
       my_nums::the_numbers
end

function print_nums(my_struct::the_numbers)::Cdouble
	return my_struct.the_doubles[1] + my_struct.the_doubles[2];
end

function quick_sum_struct(my_struct::test_struct_1)::Cdouble
	return my_struct.my_nums.the_doubles[1] + my_struct.my_nums.the_doubles[2];

end

if abspath(PROGRAM_FILE) == @__FILE__
	my_numbers = the_numbers([1,2,3],[4,5,6],[7.7,8.8,9.9])

	res_0 = print_nums(my_numbers)
	println("Res_0 is: ",  res_0)

	my_n_struct = test_struct_1(3, my_numbers)
	
	res = quick_sum_struct(my_n_struct)
	println("Res is: ",  res)
end

end

and the C++ code:

#include <iostream>
#include <string>

#include <unistd.h>

#include <julia.h>

JULIA_DEFINE_FAST_TLS()

typedef struct
{
   int* the_integers;
   int* the_integers_2;
   double* the_doubles;
}the_numbers;

typedef struct 
{
	int a;
	the_numbers my_nums;
}test_struct_1;

void fill_my_struct(test_struct_1* my_struct)
{
	

	const size_t array_size=5000*5000;
        my_struct->a = array_size;
	my_struct->my_nums.the_integers = new int[array_size];
	my_struct->my_nums.the_integers_2 = new int[array_size];
	my_struct->my_nums.the_doubles = new double[array_size];

	for(size_t i=0; i<array_size; i++)
	{
		my_struct->my_nums.the_integers[i] = 2000;
		my_struct->my_nums.the_integers_2[i] = 4000;
		my_struct->my_nums.the_doubles[i] = 7.7;
	}
}

int main()
{

	jl_options.image_file = JULIAC_PROGRAM_LIBNAME;
    julia_init(JL_IMAGE_JULIA_HOME);

	test_struct_1 my_struct;

	fill_my_struct(&my_struct);

	// box all C variables
	
	jl_array_t *jl_my_struct_my_nums_the_integers = NULL;
	jl_array_t *jl_my_struct_my_nums_the_integers_2 = NULL;
	jl_array_t *jl_my_struct_my_nums_the_doubles = NULL;
	jl_value_t* jl_my_struct_a = NULL;

	JL_GC_PUSH4(jl_my_struct_a, 
		       jl_my_struct_my_nums_the_integers,
		       jl_my_struct_my_nums_the_integers_2,
		       jl_my_struct_my_nums_the_doubles);

	
	jl_my_struct_my_nums_the_integers = jl_ptr_to_array_1d(jl_apply_array_type((jl_value_t*)jl_int64_type, 1),
		                                                               my_struct.my_nums.the_integers,
		                                                               my_struct.a,
		                                                               0);
	jl_my_struct_my_nums_the_integers_2 = jl_ptr_to_array_1d(jl_apply_array_type((jl_value_t*)jl_int64_type, 1),
		                                                               my_struct.my_nums.the_integers_2,
		                                                               my_struct.a,
		                                                               0);
	jl_my_struct_my_nums_the_doubles = jl_ptr_to_array_1d(jl_apply_array_type((jl_value_t*)jl_float64_type, 1),
		                                                               my_struct.my_nums.the_doubles,
		                                                               my_struct.a,
		                                                               0);
	jl_my_struct_a = jl_box_int64(my_struct.a);

	//box done
	jl_value_t* mod = (jl_value_t*) jl_eval_string("MyModule");

	//first create the "the_numbers" struct
	jl_value_t* jl_the_numbers = jl_new_struct((jl_datatype_t*)jl_get_function((jl_module_t*)mod, "the_numbers"),
		                                     jl_my_struct_my_nums_the_integers,
		                                     jl_my_struct_my_nums_the_integers_2,
		                                     jl_my_struct_my_nums_the_doubles);
	jl_value_t* jl_test_struct_1 = jl_new_struct((jl_datatype_t*)jl_get_function((jl_module_t*)mod, "test_struct_1"),
		                                          jl_my_struct_a,
		                                          jl_the_numbers);


	jl_function_t* jl_print_struct = jl_get_function((jl_module_t*)mod, "print_struct");
	jl_value_t* res = jl_call1(jl_print_struct, jl_test_struct_1);

	std::cout << "Res: " << jl_unbox_float64(res) << std::endl;

	JL_GC_POP();

	jl_atexit_hook(0);

return 0;
}

[yuyichao]

@Vangelis

       the_integers::Vector{Cint}

This is NOT what your C code have and this is a bad way to pass data from C to julia.

As I said many times, you want pointers here instead of arrays or NTuples. If you want, you can do the conversion to julia array in julia code.

[StefanKarpinski]

Just to spell it out, the C type int* corresponds to the Julia type Ptr{Cint} and this is what you must use in your structure to communicate between Julia and C. When you construct such an object from Vector{Cint} objects in Julia, you will need to use pointer(v) to get a pointer to the vector memory and GC.@preserve on the vectors objects to ensure that the pointers remain valid during the time they are used.

May I ask why you need to call Julia from C? It’s usually much easier to do this the other way around and call C from Julia.

[barnett567]

Dear Stefan & co,

Just to answer your final question as to why we want to call Julia from C (or Fortran): It is essential if Julia libraries are to be used with existing scientific codebases. One does not always have the luxury of declaring that Julia be the top-level/driver language If I am to write Julia code, Fortran users need to be able to call it, just like we’ve had C-Fortran interoperability for decades. (With MATLAB, this cannot be done, driving my move towards Julia.) I have a set of minimal demos of this task on github, but I’m finding it a challenge:

I made examples of wrapping jl code from Fortran & C; how generalize?

I am interested in calling Julia from Fortran (eg F90), and since I couldn’t find any, I built an example of that which goes through a C wrapper layer; see here: https://github.com/ahbarnett/multilingual-julia It adapts code from the manual (Ch 30: Embedding) plus a SO post which explains how to call general J code. F is responsible for preallocating output arrays, in the classical F style. Multithreading works. It works but feels too clunky, involving text commands and a lot of jl_* calls pa…

What I’m really missing is a set of @cfunction usage examples with arrays. Any pointers (pun indended) welcome
Best wishes, Alex