Thanks. It appears you’re right. I changed the MyLib.jl file to read as follows:
# MyLib.jl/src/MyLib.jl
module MyLib
export MyOuterStruct, MyInnerStruct, get_inner
const ARRLENGTH = 5
struct MyInnerStruct
n::Cint
arr::NTuple{ARRLENGTH,Cfloat}
end
struct MyOuterStruct
inner::MyInnerStruct
end
Base.@ccallable function get_inner_first_val(msp::Ptr{MyOuterStruct})::Cfloat
GC.@preserve msp begin
ms = unsafe_load(msp)
inner = ms.inner
arr = inner.arr
return arr[1]
end
end
end
The rest of the code (the build files and the C test program) is the same. After re-compiling the library with PackageCompiler and re-compiling/linking the test.c file, the get_inner_first_val function now works as expected:
~/MyLib/test » ./test ubuntu@ip-172-31-47-177
1.000000
Unfortunately, the size of the array member in the real code is much, much larger; the equivalent of ARRLENGTH is currently set to 26214400 (the array member is meant to store image data, and the images are all quite large).
Updating ARRLENGTH to match the production value in mylib.h and MyLib.jl, with no other changes, test.c segfaults. Of course, in test.c the MyOuterStruct is allocated on the stack. I updated test.c to instead allocate the MyOuterStruct on the heap:
// MyLib.jl/test/test.c
#include <stdio.h>
#include <stdlib.h>
#include "julia_init.h"
#include "mylib.h"
int main(int argc, char const* argv[]) {
init_julia(argc, argv);
MyOuterStruct* outer = malloc(sizeof(MyOuterStruct));
outer->inner.n = 3;
for (int i = 0; i < outer->inner.n; i++) {
outer->inner.arr[i] = 1.0;
}
float v = get_inner_first_val(outer);
printf("%f\n", v);
free(outer);
shutdown_julia(0);
return 0;
}
This works, succesfully printing the first value stored in inner.arr. However, this takes a very long time (several minutes) to execute. That kind of execution speed is totally unacceptable for the context my library is meant to be used in (images should be loaded and analyzed in <1s.
My thought here was this long runtime this was related to JITing during the execution of the test program, so I added some statements to the pre-compilation file:
# MyLib.jl/build/generate_precompile.jl
using MyLib
arr = Tuple(zeros(Cfloat, ARRLENGTH));
inner = MyInnerStruct(zero(Cfloat), arr)
outer = MyOuterStruct(inner)
outer_ref = Ref(outer)
GC.@preserve outer_ref begin
outer_p = Base.unsafe_convert(Ptr{MyOuterStruct}, outer_ref)
cx = get_inner_first_val(outer_p)
end
However, the construction of inner here throws a StackOverFlowError, halting the compilation of the library.
This makes sense, since Tuples are not intended to store this many elements.
On the other hand, if I switch to a regular float* arr member in the C code and Ptr{Cfloat} in Julia, the whole program runs in about 0.2s. For completeness, here are the updated MyLib.jl, mylib.h, and test.c:
# MyLib.jl/src/MyLib.jl
module MyLib
export MyOuterStruct, MyInnerStruct, get_inner_first_val, ARRLENGTH
const ARRLENGTH = 5120^2
struct MyInnerStruct
n::Cint
arr::Ptr{Cfloat}
end
struct MyOuterStruct
inner::MyInnerStruct
end
Base.@ccallable function get_inner_first_val(msp::Ptr{MyOuterStruct})::Cfloat
GC.@preserve msp begin
ms = unsafe_load(msp)
inner = ms.inner
arrp = inner.arr
GC.@preserve arrp begin
arr = unsafe_wrap(Array, arrp, inner.n)
return arr[1]
end
end
end
end
// MyLib.jl/build/mylib.h
#include "julia.h"
#include "uv.h"
#define ARRLENGTH 5120 * 5120
typedef struct MyInnerStruct {
int n;
float* arr;
} MyInnerStruct;
typedef struct MyOuterStruct {
MyInnerStruct inner;
} MyOuterStruct;
float get_inner_first_val(MyOuterStruct* ms);
// MyLib.jl/test/test.c
#include <stdio.h>
#include <stdlib.h>
#include "julia_init.h"
#include "mylib.h"
int main(int argc, char const* argv[]) {
init_julia(argc, argv);
MyOuterStruct* outer = malloc(sizeof(MyOuterStruct));
float* arr = calloc(ARRLENGTH, sizeof(float));
outer->inner.arr = arr;
outer->inner.n = 100;
for (int i = 0; i < outer->inner.n; i++) {
outer->inner.arr[i] = 1.0;
}
float v = get_inner_first_val(outer);
printf("%f\n", v);
free(outer);
shutdown_julia(0);
return 0;
}
It seems like I’ve run into a fundamental design problem here; trying to pass fixed-size array members of the required size between Julia and C doesn’t seem to be feasible. I don’t have direct control over the C code that will call my library, but I could perhaps suggest some alterations to allow this set-up to work.