PackageCompiler libs and Python: Passing numpy arrays (or structs)

General Usage
1

I have succesfully tried out the PackageCompiler library + Python Ctypes example from

Sadly the example is extremely simple and does not show how to pass parameters, especially arrays from python.

I would be thankful for any working (and idiomatic) snippet passing a numpy array to a PackageCompiler-generated Julia library , but i will also post what I tried so far.
(My version prints wrong data when i request the array size.)

Since i was experimenting with the StaticCompiler first, i know two possibilities how it can be done there

a) with separate pointer and length, shown here:
Successful Static Compilation of Julia Code for use in Production

b) using the same info, but packed into a struct and then passing a reference to the struct as shown here:
GitHub - brenhinkeller/StaticTools.jl: Enabling StaticCompiler.jl-based compilation of (some) Julia code to standalone native binaries by avoiding GC allocations and llvmcall-ing all the things!

I found version (b) to be a bit more organized and thus decided to try this. Here are my code snippets:

Julia - tries to only output the size field. So actually this only tests struct passing and not even the actual array functionality. Probaly there is also some stuff missing to get true array functionality (getindex, setindex etc)

struct MallocArray{T,N} <: DenseArray{T,N}
	pointer::Ptr{T}
	length::Int
	size::NTuple{N, Int}
end
const MallocMatrix{T} = MallocArray{T,2}
Base.@ccallable function func1(	a::RefValue{MallocMatrix{Float64}},
								b::RefValue{MallocMatrix{Float64}}) :: Cvoid
	println(a[].size)
	println(b[].size)
end

Python

class MallocMatrix(ct.Structure):
    _fields_ = [("pointer", ct.c_void_p),
                ("length", ct.c_int64),
                ("s1", ct.c_int64),
                ("s2", ct.c_int64)]
                
def getMatrixptr(A):
    ptr = A.ctypes.data_as(ct.c_void_p)
    a = MallocMatrix(ptr, ct.c_int64(A.size), ct.c_int64(A.shape[1]), ct.c_int64(A.shape[0]))
    return ct.byref(a)

a = np.zeros((7,2))
b = np.zeros((10,2))
print("func1")
lib.func1(getMatrixptr(a),getMatrixptr(b))

Output (should be 7,2 and 10,2)

func1
(140732223482816, 139903138553959)
(8, 1)
2

This is basically what you’re after:

But FYI, I don’t believe it’s been announced yet, so use (or study) it at your own risk.

I would recommend you use JuliaCall (part of PythonCall.jl):

I believe that package is the future of calling to or from Python. PyJulia (based in PyCall.jl) is an older alternative.

PackageCompiler.jl is useful for AOT (that you likely do not need), and recent Julia (the beta?) fully precompiles packages to machine code, so there’s less need for explicit AOT. PackageCompiler.jl supports all Julia code, unlike StaticTools/StaticCompiler, where you need to jump through some hoops, so you really want to ask yourself is it worth it (do I need really small libraries?)?

You can make a C-callable library with PackageCompiler.jl for AOT compiling (or with StaticCompiler), but then still need to figure out how to make a Python extension using it, and the first project I linked to is for that, if you really need it to be a Python extension, not just the languages working together somehow.

3

Thanks for your reply. Your first suggestion looks very interesting as a pre-packaged and convenient solution at first glance. I will see if i can successfully use it and get back. (Still curious what is wrong with my handwritten solution though)

Concerning the discussion about PyJulia (or newer replacements): The reason why i experimented with both StaticCompiler and PackageCompiler is that i do need/want AOT explicitly. I can install julia on my own computer at work, but I do not have that option on other computers where i need to deploy my code. Secondly, i am simply interested to make this work because i generally think that it should be possible to properly compile a language that is already JITed by default.

4

I believe JuliaCall downloads Julia for you on first use. If you don’t want that (want to distribute code without needing internet connection), then I suppose you can run your code once, and distribute what it has downloaded for you, Julia and all the packages Julia needs. If it’s a matter of size, then I think you can use PackageCompiler to compile the code into a sysimage (I believe not contrary to using JuliaCall), and you could in many (most?) cases also strip out LLVM and other big dependencies of Julia.

AOT means the code you distribute is tied to one platform. I haven’t looked at it from the Python perspective, but that would also apply I guess. They must solve it by distributing different binaries based on platform (or a fat binary?). So even if Python code is portable, by default, with extensions (or AOT Julia), you are making distribution much harder.

5

So far, i have some open question regarding the “JNumPy” solution and could not implement it easily, but i think i should open a new thread for that in case i can’t figure it out and in case i am still interested at that point.

At the moment i still wonder why my own code does not work. I have now switched out my handwritten “MallocArray” part-implementation with simply including StaticTools, so that the whole julia file now looks like this:


module myPkg
export func1
using StaticTools
using Base: RefValue
	Base.@ccallable function func1(	a::RefValue{MallocMatrix{Float64}},
									b::RefValue{MallocMatrix{Float64}}) :: Cvoid
		println(size(a[]))
		println(size(b[]))
	end
end

This also means, that the code now looks like an older version which i used with StaticCompiler and which seemed to work properly at the simple level presented here (although my final application never worked). While i could not use println there to access the array sizes i was writing numbers to the array there and i assume there would have been problems if the array size had been completely wrong as it is here.

The python-side has not changed since the original post and it also looks very much like what i had previously (with StaticCompiler), except for the initialisation part, which i have not shown in this thread so far.

libpath = os.path.abspath("./myPkgCompiled/lib/libwhatever.so")
lib = ct.CDLL(libpath)
try:
    jl_init_with_image = lib.jl_init_with_image
except AttributeError:
    jl_init_with_image = lib.jl_init_with_image__threading
jl_init_with_image.argtypes = [ct.c_char_p, ct.c_char_p]
jl_init_with_image(None, str(libpath).encode("utf-8"))

I don’t understand why this version does not work.
I noticed, that some examples set “argtypes” for the functions that are called(as is done for “jl_init_with_image”), but i did not have to do this (?) with the static compiler either - additionally i do not know what the correct argtypes for the struct pointers would be.

6

I do have a question about JNumpy. I downloaded the package and was able to run the basic example. However i never found any references to PackageCompiler and no *.so or *.o file was created. I have the feeling that this is more like a PyJulia replacement then like a helper for using compiled code. Or was your suggestion that it is possible to PackageCompile the julia code from the example modules with PackageCompiler (by hand) and it will provide the typeconversions that i failed to implement myself?

7

You’re right, I assumed some .so made since for “writing Python C extensions”. It seems to be a PyJulia replacement, but will download Julia for you, so it’s maybe more of a juliacall (from PythonCall.jl) replacement. And I’m not sure it’s better (has not been announced), I just believe in the developer (also PyCall.jl’s developer and it’s been released). I just now discovered another project of his, and it supports sysimages, so may be of help. I don’t understand all of his docs, both partially in Chinese (everything important also in English?):

8

Apparently i have now managed to accomplish what i wanted using “Variety a” from my original post (interested people please consider the link there for code examples). The linked post mainly explains the circumvention of pitfalls with the use of StaticCompiler. However this also involves the passing of arrays allocated on the numpy side into a julia function. The trick there is to pass pointer and length separately and then create a julia array from this. I would have preferred to pass those data packed into a struct as i tried in my previous posts. But at least it is a working solution. However i have not tried multidimensional arrays yet.

1 Like