This post is asking about the practical status of Julia small binary compilation, particularly for interfacing with Numpy Arrays in Python.
Suppose:
(1) There is a large model written in python, where data is passed around between NumPy arrays (as well as lists and non-performant data structures). For various reasons, the model must remain as Python code
(2) Julia is a better language for the model, both for reasons of performance and polymorphism.
Julia small binary compilation is still experimental. That said, how hard is it to:
(1) write a large module in Julia that accepts arrays of data and outputs arrays of data
(2) compile it to a small binary
(3) write a Python wrapper, which passes Numpy Arrays to the Julia module and outputs Numpy arrays from Julia.
How hard is it to compile the Julia binaries for different architectures?
Are there publicly available code bases that demonstrate this sort of thing, beginning to end?
Most likely not. There are currently only a small number of packages available for creating small binaries.
You say you want to write (or translate) a model. Could you provide a bit more information about the model you wish to implement? Do you need ModellingToolkit? Do you need DifferentialEquations? Which packages would you need?
Probably not, but, because it sounds like you have opinions on the feasibility of compiling code that relies on those packages, it might be instructive if you told me how using those (I presume) won’t work.
Also, suppose the compiled Julia code wrapped in a python package does something trivial: accept a python numpy array, add 1 to each element, and output a python numpy array (not a practical use case, just thinking about how interface would work). How hard would it be to move these numpy arrays in and out of the python-wrapped julia-compiled code?
NumPy arrays don’t naturally cross language barriers, you pass a bunch of metadata that the other language needs to wrap to some degree. Any chance PythonCall is JuliaC friendly?
I also have an example for this: GitHub - RomeoV/RunwayLib.jl at 765c499d89573d16036a4963d3b820bd64e68066
You may be interested in the juliac and poseest_py directories, as well as the CI files. Also, one challenging thing was to set the depot path correctly for python.
However, more recently i have deprecated the approach of wrapping JuliaC’d code in a python module and instead replaced it with JuliaCall.jl as that is sufficient for my use case. The main issue was constantly trying to maintain multiple implementations of the API (Julia, C, python), but that would be better if the API is mostly stable.