Gradual Julia-ization of Python libraries

General Usage
, ,
21

I believe this is something heavily discussed in TongYuan between @JohnnyChen94 and me. We have engaged in some local open-source activities and have considered quite a lot about user-side adoption of Julia-based scientific computing frameworks (this is necessary if we want to make the use of Julia grow among the target users and make the company success).

We have also tried a lot.

JNumPy is very similar to what @kdheepak assumed about the so-called “PythonLimitedApi.jl”. Integrating PackageCompiler.jl can be awesome but we previously do not consider this, because loading multiple shared libraries that are all created by PackageCompiler.jl is a challenge.

module example

using TyPython
using TyPython.CPython

@export_py function mat_mul(a::AbstractArray, b::AbstractArray)::Array
    return a * b
end

function init()
    @export_pymodule _example begin
        jl_mat_mul = Pyfunc(mat_mul)
    end
end

# the following code is optional,
# but makes Python code loading much faster since the second time.
precompile(init, ())

end

What’s more about PackageCompiler.jl: Chen and I have thought that we might provide a battled-included build of PackageCompiler.jl where we can include a large number of Julia libraries for regular scientic computing scenarios, and finally export well-designed C functions for Python integration. This looks very promising for commercial products, and I feel sad that Syslab (TongYuan’s scientific computing IDE product based on Julia and VSCode) didn’t take this approach (it seems that it’s a bit late). However, this approach is still limited because Julia applications cannot be modular, running multiple Julia sysimages in a single process is pretty problematic. Chen said he was willing to solve things by inter-process solutions (like containers, local/remote services, etc.)

SyslabCC is an AOT compiler developed by TongYuan, and it’s highly welcome for you to have collaborations with us. This AOT compiler is capable of producing libraries for integrating with languages such as Python. We have recently supported C++ code generation and active/passive multi-threading use (e.g., Threads.@threads for-loops). The performance is competitive, at least not notably slower than native Julia for compuation tasks.

image
image1340×835 11.4 KB

You can run the benchmarks if you have Syslab community version installed, the source code is located at JuliaCon2024-JuliaAOT/benchmarks at main · Suzhou-Tongyuan/JuliaCon2024-JuliaAOT (github.com).

We are currently using our AOT compiler to write simulation algorithms and integrate them in Sysplorer, system multi-disciplinary modeling and simulation environment product in TongYuan (I believe the company actually earns money with this). The combination of high performance and rich ecosystem is a game changer for this area, as for all algorithms we wrote with Julia AOT are far faster than some traditional “established solution” for simulation.

10 Likes