Segmentation fault while trying to call a Julia function compiled to .so file from Python

Hello,

I am using the examples from PackageCompiler.jl, specifically the MyLib one, and have issues with it. I have compiled MyLib using build.jl file given with it and have created .so files. Now I am trying to use these .so files in Python using ctypes. I have been able to load the .so file successfully, get the function pointer, assign its types, etc. Just one last hurdle and that is segmentation fault occurring while calling the function.

Even the simplest print_hello() that I created as a test does not work and gives the segmentation fault.

Any pointers will be really helpful.

Thanks!

It seem you’re going by this, i.e. 1. or 2.: Libraries · PackageCompiler

I’ve not done this so I can’t help much, except point to a workaround: It seems you really want to compile a library ; to use from Python, but you don’t have to. You can use PythonCall.jl and it will download Julia for you and your Julia package dependencies. And they will be as fast since Julia precompiles them. Only on first use (probably) noticeably slower.

Compiled binaries apps (and also libraries) will not be small (that way), as you’re trying, just about the same size.

Can you show us the error? It seems to me your error may actually be on the Python side. I would first get you Julia code working (then with/from Python), then if you need an app look into AppBundler.jl, then Julia will be your main language (you can still call Python, I believe, it should also then just download Python for you, if you insist on both languages bundled in one distributable file, I think it may be harder, but possible). I would only compile with any method if you’re sure it’s needed and helpful. Then as you do, or several other options, e.g. StaticCompiler.jl (with many limitations, could be best if you really need that and small binaries, or compile to webassembly, then GC not an issue.

PackageCompiler.jl creates a system image in the form of a .so file. However, this is not meant to be loaded directly. You still need to initialize the Julia runtime.

Basic initialization is covered below.

https://docs.julialang.org/en/v1/manual/embedding/

If your primary objective is to use this from Python, I recommend using the (py)juliacall package to initialize thr Julia runtime, passing the system image in as an option.

2 Likes

Thanks for your response. I do not want users to install Julia separately, that is the main point. The code I am using right now is this one: PackageCompiler.jl/examples/MyLib at master · JuliaLang/PackageCompiler.jl · GitHub

I am compiling MyLib using PackageCompiler and the generated so file “libmylib.so” I am loading that directly into Python without loading the “libjulia.so”, so initialisation is not done.

Yess, the error is on the Python end: segmentation fault, now when I use the libjulia.so to init Julia, it shows me another error of sys.so not found.

Please check the other reply. Thanks for your response. I am looking for a solution wherein Julia need not be installed, but the code can be transferred into binaries for usage in other language like Python.

Hi,

Do you have a specific reason for creating the .so file as opposed to create_app which creates an application (.exe on windows) with a bundle of files to be “deployed” independent of Julia being installed on the target machine? Note that, although the documentation is great, the julia_main method required is indeed empty and the ARGs are the normal “magic object” type that are passed in at the command line to be consumed by your actual Julia code.

We have found that to be a more suitable way to Dev in Julia then pass over to python Devs for integration or library exposure since, as to the above, outside of the original machine used to create the app (which is the same for the .so) file it clearly separates the dependency without much more work.

Regards,

This is not generally possible at the moment. Julia binary code requires the Julia runtime. You must initialize the Julia runtime.

There is some experimental approaches to accomplish what you want, but they require highly specialized Julia code.

If your goal is to have a runtime free language, then Julia is not the language you are looking for.

2 Likes

PackageCompiler.jl essentially copies the Julia runtime into the shared library directory, right? If one were to do that, would this approach be viable?

Sure, but you still need to initialize Julia first.

1 Like

Yes, you’ll have to use init_julia.

fwiw, @thecodingbugs I tried this approach (in 2019, things may have changed since then, so take this with a grain of salt) and got it to work with a simple Julia program but compilation of a reasonably sized program took a prohibitively long amount of time.

You might be better off using juliacall: Guide · PythonCall & JuliaCall and having the user install Julia. You can even automate the installation of Julia fairly easily using juliaup and Python subprocess commands.

I did this in python, but now I am not able to find the Julia/sys.so file this is the error it shows.

Can you tell me what steps exactly you followed for Julia code compilation?

I’ve linked it above - if you’ve created the .so it’s almost identical but for the additional julia_main method.

Regards,

After the compilation in julia_init function - when called from python, the debugger shows that lib/julia//sys.so file is not found.

That looks like it is looking in the default location. Where is the sys.so file?

Did you pass the location of the system image to the initialization function like juliacall does?

I copied the sys.so file from “Download Julia page, and downloaded Julia binaries”, and other files as well which were required to run the script. Now, still I am getting segmentation fault but now I have more information:

signal(11.1): Segmentation fault
in expression starting at none:0
unknown function (ip: (nil))
Allocations: 2907 (Pool: 2899; Big: 8); GC: 0

When you used PackageCompiler.jl in the beginning, what did you call the system image? That is the sys.so we are looking for.

What did you do to generate the current error?

Have you tried using juliacall yet? I’m not seeing a path forward until you can establish that works first.

I using this : PackageCompiler.jl/examples/MyLib at master · JuliaLang/PackageCompiler.jl · GitHub as an example to see whether it works or not. In this link there is a folder build and in that folder there is a file which Julia provides build.jl I used this file to build the .so file directly for the MyLib package that is provided in the above link. For creation of system image. the file generated is sysimage.so, while in the error before it showed it said cannot find sys.so file, so I suppose both of them are different and once I put sys.so from Julia binaries, it threw the error for so files that are packaged along with the sys.so, so I copied them as well and all the error went away. The segmentation fault error threw shows the memory access of 0 the address was no random hex number but just zeros. I am looking into JuliaCall now, but I need to know if it is any better than using PackageCompiler.jl in terms of speed and simplicity. Let me know. Thanks in advance!

You can use PackageCompiler.jl and juliacall together by passing the location of the system image to juliacall.

Search for PYTHON_JULIACALL_SYSIMAGE at Guide · PythonCall & JuliaCall

Thr technologies are complimentary. Juliacall makes it easy to load and initialize Julia from Python. PackageCompiler.jl mainly helps you create a system image.

If you compile your code ahead-of-time (AOT) into the system image, then Julia will not have do this compilation later. Then you can invoke this via juliacall.

Notice that even in the example you linked C has to invoke julia_init and relay command line argiments.

2 Likes

Yes you are right, let me try this solution and revert! Thanks for your responses.