Segmentation fault while trying to call a Julia function compiled to .so file from Python

thecodingbugs · April 18, 2024, 10:37am

Hello,

I am using the examples from PackageCompiler.jl, specifically the MyLib one, and have issues with it. I have compiled MyLib using build.jl file given with it and have created .so files. Now I am trying to use these .so files in Python using ctypes. I have been able to load the .so file successfully, get the function pointer, assign its types, etc. Just one last hurdle and that is segmentation fault occurring while calling the function.

Even the simplest print_hello() that I created as a test does not work and gives the segmentation fault.

Any pointers will be really helpful.

Thanks!

Palli · April 18, 2024, 11:03am

It seem you’re going by this, i.e. 1. or 2.: Libraries · PackageCompiler

I’ve not done this so I can’t help much, except point to a workaround: It seems you really want to compile a library ; to use from Python, but you don’t have to. You can use PythonCall.jl and it will download Julia for you and your Julia package dependencies. And they will be as fast since Julia precompiles them. Only on first use (probably) noticeably slower.

Compiled binaries apps (and also libraries) will not be small (that way), as you’re trying, just about the same size.

Can you show us the error? It seems to me your error may actually be on the Python side. I would first get you Julia code working (then with/from Python), then if you need an app look into AppBundler.jl, then Julia will be your main language (you can still call Python, I believe, it should also then just download Python for you, if you insist on both languages bundled in one distributable file, I think it may be harder, but possible). I would only compile with any method if you’re sure it’s needed and helpful. Then as you do, or several other options, e.g. StaticCompiler.jl (with many limitations, could be best if you really need that and small binaries, or compile to webassembly, then GC not an issue.

mkitti · April 18, 2024, 2:10pm

PackageCompiler.jl creates a system image in the form of a .so file. However, this is not meant to be loaded directly. You still need to initialize the Julia runtime.

Basic initialization is covered below.

https://docs.julialang.org/en/v1/manual/embedding/

If your primary objective is to use this from Python, I recommend using the (py)juliacall package to initialize thr Julia runtime, passing the system image in as an option.

thecodingbugs · April 22, 2024, 11:39pm

Thanks for your response. I do not want users to install Julia separately, that is the main point. The code I am using right now is this one: PackageCompiler.jl/examples/MyLib at master · JuliaLang/PackageCompiler.jl · GitHub

I am compiling MyLib using PackageCompiler and the generated so file “libmylib.so” I am loading that directly into Python without loading the “libjulia.so”, so initialisation is not done.

Yess, the error is on the Python end: segmentation fault, now when I use the libjulia.so to init Julia, it shows me another error of sys.so not found.

thecodingbugs · April 22, 2024, 11:40pm

Please check the other reply. Thanks for your response. I am looking for a solution wherein Julia need not be installed, but the code can be transferred into binaries for usage in other language like Python.

djholiver · April 23, 2024, 8:11am

Hi,

Do you have a specific reason for creating the .so file as opposed to create_app which creates an application (.exe on windows) with a bundle of files to be “deployed” independent of Julia being installed on the target machine? Note that, although the documentation is great, the julia_main method required is indeed empty and the ARGs are the normal “magic object” type that are passed in at the command line to be consumed by your actual Julia code.

We have found that to be a more suitable way to Dev in Julia then pass over to python Devs for integration or library exposure since, as to the above, outside of the original machine used to create the app (which is the same for the .so) file it clearly separates the dependency without much more work.

Regards,

mkitti · April 23, 2024, 8:08pm

This is not generally possible at the moment. Julia binary code requires the Julia runtime. You must initialize the Julia runtime.

There is some experimental approaches to accomplish what you want, but they require highly specialized Julia code.

If your goal is to have a runtime free language, then Julia is not the language you are looking for.

kdheepak · April 23, 2024, 9:33pm

PackageCompiler.jl essentially copies the Julia runtime into the shared library directory, right? If one were to do that, would this approach be viable?

mkitti · April 23, 2024, 10:12pm

Sure, but you still need to initialize Julia first.

github.com

simonbyrne/libcg/blob/master/main-c/main.c#L25


      
              for (int i = 1; i < len - 1; i++)
              {
                  y[i] = x[i] - c * (x[i - 1] + x[i + 1]);
              }
              y[len - 1] = x[len - 1] - c * x[len - 2];
              return 0;
          }
          
          int main(int argc, char *argv[])
          {
              init_julia(argc, argv);
          
              int ret;
              double *b = (double *)malloc(len * sizeof(double));
              double *x = (double *)malloc(len * sizeof(double));
          
              for (int i = 0; i < len; i++)
              {
                  x[i] = 0.0;
                  b[i] = 1.0;
              };

kdheepak · April 24, 2024, 2:13pm

Yes, you’ll have to use init_julia.

fwiw, @thecodingbugs I tried this approach (in 2019, things may have changed since then, so take this with a grain of salt) and got it to work with a simple Julia program but compilation of a reasonably sized program took a prohibitively long amount of time.

You might be better off using juliacall: Guide · PythonCall & JuliaCall and having the user install Julia. You can even automate the installation of Julia fairly easily using juliaup and Python subprocess commands.

thecodingbugs · April 25, 2024, 9:26am

I did this in python, but now I am not able to find the Julia/sys.so file this is the error it shows.

thecodingbugs · April 25, 2024, 9:28am

Can you tell me what steps exactly you followed for Julia code compilation?

djholiver · April 25, 2024, 4:15pm

I’ve linked it above - if you’ve created the .so it’s almost identical but for the additional julia_main method.

Regards,

thecodingbugs · April 26, 2024, 5:18am

After the compilation in julia_init function - when called from python, the debugger shows that lib/julia//sys.so file is not found.

mkitti · April 26, 2024, 8:20am

That looks like it is looking in the default location. Where is the sys.so file?

Did you pass the location of the system image to the initialization function like juliacall does?

github.com

JuliaPy/PythonCall.jl/blob/e99aa43b3eaf457fcac17ceb9f928f13583d0a13/pysrc/juliacall/init.py#L199


      
          
          # initialise julia
          try:
              jl_init = lib.jl_init_with_image__threading
          except AttributeError:
              jl_init = lib.jl_init_with_image
          jl_init.argtypes = [c.c_char_p, c.c_char_p]
          jl_init.restype = None
          jl_init(
              (default_bindir if bindir is None else bindir).encode('utf8'),
              None if sysimg is None else sysimg.encode('utf8'),
          )
          
          # initialise PythonCall
          jl_eval = lib.jl_eval_string
          jl_eval.argtypes = [c.c_char_p]
          jl_eval.restype = c.c_void_p
          def jlstr(x):
              return 'raw"' + x + '"'
          script = '''
          try

thecodingbugs · April 29, 2024, 6:46am

I copied the sys.so file from “Download Julia page, and downloaded Julia binaries”, and other files as well which were required to run the script. Now, still I am getting segmentation fault but now I have more information:

signal(11.1): Segmentation fault
in expression starting at none:0
unknown function (ip: (nil))
Allocations: 2907 (Pool: 2899; Big: 8); GC: 0

mkitti · April 29, 2024, 11:58am

When you used PackageCompiler.jl in the beginning, what did you call the system image? That is the sys.so we are looking for.

What did you do to generate the current error?

Have you tried using juliacall yet? I’m not seeing a path forward until you can establish that works first.