Trying PackageCompiler.jl lib example on Python

el-analista · June 11, 2024, 4:14pm

Hey there,

I am testing PackageCompiler with the examples in the repo. I am able to compile the library correctly following the documentation but then when I try to use it in Python with the following code I just get

segmentation fault  python test_lib.py

import ctypes
import os

print(os.getcwd())
mylib = ctypes.CDLL('MyLibCompiled/lib/libinc.so')
mylib.increment32.argtypes = [ctypes.c_int]
mylib.increment32.restype = ctypes.c_int
mylib.increment64.argtypes = [ctypes.c_long]
mylib.increment64.restype = ctypes.c_long
count32 = 10
result32 = mylib.increment32(count32)
print(f"Result of increment32: {result32}")

Debugging it, basically it crashes when calling the increment method.
Wondering if anyone could point me to debug what is going on.
Many thanks,
Eduard

mkitti · June 12, 2024, 1:33am

You need to initilaize Julia:
https://docs.julialang.org/en/v1/manual/embedding/

el-analista · June 12, 2024, 9:33am

Good point!
I am trying to initialize it with

import ctypes
import os
from ctypes import c_void_p, RTLD_GLOBAL

libjulia = ctypes.CDLL('MyLibCompiled/lib/libjulia.so', RTLD_GLOBAL)
libjulia.jl_init()

however when doing jl_init() I get

ERROR: could not load library "MyLibCompiled/lib/../bin/../lib/julia/sys.so"
MyLibCompiled/lib/../bin/../lib/julia/sys.so: cannot open shared object file: No such file or directory

As the path is trimmed I cant really know what is happening but it seems is trying to find this file but it didnt get ocmpiled?

mkitti · June 12, 2024, 2:33pm

So you need to set jl_options with the location of the system image you just compiled.

Frankly, you should probably use the Python package juliacall to at least initialize Julia, then try to call the C function pointer.

el-analista · June 13, 2024, 7:55pm

Thanks again Mark! I would prefer to avoid using juliacall as I plan to call from other languages such as java. I found a libcg repo which successfully build and calls a librar From c and rust however i read that there are some libraries that don’t work well and that everything needs to be statically typed which limits the usage quite a bit

mkitti · June 14, 2024, 2:33am

This is slightly easier to describe from either C or Zig than Python.

You can see the julia_init function the libcg function that libcg uses works:

github.com

JuliaLang/PackageCompiler.jl/blob/master/src/julia_init.c

#include <limits.h>
#include <stdio.h>
#include <stdlib.h>

#ifdef _MSC_VER
JL_DLLEXPORT char *dirname(char *);
#else
#include <libgen.h>
#endif

// Julia headers (for initialization and gc commands)
#include "julia.h"
#include "julia_init.h"
#include "uv.h"

void setup_args(int argc, char **argv) {
    uv_setup_args(argc, argv);
    jl_parse_opts(&argc, &argv);
}

This file has been truncated. show original

Basically, we need to communicate to Julia the location of the system image. You can either populate the jl_options struct directly or use the command line option processor. The -J or --sysimage command line option indicates the system image location.

As I say this though, I suspect that the juliac tool is about to be reincarnated, which might provide a whole new pathway to do this.

The bottom line is that you must initialize the Julia runtime before invoking that C pointer.