Calling Julia shared library from Python, linking libraries

General Usage
1

Hi all,
I’m trying to get PackageCompiler to emit Python wrappers for shared libraries compiled from Julia packages.

I got as far as this minimal test:

https://github.com/JuliaLang/PackageCompiler.jl/tree/sd/py_so/test/pytest

Which actually works fine and registers a c-extension with python, but when I import it in python it throws:

>>> import hello

/usr/local/lib/python2.7/dist-packages/hello-1.0-py2.7-linux-x86_64.egg/hello.py:3: UserWarning: Module hello was already imported from /usr/local/lib/python2.7/dist-packages/hello-1.0-py2.7-linux-x86_64.egg/hello.pyc, but /home/s/.julia/v0.6/PackageCompiler/test/pytest is being added to sys.path

fatal: error thrown and no exception handler available.

Base.InitError(mod=:Sys, error=ErrorException("could not load symbol "jl_cpu_cores":

python: undefined symbol: jl_cpu_cores"))

The linking + compilation commands setuptools generate from the above linked setup.py are:

creating build/temp.linux-x86_64-2.7

x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DJULIA_ENABLE_THREADING=1 -I/home/s/juliastuff/julia6_2/include/julia -I/usr/include/python2.7 -c pymodule.c -o build/temp.linux-x86_64-2.7/pymodule.o

creating build/lib.linux-x86_64-2.7

x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/pymodule.o /home/s/.julia/v0.6/PackageCompiler/test/pytest/pyshared.o -L/home/s/juliastuff/julia6_2/lib -Wl,-R/home/s/juliastuff/julia6_2/lib -Wl,-R/home/s/juliastuff/julia6_2/lib/julia -Wl,-R/home/s/.julia/v0.6/PackageCompiler/test/pytest/ -ljulia -o build/lib.linux-x86_64-2.7/hello.so -DJULIA_ENABLE_THREADING=1 -std=gnu99 -fPIC -Wl,--export-dynamic

creating build/bdist.linux-x86_64

creating build/bdist.linux-x86_64/egg

copying build/lib.linux-x86_64-2.7/hello.so -> build/bdist.linux-x86_64/egg

creating stub loader for hello.so

byte-compiling build/bdist.linux-x86_64/egg/hello.py to hello.pyc

I basically try to link pyshared.o to the c extension. pyshared.o is generated from pyshared.jl with those commands.

I hope, that there is just a wrong flag somewhere, and someone with a bit more experience with julia internals and linking libraries can quickly spot the mistake :wink:

Best,
Simon

2 Likes
2

Running into the same error when embedding Julia in Java:

g++ -I$JNI_ARCH_INC -I$JNI_INC -I$JULIA_INC -fPIC -Wall -c -g -o target/interop.o interop.cpp
g++ -o target/libinterop.so target/interop.o -L$JULIA_LIB -ljulia -ldl -shared -Wl,-rpath,"$JULIA_LIB" -Wl,-rpath,"$JULIA_LIB/julia"

@sdanisch - did you ever find out what was going on?

3

Mitigated by explicitly specifying the LD_PRELOAD=libjulia.so. It’s a hack, so I’d like to solve the issue properly. Somehow the shared Julia lib isn’t picked up even though all the information is present in the libinterop.so which I load from Java.

Would be grateful for any hints.