Embedding julia: libLLVM-9jl.so not found

crbinz · February 11, 2021, 7:54pm

I’m trying to compile a simple C++ program that embeds some julia code. I’ve gotten this to work on macOS, but when trying to compile on my CentOS machine, I get this warning, followed by lots of errors:

/usr/bin/ld: warning: libLLVM-9jl.so, needed by /opt/julia-1.5.3/lib/libjulia.so, not found (try using -rpath or -rpath-link)

I compile with g++ -o jl_test -fPIC -I/opt/julia-1.5.3/include/julia -L/opt/julia-1.5.3/lib -Wl,-rpath,/opt/julia-1.5.3/lib src/embedded_julia.cpp -ljulia

Does anyone have any suggestions for fixing this or further diagnosing the issue?

pixel27 · February 11, 2021, 9:30pm

Instead of rpath I would try -L/opt/julia-1.5.3/lib/julia that’s where libLLVM-9jl.so lives on my computer.

crbinz · February 11, 2021, 9:51pm

Thanks, that was ultimately the issue, which I fixed simply by using the makefile in the docs. Now, though, I am getting many errors like this:

/opt/julia-1.5.3/lib/libjulia.so: undefined reference to `vtable for std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21'

followed by, at the end:

collect2: error: ld returned 1 exit status
make: *** [<builtin>: src/embedded_julia] Error 1

I’m having a hard time figuring out what this could be. I thought that maybe I need a newer version of gcc/g++, since strings /lib64/libsdtc++.so.6 | grep GLIBCXX does not show GLIBCXX_3.4.21 (mentioned in the errors), but I tried updating to gcc v8.3.1 and I’m running into the same issues…

EDIT: I see now that /opt/julia-1.5.3/lib/julia contains its own libstdc++, which does show the necessary GLIBCXX version, so I’m at a loss…

giordano · February 11, 2021, 10:55pm

I tried to build the embedding example in a Centos 7 Docker container:

FROM centos:7

RUN yum install -y gcc make

RUN curl -sL "https://julialang-s3.julialang.org/bin/linux/x64/1.5/julia-1.5.3-linux-x86_64.tar.gz" | tar xzf - -C /usr/local --strip-components=1

COPY embed_example.c Makefile /root/

WORKDIR "/root"

Inside the container:

[root@5a985425fcb5 ~]# gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[root@5a985425fcb5 ~]# make
cc -std=gnu99 -I'/usr/local/include/julia' -fPIC  -L'/usr/local/lib' -Wl,--export-dynamic  embed_example.c  -Wl,-rpath,'/usr/local/lib' -Wl,-rpath,'/usr/local/lib/julia' -ljulia -o embed_example
[root@5a985425fcb5 ~]# ./embed_example 
1.4142135623730951
[root@5a985425fcb5 ~]# ldd embed_example
        linux-vdso.so.1 =>  (0x00007ffc985ba000)
        libjulia.so.1 => /usr/local/lib/libjulia.so.1 (0x00007f7541713000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f7541345000)
        libunwind.so.8 => /usr/local/lib/julia/libunwind.so.8 (0x00007f754110c000)
        libLLVM-9jl.so => /usr/local/lib/julia/libLLVM-9jl.so (0x00007f753d9a9000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f753d7a5000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f753d59d000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f753d381000)
        libstdc++.so.6 => /usr/local/lib/julia/libstdc++.so.6 (0x00007f753d003000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f753cd01000)
        libgcc_s.so.1 => /usr/local/lib/julia/libgcc_s.so.1 (0x00007f753cae9000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f7541ee2000)

Can you provide more information about your environment?

pixel27 · February 12, 2021, 12:30am

I’m wondering if you should remove the “normal” lib paths when doing the link. I’m not sure how to do that. Maybe setting LD_LIBRARY_PATH to an empty string?

crbinz · February 12, 2021, 4:11pm

I’m fairly certain this is the relevant difference. I found this thread, which describes the problem I was having (@pixel27, you were on the right track), so I’m trying to follow the advice there.

Unfortunately, now I keep running into /opt/julia-1.5.3/lib/julia/libstdc++.so.6: error adding symbols: DSO missing from command line, and I haven’t been able to figure that out yet…

crbinz · February 12, 2021, 5:58pm

Actually, I get the same results as you when I try to compile the (C) example… my problems arise when I try to compile a C++ example…

giordano · February 12, 2021, 6:19pm

If you actually share your code it may be easier for someone to try and reproduce your problem

crbinz · February 12, 2021, 6:26pm

Sure, good point. Here’s the entire simplified C++ file:

#include <julia.h>

int main(int argc, char *argv[])
{
    /* required: setup the Julia context */
    jl_init();
    jl_value_t* ans = jl_eval_string("println(sqrt(2.0))");    
    jl_atexit_hook(0);    
    return 0;
}

And here is the output of make, using the Makefile just as it appears in the docs.

giordano · February 12, 2021, 6:27pm

This is so simplified that is basically C, isn’t it?

crbinz · February 12, 2021, 7:11pm

I guess so… it’s just the minimum (non)working example I came up with.

pixel27 · February 12, 2021, 8:29pm

I think you might have gone too simple.

I installed CentOS Stream in a VirtualBox VM then downloaded Julia 1.5.3 and just extracted it into my home directory, .bashrc was updated to add $HOME/julia-1.5.3/bin to the path.

I started with the Makefile you specified and added the compile commands (the C file I saved as test.c) which ended up with:

JL_SHARE = $(shell julia -e 'print(joinpath(Sys.BINDIR, Base.DATAROOTDIR, "julia"))')
CFLAGS   += $(shell $(JL_SHARE)/julia-config.jl --cflags)
CXXFLAGS += $(shell $(JL_SHARE)/julia-config.jl --cflags)
LDFLAGS  += $(shell $(JL_SHARE)/julia-config.jl --ldflags)
LDLIBS   += $(shell $(JL_SHARE)/julia-config.jl --ldlibs)


all:
	julia -e 'using InteractiveUtils;versioninfo()'
	gcc -o test $(CFLAGS) $(LDFLAGS) $(LDLIBS) test.c

I didn’t include the CXXFLAGS because there is no C++ in that file . My make works fine:

[josh@centos ~]$ make
julia -e 'using InteractiveUtils;versioninfo()'
Julia Version 1.5.3
Commit 788b2c77c1 (2020-11-09 13:37 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-3820 CPU @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, sandybridge)
gcc -o test -std=gnu99 -I'/home/josh/julia-1.5.3/include/julia' -fPIC -L'/home/josh/julia-1.5.3/lib' -Wl,--export-dynamic -Wl,-rpath,'/home/josh/julia-1.5.3/lib' -Wl,-rpath,'/home/josh/julia-1.5.3/lib/julia' -ljulia test.c

And the program can run:

[josh@centos ~]$ ./test 
1.4142135623730951

crbinz · February 12, 2021, 10:07pm

Thanks - to be clear, when I try to compile the C test case, it works for me too. My end goal is not to compile this particular toy problem - I have a need to embed the julia code in some (actual) C++ code. I provided the toy problem because it gives me the same errors as my actual C++ example.

giordano · February 13, 2021, 12:51am

Dockerfile:

FROM centos:7

RUN yum install -y gcc gcc-c++ make

RUN curl -sL "https://julialang-s3.julialang.org/bin/linux/x64/1.5/julia-1.5.3-linux-x86_64.tar.gz" | tar xzf - -C /usr/local --strip-components=1

COPY embed_example.cpp Makefile /root/

WORKDIR "/root"

embed_example.cpp:

#include <julia.h>

int main(int argc, char *argv[])
{
    /* required: setup the Julia context */
    jl_init();
    jl_value_t* ans = jl_eval_string("println(sqrt(2.0))");    
    jl_atexit_hook(0);    
    return 0;
}

Makefile:

JL_SHARE = $(shell julia -e 'print(joinpath(Sys.BINDIR, Base.DATAROOTDIR, "julia"))')
CFLAGS   += $(shell $(JL_SHARE)/julia-config.jl --cflags)
CXXFLAGS += $(shell $(JL_SHARE)/julia-config.jl --cflags)
LDFLAGS  += $(shell $(JL_SHARE)/julia-config.jl --ldflags)
LDLIBS   += $(shell $(JL_SHARE)/julia-config.jl --ldlibs)
LIBSTDCXX = $(shell julia -E 'joinpath(dirname(Sys.BINDIR), "lib", "julia", "libstdc++.so.6")')

all: embed_example

embed_example: embed_example.cpp
	$(CXX) $(CXXFLAGS) $(LDFLAGS) $< $(LDLIBS) -nodefaultlibs $(LIBSTDCXX) -lc -o $@

In the container:

[root@ab82e3a1343c ~]# make
g++ -std=gnu99 -I'/usr/local/include/julia' -fPIC -L'/usr/local/lib' -Wl,--export-dynamic embed_example.cpp -Wl,-rpath,'/usr/local/lib' -Wl,-rpath,'/usr/local/lib/julia' -ljulia -nodefaultlibs "/usr/local/lib/julia/libstdc++.so.6" -lc -o embed_example
cc1plus: warning: command line option '-std=gnu99' is valid for C/ObjC but not for C++ [enabled by default]
[root@ab82e3a1343c ~]# ./embed_example 
1.4142135623730951
[root@ab82e3a1343c ~]# ldd ./embed_example 
        linux-vdso.so.1 =>  (0x00007ffe54cc2000)
        libjulia.so.1 => /usr/local/lib/libjulia.so.1 (0x00007fa1defbb000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fa1debed000)
        libunwind.so.8 => /usr/local/lib/julia/libunwind.so.8 (0x00007fa1de9b4000)
        libLLVM-9jl.so => /usr/local/lib/julia/libLLVM-9jl.so (0x00007fa1db251000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fa1db04d000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fa1dae45000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa1dac29000)
        libstdc++.so.6 => /usr/local/lib/julia/libstdc++.so.6 (0x00007fa1da8ab000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fa1da5a9000)
        libgcc_s.so.1 => /usr/local/lib/julia/libgcc_s.so.1 (0x00007fa1da391000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fa1df78a000)

The trick (suggested by @staticfloat) is to use -nodefaultlibs to not use the standard libraries (both C and C++), specify the full path to Julia’s libstdc++ and then use system’s libc.

Side note: this oversimplified example which is pure C would compile and run also without passing libstdc++ to the compiler, and the executable will not link to it. This is fine because the program is pure C. libllvm needs libstdc++, but it’ll look for $$ORIGIN/libstdc++ and find the one sitting next to it.

crbinz · February 13, 2021, 6:48pm

First of all, thanks again for your help, I appreciate it since I am fairly out of my depth here. I can indeed compile my original simple example with the modified makefile, however when I try it on my actual code (which I cannot share, unfortunately), I get an error. I was able to reproduce the error by just adding a string definition (again, unused here, sorry for the contrived code):

embed_example.cpp:

#include <julia.h>
#include <string>

int main(int argc, char *argv[])
{
    std::string teststr = "abc";
    /* required: setup the Julia context */
    jl_init();
    jl_value_t* ans = jl_eval_string("println(sqrt(2.0))");
    jl_atexit_hook(0);
    return 0;
}

Makefile:

JL_SHARE = $(shell julia -e 'print(joinpath(Sys.BINDIR, Base.DATAROOTDIR, "julia"))')
CFLAGS   += $(shell $(JL_SHARE)/julia-config.jl --cflags)
CXXFLAGS += $(shell $(JL_SHARE)/julia-config.jl --cflags)
LDFLAGS  += $(shell $(JL_SHARE)/julia-config.jl --ldflags)
LDLIBS   += $(shell $(JL_SHARE)/julia-config.jl --ldlibs)
LIBSTDCXX = $(shell julia -E 'joinpath(dirname(Sys.BINDIR), "lib", "julia", "libstdc++.so.6")')

all: embed_example

embed_example: embed_example.cpp
	$(CXX) $(CXXFLAGS) $(LDFLAGS) $< $(LDLIBS) -nodefaultlibs $(LIBSTDCXX) -lc -o $@

make output:

g++ -std=gnu99 -I'/opt/julia-1.5.3/include/julia' -fPIC -L'/opt/julia-1.5.3/lib' -Wl,--export-dynamic embed_example.cpp -Wl,-rpath,'/opt/julia-1.5.3/lib' -Wl,-rpath,'/opt/julia-1.5.3/lib/julia' -ljulia -nodefaultlibs "/opt/julia-1.5.3/lib/julia/libstdc++.so.6" -lc -o embed_example
cc1plus: warning: command line option ‘-std=gnu99’ is valid for C/ObjC but not for C++ [enabled by default]
/usr/bin/ld: /tmp/ccqNnBOC.o: undefined reference to symbol '_Unwind_Resume@@GCC_3.0'
/opt/julia-1.5.3/lib/julia/libgcc_s.so.1: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make: *** [embed_example] Error 1

giordano · February 13, 2021, 7:13pm

I created a gist. This Makefile file works for me:

JL_SHARE  = $(shell julia -E 'joinpath(Sys.BINDIR, Base.DATAROOTDIR, "julia")')
LIBSTDCXX = $(shell julia -E 'joinpath(dirname(Sys.BINDIR), "lib", "julia", "libstdc++.so.6")')
LIBGCC_S  = $(shell julia -E 'joinpath(dirname(Sys.BINDIR), "lib", "julia", "libgcc_s.so.1")')
CFLAGS   += $(shell $(JL_SHARE)/julia-config.jl --cflags)
CXXFLAGS += $(shell $(JL_SHARE)/julia-config.jl --cflags)
LDFLAGS  += $(shell $(JL_SHARE)/julia-config.jl --ldflags)
LDLIBS   += $(shell $(JL_SHARE)/julia-config.jl --ldlibs) -nodefaultlibs $(LIBSTDCXX) -lc $(LIBGCC_S) 

all: embed_example

In the container:

[root@d19e7a2a4c20 ~]# make
g++ -std=gnu99 -I'/usr/local/include/julia' -fPIC -L'/usr/local/lib' -Wl,--export-dynamic embed_example.cpp -Wl,-rpath,'/usr/local/lib' -Wl,-rpath,'/usr/local/lib/julia' -ljulia -nodefaultlibs "/usr/local/lib/julia/libstdc++.so.6" -lc "/usr/local/lib/julia/libgcc_s.so.1" -o embed_example
cc1plus: warning: command line option '-std=gnu99' is valid for C/ObjC but not for C++ [enabled by default]
[root@d19e7a2a4c20 ~]# ./embed_example 
1.4142135623730951

GunnarFarneback · February 13, 2021, 7:55pm

An alternative approach that goes around these difficulties (in exchange for some other difficulties) is to load libjulia dynamically at runtime. See GitHub - GunnarFarneback/DynamicallyLoadedEmbedding.jl: Embed Julia with dynamical loading of libjulia at runtime. for information about that approach.

crbinz · February 14, 2021, 12:47am

Thanks very much - this works great for me. I guess I am still confused why we still need to specify the paths for libstdc++ and libgcc though, after including -Wl,-rpath,'/opt/julia-1.5.3/lib/julia, but I suspect that’s just more of my inexperience with gcc showing. I appreciate your help.

@GunnarFarneback - I will check this out, thanks.

giordano · February 14, 2021, 12:52am

I think because those standard libraries have higher priority compared to Julia’s library in the search paths. You can see with -Wl,--trace that if you ask for -lstdc++ the linker will always look for GCC’s libstdc++