Embedding julia: libLLVM-9jl.so not found

I’m trying to compile a simple C++ program that embeds some julia code. I’ve gotten this to work on macOS, but when trying to compile on my CentOS machine, I get this warning, followed by lots of errors:

/usr/bin/ld: warning: libLLVM-9jl.so, needed by /opt/julia-1.5.3/lib/libjulia.so, not found (try using -rpath or -rpath-link)

I compile with g++ -o jl_test -fPIC -I/opt/julia-1.5.3/include/julia -L/opt/julia-1.5.3/lib -Wl,-rpath,/opt/julia-1.5.3/lib src/embedded_julia.cpp -ljulia

Does anyone have any suggestions for fixing this or further diagnosing the issue?

Instead of rpath I would try -L/opt/julia-1.5.3/lib/julia that’s where libLLVM-9jl.so lives on my computer.

1 Like

Thanks, that was ultimately the issue, which I fixed simply by using the makefile in the docs. Now, though, I am getting many errors like this:

/opt/julia-1.5.3/lib/libjulia.so: undefined reference to `vtable for std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21'

followed by, at the end:

collect2: error: ld returned 1 exit status
make: *** [<builtin>: src/embedded_julia] Error 1

I’m having a hard time figuring out what this could be. I thought that maybe I need a newer version of gcc/g++, since strings /lib64/libsdtc++.so.6 | grep GLIBCXX does not show GLIBCXX_3.4.21 (mentioned in the errors), but I tried updating to gcc v8.3.1 and I’m running into the same issues…

EDIT: I see now that /opt/julia-1.5.3/lib/julia contains its own libstdc++, which does show the necessary GLIBCXX version, so I’m at a loss…

I tried to build the embedding example in a Centos 7 Docker container:

FROM centos:7

RUN yum install -y gcc make

RUN curl -sL "https://julialang-s3.julialang.org/bin/linux/x64/1.5/julia-1.5.3-linux-x86_64.tar.gz" | tar xzf - -C /usr/local --strip-components=1

COPY embed_example.c Makefile /root/

WORKDIR "/root"

Inside the container:

[root@5a985425fcb5 ~]# gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[root@5a985425fcb5 ~]# make
cc -std=gnu99 -I'/usr/local/include/julia' -fPIC  -L'/usr/local/lib' -Wl,--export-dynamic  embed_example.c  -Wl,-rpath,'/usr/local/lib' -Wl,-rpath,'/usr/local/lib/julia' -ljulia -o embed_example
[root@5a985425fcb5 ~]# ./embed_example 
1.4142135623730951
[root@5a985425fcb5 ~]# ldd embed_example
        linux-vdso.so.1 =>  (0x00007ffc985ba000)
        libjulia.so.1 => /usr/local/lib/libjulia.so.1 (0x00007f7541713000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f7541345000)
        libunwind.so.8 => /usr/local/lib/julia/libunwind.so.8 (0x00007f754110c000)
        libLLVM-9jl.so => /usr/local/lib/julia/libLLVM-9jl.so (0x00007f753d9a9000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f753d7a5000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f753d59d000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f753d381000)
        libstdc++.so.6 => /usr/local/lib/julia/libstdc++.so.6 (0x00007f753d003000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f753cd01000)
        libgcc_s.so.1 => /usr/local/lib/julia/libgcc_s.so.1 (0x00007f753cae9000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f7541ee2000)

Can you provide more information about your environment?

I’m wondering if you should remove the “normal” lib paths when doing the link. I’m not sure how to do that. Maybe setting LD_LIBRARY_PATH to an empty string?

I’m fairly certain this is the relevant difference. I found this thread, which describes the problem I was having (@pixel27, you were on the right track), so I’m trying to follow the advice there.

Unfortunately, now I keep running into /opt/julia-1.5.3/lib/julia/libstdc++.so.6: error adding symbols: DSO missing from command line, and I haven’t been able to figure that out yet…

Actually, I get the same results as you when I try to compile the © example… my problems arise when I try to compile a C++ example…

If you actually share your code it may be easier for someone to try and reproduce your problem :slightly_smiling_face:

Sure, good point. Here’s the entire simplified C++ file:

#include <julia.h>

int main(int argc, char *argv[])
{
    /* required: setup the Julia context */
    jl_init();
    jl_value_t* ans = jl_eval_string("println(sqrt(2.0))");    
    jl_atexit_hook(0);    
    return 0;
}

And here is the output of make, using the Makefile just as it appears in the docs.

This is so simplified that is basically C, isn’t it?

I guess so… it’s just the minimum (non)working example I came up with.

I think you might have gone too simple.

I installed CentOS Stream in a VirtualBox VM then downloaded Julia 1.5.3 and just extracted it into my home directory, .bashrc was updated to add $HOME/julia-1.5.3/bin to the path.

I started with the Makefile you specified and added the compile commands (the C file I saved as test.c) which ended up with:

JL_SHARE = $(shell julia -e 'print(joinpath(Sys.BINDIR, Base.DATAROOTDIR, "julia"))')
CFLAGS   += $(shell $(JL_SHARE)/julia-config.jl --cflags)
CXXFLAGS += $(shell $(JL_SHARE)/julia-config.jl --cflags)
LDFLAGS  += $(shell $(JL_SHARE)/julia-config.jl --ldflags)
LDLIBS   += $(shell $(JL_SHARE)/julia-config.jl --ldlibs)


all:
	julia -e 'using InteractiveUtils;versioninfo()'
	gcc -o test $(CFLAGS) $(LDFLAGS) $(LDLIBS) test.c

I didn’t include the CXXFLAGS because there is no C++ in that file :slight_smile: . My make works fine:

[josh@centos ~]$ make
julia -e 'using InteractiveUtils;versioninfo()'
Julia Version 1.5.3
Commit 788b2c77c1 (2020-11-09 13:37 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-3820 CPU @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, sandybridge)
gcc -o test -std=gnu99 -I'/home/josh/julia-1.5.3/include/julia' -fPIC -L'/home/josh/julia-1.5.3/lib' -Wl,--export-dynamic -Wl,-rpath,'/home/josh/julia-1.5.3/lib' -Wl,-rpath,'/home/josh/julia-1.5.3/lib/julia' -ljulia test.c

And the program can run:

[josh@centos ~]$ ./test 
1.4142135623730951

Thanks - to be clear, when I try to compile the C test case, it works for me too. My end goal is not to compile this particular toy problem - I have a need to embed the julia code in some (actual) C++ code. I provided the toy problem because it gives me the same errors as my actual C++ example.

Dockerfile:

FROM centos:7

RUN yum install -y gcc gcc-c++ make

RUN curl -sL "https://julialang-s3.julialang.org/bin/linux/x64/1.5/julia-1.5.3-linux-x86_64.tar.gz" | tar xzf - -C /usr/local --strip-components=1

COPY embed_example.cpp Makefile /root/

WORKDIR "/root"

embed_example.cpp:

#include <julia.h>

int main(int argc, char *argv[])
{
    /* required: setup the Julia context */
    jl_init();
    jl_value_t* ans = jl_eval_string("println(sqrt(2.0))");    
    jl_atexit_hook(0);    
    return 0;
}

Makefile:

JL_SHARE = $(shell julia -e 'print(joinpath(Sys.BINDIR, Base.DATAROOTDIR, "julia"))')
CFLAGS   += $(shell $(JL_SHARE)/julia-config.jl --cflags)
CXXFLAGS += $(shell $(JL_SHARE)/julia-config.jl --cflags)
LDFLAGS  += $(shell $(JL_SHARE)/julia-config.jl --ldflags)
LDLIBS   += $(shell $(JL_SHARE)/julia-config.jl --ldlibs)
LIBSTDCXX = $(shell julia -E 'joinpath(dirname(Sys.BINDIR), "lib", "julia", "libstdc++.so.6")')

all: embed_example

embed_example: embed_example.cpp
	$(CXX) $(CXXFLAGS) $(LDFLAGS) $< $(LDLIBS) -nodefaultlibs $(LIBSTDCXX) -lc -o $@

In the container:

[root@ab82e3a1343c ~]# make
g++ -std=gnu99 -I'/usr/local/include/julia' -fPIC -L'/usr/local/lib' -Wl,--export-dynamic embed_example.cpp -Wl,-rpath,'/usr/local/lib' -Wl,-rpath,'/usr/local/lib/julia' -ljulia -nodefaultlibs "/usr/local/lib/julia/libstdc++.so.6" -lc -o embed_example
cc1plus: warning: command line option '-std=gnu99' is valid for C/ObjC but not for C++ [enabled by default]
[root@ab82e3a1343c ~]# ./embed_example 
1.4142135623730951
[root@ab82e3a1343c ~]# ldd ./embed_example 
        linux-vdso.so.1 =>  (0x00007ffe54cc2000)
        libjulia.so.1 => /usr/local/lib/libjulia.so.1 (0x00007fa1defbb000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fa1debed000)
        libunwind.so.8 => /usr/local/lib/julia/libunwind.so.8 (0x00007fa1de9b4000)
        libLLVM-9jl.so => /usr/local/lib/julia/libLLVM-9jl.so (0x00007fa1db251000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fa1db04d000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fa1dae45000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa1dac29000)
        libstdc++.so.6 => /usr/local/lib/julia/libstdc++.so.6 (0x00007fa1da8ab000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fa1da5a9000)
        libgcc_s.so.1 => /usr/local/lib/julia/libgcc_s.so.1 (0x00007fa1da391000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fa1df78a000)

The trick (suggested by @staticfloat) is to use -nodefaultlibs to not use the standard libraries (both C and C++), specify the full path to Julia’s libstdc++ and then use system’s libc.

Side note: this oversimplified example which is pure C would compile and run also without passing libstdc++ to the compiler, and the executable will not link to it. This is fine because the program is pure C. libllvm needs libstdc++, but it’ll look for $$ORIGIN/libstdc++ and find the one sitting next to it.

1 Like

First of all, thanks again for your help, I appreciate it since I am fairly out of my depth here. I can indeed compile my original simple example with the modified makefile, however when I try it on my actual code (which I cannot share, unfortunately), I get an error. I was able to reproduce the error by just adding a string definition (again, unused here, sorry for the contrived code):

embed_example.cpp:

#include <julia.h>
#include <string>

int main(int argc, char *argv[])
{
    std::string teststr = "abc";
    /* required: setup the Julia context */
    jl_init();
    jl_value_t* ans = jl_eval_string("println(sqrt(2.0))");
    jl_atexit_hook(0);
    return 0;
}

Makefile:

JL_SHARE = $(shell julia -e 'print(joinpath(Sys.BINDIR, Base.DATAROOTDIR, "julia"))')
CFLAGS   += $(shell $(JL_SHARE)/julia-config.jl --cflags)
CXXFLAGS += $(shell $(JL_SHARE)/julia-config.jl --cflags)
LDFLAGS  += $(shell $(JL_SHARE)/julia-config.jl --ldflags)
LDLIBS   += $(shell $(JL_SHARE)/julia-config.jl --ldlibs)
LIBSTDCXX = $(shell julia -E 'joinpath(dirname(Sys.BINDIR), "lib", "julia", "libstdc++.so.6")')

all: embed_example

embed_example: embed_example.cpp
	$(CXX) $(CXXFLAGS) $(LDFLAGS) $< $(LDLIBS) -nodefaultlibs $(LIBSTDCXX) -lc -o $@

make output:

g++ -std=gnu99 -I'/opt/julia-1.5.3/include/julia' -fPIC -L'/opt/julia-1.5.3/lib' -Wl,--export-dynamic embed_example.cpp -Wl,-rpath,'/opt/julia-1.5.3/lib' -Wl,-rpath,'/opt/julia-1.5.3/lib/julia' -ljulia -nodefaultlibs "/opt/julia-1.5.3/lib/julia/libstdc++.so.6" -lc -o embed_example
cc1plus: warning: command line option ‘-std=gnu99’ is valid for C/ObjC but not for C++ [enabled by default]
/usr/bin/ld: /tmp/ccqNnBOC.o: undefined reference to symbol '_Unwind_Resume@@GCC_3.0'
/opt/julia-1.5.3/lib/julia/libgcc_s.so.1: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make: *** [embed_example] Error 1

I created a gist. This Makefile file works for me:

JL_SHARE  = $(shell julia -E 'joinpath(Sys.BINDIR, Base.DATAROOTDIR, "julia")')
LIBSTDCXX = $(shell julia -E 'joinpath(dirname(Sys.BINDIR), "lib", "julia", "libstdc++.so.6")')
LIBGCC_S  = $(shell julia -E 'joinpath(dirname(Sys.BINDIR), "lib", "julia", "libgcc_s.so.1")')
CFLAGS   += $(shell $(JL_SHARE)/julia-config.jl --cflags)
CXXFLAGS += $(shell $(JL_SHARE)/julia-config.jl --cflags)
LDFLAGS  += $(shell $(JL_SHARE)/julia-config.jl --ldflags)
LDLIBS   += $(shell $(JL_SHARE)/julia-config.jl --ldlibs) -nodefaultlibs $(LIBSTDCXX) -lc $(LIBGCC_S) 

all: embed_example

In the container:

[root@d19e7a2a4c20 ~]# make
g++ -std=gnu99 -I'/usr/local/include/julia' -fPIC -L'/usr/local/lib' -Wl,--export-dynamic embed_example.cpp -Wl,-rpath,'/usr/local/lib' -Wl,-rpath,'/usr/local/lib/julia' -ljulia -nodefaultlibs "/usr/local/lib/julia/libstdc++.so.6" -lc "/usr/local/lib/julia/libgcc_s.so.1" -o embed_example
cc1plus: warning: command line option '-std=gnu99' is valid for C/ObjC but not for C++ [enabled by default]
[root@d19e7a2a4c20 ~]# ./embed_example 
1.4142135623730951
1 Like

An alternative approach that goes around these difficulties (in exchange for some other difficulties) is to load libjulia dynamically at runtime. See GitHub - GunnarFarneback/DynamicallyLoadedEmbedding.jl: Embed Julia with dynamical loading of libjulia at runtime. for information about that approach.

Thanks very much - this works great for me. I guess I am still confused why we still need to specify the paths for libstdc++ and libgcc though, after including -Wl,-rpath,'/opt/julia-1.5.3/lib/julia, but I suspect that’s just more of my inexperience with gcc showing. I appreciate your help.

@GunnarFarneback - I will check this out, thanks.

I think because those standard libraries have higher priority compared to Julia’s library in the search paths. You can see with -Wl,--trace that if you ask for -lstdc++ the linker will always look for GCC’s libstdc++

1 Like