Link Errors while building master[17d5c55]: undefined reference to symbol '__cxa_free_exception@@CXXABI_1.3

General Usage
1

I can successfully build 0.6-rc3 but I am unable to finish building the current origin/master [17d5c55].

I’ve put a full build log here: https://gist.github.com/JonathanAnderson/ca3b7d81da3f22974f6565b99308cb74

The overview of my process is as follows:

coreavx2 ❯ git clone https://github.com/JuliaLang/julia julia_master
Cloning into 'julia_master'...
remote: Counting objects: 367503, done.
remote: Compressing objects: 100% (177/177), done.
remote: Total 367503 (delta 88), reused 136 (delta 17), pack-reused 367224
Receiving objects: 100% (367503/367503), 166.74 MiB | 24.84 MiB/s, done.
Resolving deltas: 100% (301283/301283), done.

coreavx2 ❯ cat > julia_master/Make.user
override MARCH=core-avx2
override CC=/opt/rh/devtoolset-6/root/bin/gcc
override CXX=/opt/rh/devtoolset-6/root/bin/g++
CXXFLAGS=-std=c++11
CMAKE=/usr/bin/cmake3
USE_INTEL_MKL=1
USE_INTEL_MKL_FFT=1
USE_BLAS64=1
MKLROOT=/opt/radix/vendor-centos-7.2-gcc-6.2.1-20170510/packages/mkl/install

coreavx2 ❯ make -C julia_master

This gives me the following error when linking everything

    ...
    LINK usr/lib/libjulia.so.0.7.0
make[1]: Leaving directory `/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/src'
make[1]: Entering directory `/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/ui'
    CC ui/repl.o
    LINK usr/bin/julia
/opt/rh/devtoolset-6/root/usr/libexec/gcc/x86_64-redhat-linux/6.2.1/ld: /opt/rh/devtoolset-6/root/usr/lib/gcc/x86_64-redhat-linux/6.2.1/libstdc++_nonshared.a(functexcept48.o): undefined reference to symbol '__cxa_free_exception@@CXXABI_1.3'
/usr/lib64/libstdc++.so.6: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[1]: *** [/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/bin/julia] Error 1
make[1]: Leaving directory `/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/ui'
make: *** [julia-ui-release] Error 2
make: Leaving directory `/gpfs/tcs3/users/janders/shared/coreavx2/julia_master'

This is strange since 6.0-rc3 builds just fine. Any idea what is going on here or what I should try to do next?

2

Could be a link order issue. What is the output of make VERBOSE=1? (should only re-run, and show, the failing command)

1 Like
3 
make[1]: Entering directory `/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/deps'
make[1]: Nothing to be done for `default'.
make[1]: Leaving directory `/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/deps'
make[1]: Entering directory `/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/src'
make[1]: Nothing to be done for `julia_flisp.boot.inc.phony'.
make[1]: Leaving directory `/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/src'
make[1]: Entering directory `/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/src'
/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/contrib/install.sh 644 /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-linux.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-version.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-errno.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-threadpool.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-unix.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/julia/
/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/contrib/install.sh 644 /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-linux.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-version.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-errno.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-threadpool.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-unix.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/julia/
/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/contrib/install.sh 644 /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-linux.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-version.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-errno.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-threadpool.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-unix.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/julia/
/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/contrib/install.sh 644 /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-linux.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-version.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-errno.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-threadpool.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-unix.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/julia/
/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/contrib/install.sh 644 /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-linux.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-version.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-errno.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-threadpool.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/uv-unix.h /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include/julia/
make[1]: Leaving directory `/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/src'
make[1]: Entering directory `/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/ui'
 /opt/rh/devtoolset-6/root/bin/g++ -std=c++11 -pipe -fPIC -fno-rtti -pedantic   -O3 -ggdb2 -falign-functions -momit-leaf-frame-pointer -I/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/src -I/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/src -I/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/src/support -I/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include -Wall -Wno-strict-aliasing -fno-omit-frame-pointer -Wc++-compat repl.o -o /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/bin/julia -L/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/lib/julia -L/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/lib -L/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/lib -ljulia -Wl,-Bdynamic  -Wl,--no-whole-archive -Wl,--no-as-needed -ldl -lrt -lpthread -Wl,--export-dynamic,--as-needed,--no-whole-archive /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/lib/libunwind-generic.a /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/lib/libunwind.a -Wl,--version-script=/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/src/julia.expmap -Wl,-rpath,'$ORIGIN/../lib' -Wl,-rpath-link,/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/lib -Wl,-z,origin -Wl,-rpath,/opt/radix/vendor-centos-7.2-gcc-6.2.1-20170510/packages/mkl/install/lib/intel64
/opt/rh/devtoolset-6/root/usr/libexec/gcc/x86_64-redhat-linux/6.2.1/ld: /opt/rh/devtoolset-6/root/usr/lib/gcc/x86_64-redhat-linux/6.2.1/libstdc++_nonshared.a(functexcept48.o): undefined reference to symbol '__cxa_free_exception@@CXXABI_1.3'
/usr/lib64/libstdc++.so.6: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[1]: *** [/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/bin/julia] Error 1
make[1]: Leaving directory `/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/ui'
make: *** [julia-ui-release] Error 2
4

I decided to build 0.6 and to then rebase to master to see if that worked… In the process, I’ve noticed my build process for 0.6 isn’t using my sysadmin’s devtoolset-6 which is why I think that it works. I think my next steps are to figure out how to tell the build system where the correct libc++ is.

5

I’m trying again, this time I’ve removed the custom CXX and CC overrides and before I run make, I have run /opt/rh/devtoolset-6/enable which should make sure I am using the correct compiler and the right environment. I will report back in a few hours when this is done building.

1 Like
6

I ran into this same problem again. Running make VERBOSE=1 told me that the linker couldn’t find ‘__cxa_free_exception@@CXXABI_1.3’ in
libstdc++_nonshared.a

/opt/rh/devtoolset-6/root/usr/libexec/gcc/x86_64-redhat-linux/6.2.1/ld: /opt/rh/devtoolset-6/root/usr/lib/gcc/x86_64-redhat-linux/6.2.1/libstdc++_nonshared.a(functexcept48.o): undefined reference to symbol '__cxa_free_exception@@CXXABI_1.3'

This is the command that caused that error:

g++ -march=core-avx2 -m64 -std=c++11 -pipe -fPIC -fno-rtti -pedantic   -O3 -ggdb2 -falign-functions -momit-leaf-frame-pointer -I/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/src -I/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/src -I/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/src/support -I/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/include -Wall -Wno-strict-aliasing -fno-omit-frame-pointer -Wc++-compat repl.o -o /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/bin/julia -L/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/lib/julia -L/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/lib -L/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/lib -ljulia -Wl,-Bdynamic  -Wl,--no-whole-archive -Wl,--no-as-needed -ldl -lrt -lpthread -Wl,--export-dynamic,--as-needed,--no-whole-archive /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/lib/libunwind-generic.a /gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/lib/libunwind.a -Wl,--version-script=/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/src/julia.expmap -Wl,-rpath,'$ORIGIN/../lib' -Wl,-rpath-link,/gpfs/tcs3/users/janders/shared/coreavx2/julia_master/usr/lib -Wl,-z,origin -Wl,-rpath,/opt/radix/vendor-centos-7.2-gcc-6.2.1-20170510/packages/mkl/install/lib/intel64

I could manually get this command to run by adding -lstdc++ to the end of the command. After manually running the following I was able to re-run make and build the rest of the objects.

My Make.user looks like

override MARCH=core-avx2
CXXFLAGS=-std=c++11
CMAKE=/usr/bin/cmake3
USE_INTEL_MKL=1
USE_INTEL_MKL_FFT=1
USE_BLAS64=1
MKLROOT=/opt/radix/vendor-centos-7.2-gcc-6.2.1-20170510/packages/mkl/install

and the enable script that setup devtoolset-6 sets the following environment variables

export PATH=/opt/rh/devtoolset-6/root/usr/bin${PATH:+:${PATH}}
export MANPATH=/opt/rh/devtoolset-6/root/usr/share/man:${MANPATH}
export INFOPATH=/opt/rh/devtoolset-6/root/usr/share/info${INFOPATH:+:${INFOPATH}}
export PCP_DIR=/opt/rh/devtoolset-6/root
export LD_LIBRARY_PATH=/opt/rh/devtoolset-6/root$rpmlibdir$rpmlibdir32${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

I don’t know why it couldn’t find that symbol in the .a and I’m not sure why I had to add libstdc++.so but since it works, I’m trying one more build from scratch with LDFLAGS=-lstdc++ in my Make.user and see if that builds everything from scratch. Is there any harm from having to add the extra library?

7

The devtoolset is a little strange. It has an old system copy of the shared libstdc++ so, then puts everything newer in a separate static libstdc++_nonshared.a library and defaults to statically linking that part. There may be a workaround to get it to do the right thing, but I suspect this is causing the issue. This strange split partially static libstdc++ is why we tried but decided against using the devtoolset compiler to build the generic linux binaries on the Julia buildbots.

1 Like
8

This worked! Adding LDFLAGS=-lstdc++ to the Make.user did the trick.

However, I received a warning during configure:

configure: using LDFLAGS: -L/gpfs/tcs3/users/janders/shared/coreavx2/julia_master_again/usr/lib -lstdc++ -Wl,-rpath,'$$ORIGIN' -Wl,-z,origin -L/gpfs/tcs3/users/janders/shared/coreavx2/julia_master_again/usr/lib
configure: LDFLAGS note: LDFLAGS should only be used to specify linker flags, not libraries. Use LIBS for: -lstdc++
configure: WARNING: Continuing even with errors mentioned immediately above this line.

So I am also running another build where I add LIBS=-lstdc++ since thats what the LDFLAGS note recommended…

9

LIBS=-lstdc++ does not work.

I needed to source /opt/rh/devtoolset-6/enable and here is the working Make.user I used.

override MARCH=core-avx2
CXXFLAGS=-std=c++11
LDFLAGS=-lstdc++
CMAKE=/usr/bin/cmake3
USE_INTEL_MKL=1
USE_INTEL_MKL_FFT=1
USE_BLAS64=1
MKLROOT=/opt/radix/vendor-centos-7.2-gcc-6.2.1-20170510/packages/mkl/install
10

Maybe worth making a pull-request to add a note about that to the linux section of the README (save the next person some time).

1 Like
11

Just created pull request 22448

2 Likes