Precedence of local and julia-shipped shared libraries

General Usage
1

I am working together with @sloede on libtrixi, a C/Fortran-interface library to the flow solver Trixi.jl. @sloede managed to create a standalone library using PackageCompiler.jl, see How do libraries compiled by PackageCompiler.jl interact with Preferences.jl?
When using it, I encountered a problem with different versions of local and julia-shipped libraries. In detail:

  • Our Julia project uses Preferences.jl to create LocalPreferences.toml with

    [T8code]
libt8 = "/home/install/t8code/lib/libt8.so"

  • This library was compiled locally and links against libstdc++:

    $ ldd /home/install/t8code/lib/libt8.so
      libstdc++.so.6 => /usr/lib/libstdc++.so.6

    On my machine this is libstdc++.so.6.0.32.

  • PackageCompiler.jl fetches libstdc++.so.6.0.30

  • When executing an example using the PackageCompiler.jl standalone library, we get an error

    InitError(mod=:Trixi, error=ErrorException("could not load library "/home/install/t8code/lib/libt8.so" 
/home/libtrixi/LibTrixi.jl/lib/build/lib/julia/libstdc++.so.6: version `GLIBCXX_3.4.32' not found   (required by /home/install/t8code/lib/libt8.so)"))

    because GLIBCXX_3.4.32 is only found in the system libstdc++.so.6.0.32, but not in the julia-shipped libstdc++.so.6.0.30.

  • We can work around this by using LD_PRELOAD=/usr/lib/libstdc++.so.6.0.32

Now we wonder whether it be possible to detect and avoid such conflicts. Would it possible to override the julia-shipped libraries during PackageCompiler.jl runs?

cc @giordano @sloede @vchuravy

2

What version of Julia are you using?

3

Sorry, I forgot to mention that. julia 1.9.3

4

Julia v1.9.3 is able to pick up libstdc++ newer than the one shipped by Julia:

% julia -q
julia> using Libdl

julia> filter!(contains("libstdc++"), dllist())
1-element Vector{String}:
 "/usr/lib/libstdc++.so.6"

I have no idea what PackageCompiler does.

1 Like
5

I am not even sure if this is related to PackageCompiler.jl or just becomes visible here because the load order is different here than it would be for a run where julia is the main executable (although I don’t see how).

It seems like libstdc++ is required by libjulia-internal.so.1 only, and that has $ORIGIN:$ORIGIN/.. set as the RUNPATH. So one - crude - solution could be to

  • create a new folder PREFIX/lib/julia/override
  • use patchelf to change the RUNPATH for libjulia-internal.so.1 to $ORIGIN/../julia/override:$ORIGIN:$ORIGIN/..,
  • and then symlink /usr/lib/libstdc++.so.6 to PREFIX/lib/julia/override/libstdc++.so.6.

But it feels like there should exist a better way to handle that.

For user convenience, we are looking for a solution that either prevents this problem in the first place by fixing whatever is wrong in PackageCompiler.jl/our own compilation part, or that can be (semi-)auto-detected and fixed in a post processing step. We do not want the user to have to recompile with PackageCompiler.jl (too long) nor force them to always start their problems with LD_PRELOAD=... (too error prone).

6

We have some magic in the Julia loader that basically tries to load the first libstdc++.so that it can find (and this is being run from the julia executable, which does not have RUNPATH set, so it picks up the system-wide libstdc++ if it exists), checks to see if that version is new enough to run Julia, and if so, uses that. If that system-wide libstdc++ does not exist or is too old, we will instead load the bundled libstdc++.

The “correct” way to do this is probably to copy this libstdcxxprobe() functionality into PackageCompiler.jl.

7

Interesting :thinking: So you’re saying that Julia will always dynamically determine at startup which is the “best” libstdc++.so so load?

While this makes sense when using Julia as the main program, I think for a compiled library it would be sufficient if the correct libstdc++.so would be determined (and fixed) at build time. To this effect, I don’t think we would have to copy all the C code over to PackageCompiler.jl, but rather just figure out WWJD (what would Julia do) by running something like the following from PackageCompiler.jl:

using Libdl
libstdcpp_path = filter!(contains("libstdc++"), dllist()) |> first |> abspath

One could then determine whether libstdcpp_path points to the private libjulia dir (julia_private_libdir() in PC.jl) and proceed based on the outcome:

  • if yes (e.g., /opt/julia/1.9.3/lib/julia/libstdc++.so.6), copy over libstdc++ from Julia as usual
  • if no (e.g., /usr/lib/x86_64-linux-gnu/libstdc++.so.6), create a symlink to it in the private libjulia dir

Would this be a sensible option/default for PackageCompiler.jl?

8

I think it could make sense to “bake in” the choice of libstdc++ that Julia has made; although I don’t think symlinks are a good idea, since presumably if someone is using PackageCompiler.jl they will want to redistribute the package.

9

I think we are leaving the realm of true portability across systems here anyways. OTOH, libstdc++.so is <5 MiB, so this “macht den Kohl auch nicht mehr fett” (this doesn’t fatten the cabbage, as the Germans say; i.e., by now it does not matter anymore anyways [looking suspiously at you, 500 MiB libtrixi.so]).

Thanks for your feedback. I’ll give it a try to see how ugly this would get in PC.jl.

10

Really? I think that is one of the primary use cases for PackageCompiler.jl! I’m curious what you’re using this for if you’re not planning to ship the built library?

11

Maybe I should qualify my comment first: bundling a system-local libstdc++ from one Linux system and hoping it will just work anywhere else, is IMHO quite daring (but I’m no expert, maybe this is actually a sane thing to do/expect).

We are developing the libtrixi library as a means to allowing one to control numerical simulations with Trixi.jl from C/C++/Fortran programs. Since libtrixi relies on system-local installations of third-party libraries such as MPI, HDF5, and others, and since the paths to these libraries are usually baked into the compiled library as compile-time preferences, I have zero expectation of system portability for our use case.

But I think that’s OK; in our community nobody expects binaries to be portable.

Incidentally, do you know why Julia’s libstdc++ is ~18 MB and Ubuntu’s is just 2 MB?

12

Incidentally, do you know why Julia’s libstdc++ is ~18 MB and Ubuntu’s is just 2 MB?

If you strip it, it gets much closer (2.5 MB). We usually do that automatically for BB-built binaries, but the compiler libraries are special, so they don’t get the same treatment; that would be a good improvement.

13

Do we? :thinking: If we did, libstdc++ would be stripped in CompilerSupportLibraries_jll.

14

I’ve created a PR to PackageCompiler.jl: Bundle dynamically selected libstdc++ by sloede · Pull Request #853 · JuliaLang/PackageCompiler.jl · GitHub

1 Like