Complex workaround needed. Why?

I need this workaround to avoid crashes due to version conflicts od shared libraries on Linux:

if [[ "$(uname -s)" == "Linux" ]]; then
    export FONTCONFIG_FILE=/etc/fonts/fonts.conf
    # On Ubuntu 22.04 the system has: OpenSSL 3.0.2 (too old), fontconfig 2.13 (missing
    # FcConfigSetDefaultSubstitute). The dynamic linker reuses already-loaded system libs
    # instead of the bundled ones, so we force the bundled versions in with LD_PRELOAD.
    # libiconv/libintl are absent on the system (glibc-internal on Ubuntu), so we add
    # their artifact dirs to LD_LIBRARY_PATH so dependency resolution finds them without
    # hitting circular-preload ordering problems.
    _PRELOADS=()
    _LIBDIRS=()

    # Get all Julia depot paths
    _DEPOT_PATHS=()
    if [[ -n "${JULIA_DEPOT_PATH:-}" ]]; then
        IFS=':' read -ra _DEPOT_PATHS <<< "$JULIA_DEPOT_PATH"
    else
        _DEPOT_PATHS=("$HOME/.julia")
    fi

    # libcrypto: override stale system version via LD_PRELOAD
    _BUNDLED=""
    for _depot in "${_DEPOT_PATHS[@]}"; do
        _BUNDLED=$(find "$_depot/artifacts" -maxdepth 3 -name "libcrypto.so.3" -path "*/lib/*" 2>/dev/null | head -1)
        [[ -n "$_BUNDLED" ]] && break
    done
    [[ -n "$_BUNDLED" ]] && _PRELOADS+=("$_BUNDLED")

    # libiconv + libintl: absent on Ubuntu 22.04, add dirs to LD_LIBRARY_PATH so
    # circular dep between them resolves naturally without preload ordering issues.
    for _libname in libiconv.so.2 libintl.so.8; do
        _BUNDLED=""
        for _depot in "${_DEPOT_PATHS[@]}"; do
            _BUNDLED=$(find "$_depot/artifacts" -maxdepth 3 -name "$_libname" -path "*/lib/*" 2>/dev/null | head -1)
            [[ -n "$_BUNDLED" ]] && break
        done
        if [[ -n "$_BUNDLED" ]]; then
            _dir=$(dirname "$_BUNDLED")
            [[ ":${_LIBDIRS[*]}:" != *":$_dir:"* ]] && _LIBDIRS+=("$_dir")
        fi
    done

    # libfontconfig: override stale system version via LD_PRELOAD; also add its dir to
    # LD_LIBRARY_PATH so its deps (libiconv etc.) are found during preload resolution.
    # Pick the version that exports FcConfigSetDefaultSubstitute (needed by Pango β‰₯ 1.57).
    _BUNDLED=""
    for _depot in "${_DEPOT_PATHS[@]}"; do
        for _fc in $(find "$_depot/artifacts" -maxdepth 3 -name "libfontconfig.so.1" -path "*/lib/*" 2>/dev/null); do
            if nm -D "$_fc" 2>/dev/null | grep -q FcConfigSetDefaultSubstitute; then
                _BUNDLED="$_fc"
                break 2
            fi
        done
    done
    if [[ -n "$_BUNDLED" ]]; then
        _dir=$(dirname "$_BUNDLED")
        [[ ":${_LIBDIRS[*]}:" != *":$_dir:"* ]] && _LIBDIRS+=("$_dir")
        _PRELOADS+=("$_BUNDLED")
    fi

    [[ ${#_PRELOADS[@]} -gt 0 ]] && export LD_PRELOAD=$(IFS=:; echo "${_PRELOADS[*]}")
    if [[ ${#_LIBDIRS[@]} -gt 0 ]]; then
        _extra=$(IFS=:; echo "${_LIBDIRS[*]}")
        export LD_LIBRARY_PATH="${_extra}${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
    fi
    unset _PRELOADS _LIBDIRS _BUNDLED _DEPOT_PATHS _depot _fc _dir _extra _libname
fi
echo "LD_PRELOAD was: $LD_PRELOAD"
echo "LD_LIBRARY_PATH was: $LD_LIBRARY_PATH"

# On Linux, wrap julia to also preload the bundled libglib for Julia processes only.
# The system libglib is often too old for GLMakie/Glib_jll (missing g_string_copy).
# Scoping this to Julia calls prevents breaking system commands (uname, git, sed)
# which would fail if artifact libglib's transitive deps (libintl) are not satisfied.
if [[ "$(uname -s)" == "Linux" ]]; then
    _JULIA_GLIB_PRELOADS=""
    for _depot in $(echo "${JULIA_DEPOT_PATH:-$HOME/.julia}" | tr ':' ' '); do
        _f=$(find "$_depot/artifacts" -maxdepth 3 -name "libglib-2.0.so.0" -path "*/lib/*" 2>/dev/null | head -1)
        if [[ -n "$_f" ]]; then
            _glib_dir=$(dirname "$_f")
            _intl=""
            [[ -f "$_glib_dir/libintl.so.8" ]] && _intl="$_glib_dir/libintl.so.8:"
            _JULIA_GLIB_PRELOADS="${_intl}$_f"
            break
        fi
    done
    unset _depot _f _glib_dir _intl
    if [[ -n "$_JULIA_GLIB_PRELOADS" ]]; then
        julia() {
            LD_PRELOAD="${LD_PRELOAD:+$LD_PRELOAD:}$_JULIA_GLIB_PRELOADS" command julia "$@"
        }
    fi
fi

Why is this needed for KiteModels.jl?

Possible reasons:

  • conflicts between Conda, PyPlot and Makie
  • differences between CI and local machine
  • the use of PackageCompiler

Still, I think there must be something very wrong if I need such a workaround. Any ideas what?

Might be clearer if you explained the issue you’re trying to solve, rather than the overcomplicated workaround you (?) came up with.

1 Like

You can see what happens without this workaround in the failed CI runs here: here

SciMLBase β†’ SciMLBaseMakieExt 

Failed to precompile SciMLBaseMakieExt [565f26a4-c902-5eae-92ad-e10714a9d9de] to "/tmp/testdepot/compiled/v1.12/SciMLBaseMakieExt/jl_SBQvyJ".
ERROR: LoadError: InitError: could not load library "/tmp/testdepot/artifacts/654863f618f22eeb74ce2d1b4996666e65e596e4/lib/libgio-2.0.so"
/tmp/testdepot/artifacts/654863f618f22eeb74ce2d1b4996666e65e596e4/lib/libgobject-2.0.so.0: undefined symbol: g_string_copy
Stacktrace:
  [1] #dlopen#3
    @ ./libdl.jl:120 [inlined]
  [2] dlopen(s::String, flags::UInt32)
    @ Base.Libc.Libdl ./libdl.jl:119
  [3] macro expansion

and here:


β”Œ KiteViewers
β”‚  β”Œ Warning: Could not find font /usr/share/fonts/truetype/freefont/FreeMono.ttf, using TeX Gyre Heros Makie
β”‚  β”” @ Makie /tmp/testdepot/packages/Makie/Vn16E/src/conversions.jl:1456
β””  
[ Info: PackageCompiler: Executing /tmp/test/KiteModels.jl/test/test_for_precompile.jl => /tmp/jl_packagecompiler_C8f6P5/jl_I0qAVG
ERROR: LoadError: InitError: could not load library "/tmp/testdepot/artifacts/b2e490a224990d92e99cc4463b8df48c8c454971/lib/libpangocairo-1.0.so"
/tmp/testdepot/artifacts/b2e490a224990d92e99cc4463b8df48c8c454971/lib/libpangoft2-1.0.so.0: undefined symbol: FcConfigSetDefaultSubstitute
Stacktrace:

and here

[ Info: PackageCompiler: Executing /tmp/test/KiteModels.jl/test/test_for_precompile.jl => /tmp/jl_packagecompiler_H4vwzh/jl_qyu1Y9
ERROR: LoadError: InitError: could not load library "/tmp/testdepot/artifacts/b2e490a224990d92e99cc4463b8df48c8c454971/lib/libpangocairo-1.0.so"
/tmp/testdepot/artifacts/b2e490a224990d92e99cc4463b8df48c8c454971/lib/libpangoft2-1.0.so.0: undefined symbol: FcConfigSetDefaultSubstitute

Summary

Errors when creating a system image due to version conflicts of shared libraries:

library error
libpangocairo-1.0.so undefined symbol: FcConfigSetDefaultSubstitute
libgio-2.0.so, libgobject-2.0.so.0 undefined symbol: g_string_copy
libpangoft2-1.0.so.0 undefined symbol: FcConfigSetDefaultSubstitute

These errors happen both locally and on CI. The workaround is to pre-load the Julia versions of these libraries. But why is this workaround needed ? Is there a bug in one of the packages?

This looks like Makie Precompilation failed in Linux and should be fixed by now.

1 Like