Missing shared library file when pre-compiling

acnash · October 4, 2023, 1:52pm

Hi all, I’m new to Julia but “old” regarding software development.

I’m trying to add the package BlackBoxOptim for some Julia code, but I’m having problems finding a shared C library file, “libmbedtls.so”. As far as I’m aware, the BlackBoxOptim Julia package, has a set of dependencies, one of them, MBed TLS, is the source of the missing shared C library.

I’m running Julia in a new and untouched Conda environment. It is a clean box. I added BlackBoxOptim to the julia package manager (I’m new to Julia - forgive any use of incorrect terms), but on importing the package - which pre-compiles it - I see the error:

julia> import BlackBoxOptim
[ Info: Precompiling BlackBoxOptim [a134a8b2-14d6-55f6-9291-3336d3ab0209]
ERROR: LoadError: InitError: could not load library “libmbedtls.so”
libmbedx509.so.4: cannot open shared object file: No such file or directory
Stacktrace:
[1] dlopen(s::String, flags::UInt32; throw_error::Bool)

Within my conda environment, I found the file:

/home/shodan/miniconda3/envs/julia_ness/lib/libmbedtls.so
/home/shodan/miniconda3/envs/julia_ness/lib/julia/libmbedtls.so
/home/shodan/miniconda3/pkgs/mbedtls-3.3.0-hcb278e6_0/lib/libmbedtls.so
/home/shodan/miniconda3/pkgs/julia-1.9.3-h06b7c97_0/lib/julia/libmbedtls.so

I’m new to Julia as of 1 hour ago. How do I point Julia in the direction of the correct location for this shared library file?

Many thanks and I appreciate your patience.

mkitti · October 4, 2023, 2:35pm

Did you install Julia via conda-forge? If so, from which channel? Is this on Linux?

If so, please raise this as an issue on the julia-feedstock. I am personally trying to maintain this but navigating the line between Julia and conda-forge is a bit tricky.

In particular we are trying to use mbedTLS from conda-forge: USE_SYSTEM_MBEDTLS=1.

github.com

conda-forge/julia-feedstock/blob/main/recipe/build.sh#L17-L19


      
          elif [[ "${target_platform}" == linux-* ]]; then
              export EXTRA_MAKEFLAGS="USE_SYSTEM_LIBGIT2=1 USE_SYSTEM_MBEDTLS=1"
          fi

I believe there is an issue with the configuration of the Julia standard library MbedTLS_jll.jl which must be done at build time.

The short term fix is to download Julia directly from julialang.org or to use juliaup.

A hacky fix is to manually locate the library and its dependencies in the the conda environment and manually load them via Libdl.dlopen in the correct order.

mkitti · October 4, 2023, 2:52pm

To confirm my diagnosis and isolate the error try this:

julia> using MbedTLS_jll

johnh · October 4, 2023, 2:52pm

Using juliap is very good advice. Give that a try.

acnash · October 4, 2023, 2:54pm

I see! Thanks so much for your feedback. It’s very helpful.

Yes, this is Ubuntu Linux (well, strictly, it’s WSL2), and I installed Julia using Conda via the conda-forge channel.

If I understand you correctly, the robust solution would be to delete this Conda environment, start a new one (so it’s a clean box), and use a downloaded version of Julia from the website rather than what Conda provides.

Thanks

acnash · October 4, 2023, 2:55pm

I’m seeing:

ERROR: InitError: could not load library “libmbedtls.so”
libmbedx509.so.4: cannot open shared object file: No such file or directory
Stacktrace:
[1] dlopen(s::String, flags::UInt32; throw_error::Bool)
@ Base.Libc.Libdl ./libdl.jl:117
[2] dlopen
@ ./libdl.jl:116 [inlined]
[3] dlopen(s::String)
@ Base.Libc.Libdl ./libdl.jl:116
[4] init()
@ MbedTLS_jll ~/miniconda3/envs/julia_ness/share/julia/stdlib/v1.9/MbedTLS_jll/src/MbedTLS_jll.jl:42
[5] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
@ Base ./loading.jl:1115
[6] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
@ Base ./loading.jl:1061
[7] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
@ Base ./loading.jl:1506
[8] _require(pkg::Base.PkgId, env::String)
@ Base ./loading.jl:1783
[9] _require_prelocked(uuidkey::Base.PkgId, env::String)
@ Base ./loading.jl:1660
[10] macro expansion
@ ./loading.jl:1648 [inlined]
[11] macro expansion
@ ./lock.jl:267 [inlined]
[12] require(into::Module, mod::Symbol)
@ Base ./loading.jl:1611
during initialization of module MbedTLS_jll

acnash · October 4, 2023, 2:56pm

I’ll give it a try. Thanks.

mkitti · October 4, 2023, 3:04pm

If you only objective is to use Julia, you do not need conda or a conda environment.

acnash · October 4, 2023, 3:11pm

To confirm, I resolved this by:

(1) Removing my old conda environment.
(2) Making a new conda (virtual environment).
(3) Installed Juliaup following the curl installation commandline.
(4) Reinitialised (exited and entered) my environment.
(5) Loaded Julia on the prompt.
(6) add BlackBoxOptim

Regarding your last question. I’m using Conda to keep environments separate as I’m working across multiple languages and various projects, each with its own dependencies.

Thank you so much for your help.

mkitti · October 4, 2023, 3:19pm

Yes, I understand why you are using conda. However, I’m trying to communicate that Julia is not going to respect your virtual environment boundaries out of the box. Additionally, Julia has its own environment and package system.

You will notice that Julia has created a depot outside of your $CONDA_PREFIX at ~/.julia. Basically, Julia knows nothing about your conda “virtual environment”.

You can address this partially by manually installing an activate script to let Julia operate within the prefix. For an example, see the following script.

github.com

conda-forge/julia-feedstock/blob/main/recipe/scripts/activate.sh

#!/bin/sh

export JULIA_DEPOT_PATH_BACKUP=${JULIA_DEPOT_PATH:-}
export JULIA_PROJECT_BACKUP=${JULIA_PROJECT:-}
export JULIA_LOAD_PATH_BACKUP=${JULIA_LOAD_PATH:-}

# Move ~/.julia to a conda environment specific location
export JULIA_DEPOT_PATH="$CONDA_PREFIX/share/julia:$JULIA_DEPOT_PATH"

# Create a named environment for each conda environment
# Use of @ to specify a shared named environment is new as of Julia 1.7
# https://github.com/JuliaLang/julia/pull/40025
# The name of the environment is the last directory of the CONDA_PREFIX
export JULIA_PROJECT="@${CONDA_PREFIX##*/}"
# Modify load path so that projects stack on the conda-named environment
export JULIA_LOAD_PATH="@:$JULIA_PROJECT:@stdlib"

# make julia use same cert
export JULIA_SSL_CA_ROOTS_PATH_BACKUP=${JULIA_SSL_CA_ROOTS_PATH:-}
if [[ $OSTYPE != 'darwin'* ]]; then

This file has been truncated. show original

Also see the corresponding deactivate script.

mkitti · October 4, 2023, 9:09pm

I created an issue here:

github.com/conda-forge/julia-feedstock

libmbedtls.so is missing

opened 09:06PM - 04 Oct 23 UTC

mkitti

bug

### Solution to issue cannot be found in the documentation. - [X] I checked the… documentation. ### Issue xref: https://discourse.julialang.org/t/missing-shared-library-file-when-pre-compiling/104575 ``` julia> using MbedTLS_jll ERROR: InitError: could not load library “libmbedtls.so” libmbedx509.so.4: cannot open shared object file: No such file or directory Stacktrace: [1] dlopen(s::String, flags::UInt32; throw_error::Bool) @ Base.Libc.Libdl ./libdl.jl:117 [2] dlopen @ ./libdl.jl:116 [inlined] [3] dlopen(s::String) @ Base.Libc.Libdl ./libdl.jl:116 [4] init() @ MbedTLS_jll ~/miniconda3/envs/julia_ness/share/julia/stdlib/v1.9/MbedTLS_jll/src/MbedTLS_jll.jl:42 [5] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String) @ Base ./loading.jl:1115 [6] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any}) @ Base ./loading.jl:1061 [7] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128) @ Base ./loading.jl:1506 [8] _require(pkg::Base.PkgId, env::String) @ Base ./loading.jl:1783 [9] _require_prelocked(uuidkey::Base.PkgId, env::String) @ Base ./loading.jl:1660 [10] macro expansion @ ./loading.jl:1648 [inlined] [11] macro expansion @ ./lock.jl:267 [inlined] [12] require(into::Module, mod::Symbol) @ Base ./loading.jl:1611 during initialization of module MbedTLS_jll ``` ### Installed packages ```shell n/a ``` ### Environment info ```shell n/a ```

Note that the conda-forge packaging is not an official packaging by the Julia project. At the moment it is a best effort exercise by me and and ngam. Volunteers are needed to help with this effort.

asan_emirsaleh · November 26, 2023, 3:49pm

Hello!
I faced the same issue.
library is in conda, OS is Arch

$ sudo find / -name libmbedx509.so.4
/home/asan/micromamba/pkgs/mbedtls-3.3.0-hcb278e6_0/lib/libmbedx509.so.4
/home/asan/miniconda3/pkgs/mbedtls-3.3.0-hcb278e6_0/lib/libmbedx509.so.4
/home/asan/miniconda3/envs/julia-env/lib/libmbedx509.so.4

While installing IJulia, error occured:

precompile
Precompiling project...
  ✗ MbedTLS
  ✗ IJulia
  0 dependencies successfully precompiled in 5 seconds. 14 already precompiled.

ERROR: The following 1 direct dependency failed to precompile:

IJulia [7073ff75-c697-5162-941a-fcdaad2a7d2a]

Failed to precompile IJulia [7073ff75-c697-5162-941a-fcdaad2a7d2a] to "/home/asan/miniconda3/envs/julia-env/share/julia/compiled/v1.9/IJulia/jl_puWdCi".
ERROR: LoadError: InitError: could not load library "libmbedtls.so"
libmbedx509.so.4: cannot open shared object file: No such file or directory
Stacktrace:
  [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
    @ Base.Libc.Libdl ./libdl.jl:117
  [2] dlopen
    @ ./libdl.jl:116 [inlined]
  [3] dlopen(s::String)
    @ Base.Libc.Libdl ./libdl.jl:116
  [4] __init__()
    @ MbedTLS_jll ~/miniconda3/envs/julia-env/share/julia/stdlib/v1.9/MbedTLS_jll/src/MbedTLS_jll.jl:42
  [5] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1115
  [6] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1061
  [7] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base ./loading.jl:1506
  [8] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1783
  [9] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1660
 [10] macro expansion
    @ ./loading.jl:1648 [inlined]
 [11] macro expansion
    @ ./lock.jl:267 [inlined]
 [12] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1611
 [13] include
    @ ./Base.jl:457 [inlined]
 [14] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
    @ Base ./loading.jl:2049
 [15] top-level scope
    @ stdin:3
during initialization of module MbedTLS_jll
in expression starting at /home/asan/miniconda3/envs/julia-env/share/julia/packages/MbedTLS/lqmet/src/MbedTLS.jl:1
in expression starting at stdin:3
ERROR: LoadError: Failed to precompile MbedTLS [739be429-bea8-5141-9913-cc70e7f3736d] to "/home/asan/miniconda3/envs/julia-env/share/julia/compiled/v1.9/MbedTLS/jl_FMQBYx".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:2300
  [3] compilecache
    @ ./loading.jl:2167 [inlined]
  [4] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1805
  [5] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1660
  [6] macro expansion
    @ ./loading.jl:1648 [inlined]
  [7] macro expansion
    @ ./lock.jl:267 [inlined]
  [8] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1611
  [9] include(mod::Module, _path::String)
    @ Base ./Base.jl:457
 [10] include(x::String)
    @ IJulia ~/miniconda3/envs/julia-env/share/julia/packages/IJulia/AQu2H/src/IJulia.jl:33
 [11] top-level scope
    @ ~/miniconda3/envs/julia-env/share/julia/packages/IJulia/AQu2H/src/IJulia.jl:309
 [12] include
    @ ./Base.jl:457 [inlined]
 [13] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
    @ Base ./loading.jl:2049
 [14] top-level scope
    @ stdin:3
in expression starting at /home/asan/miniconda3/envs/julia-env/share/julia/packages/IJulia/AQu2H/src/hmac.jl:1
in expression starting at /home/asan/miniconda3/envs/julia-env/share/julia/packages/IJulia/AQu2H/src/IJulia.jl:1
in expression starting at stdin:3

mkitti · November 26, 2023, 6:26pm

Hi asan,

Can you see if the official Julia binaries work? This issue is specific to conda-forge.