`HTTP.get()` crashes Julia completely

It should be reproducible.

using CSV
using HTTP

f = CSV.File(HTTP.get("https://covidtracking.com/api/v1/states/daily.csv").body) 


julia: symbol lookup error: /home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedtls.so: undefined symbol: mbedtls_x509_crt_verify_restartable
[affans@hpc covid19abm]$

Just upgraded to CSV 0.7 and this started happening. So seems that maybe CSV upgraded an artifact or something? I saw lots of artifacts and libraries being downloaded when CSV was installing/building.

Version info, environment, and OS are giving in posts below.

EDIT:
It has nothing to do with CSV. I can reproduce with HTTP alone.

julia> using HTTP

julia> HTTP.get("https://covidtracking.com/api/v1/states/daily.csv").body
julia: symbol lookup error: /home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedtls.so: undefined symbol: mbedtls_x509_crt_verify_restartable
[affans@hpc contact_tracing]$

I can’t reproduce your issue in this environment:

(tmp) pkg> status -m
Status `/tmp/Manifest.toml`
  [336ed68f] CSV v0.7.4
  [324d7699] CategoricalArrays v0.8.1
  [34da2185] Compat v3.13.0
  [9a962f9c] DataAPI v1.3.0
  [a93c6f00] DataFrames v0.21.4
  [864edb3b] DataStructures v0.17.19
  [e2d170a0] DataValueInterfaces v1.0.0
  [cd3eb016] HTTP v0.8.16
  [83e8ac13] IniFile v0.5.0
  [41ab1584] InvertedIndices v1.0.0
  [82899510] IteratorInterfaceExtensions v1.0.0
  [682c06a0] JSON v0.21.0
  [739be429] MbedTLS v1.0.2
  [c8ffd9c3] MbedTLS_jll v2.16.6+1
  [e1d29d7a] Missings v0.4.3
  [bac558e1] OrderedCollections v1.3.0
  [69de0a69] Parsers v1.0.7
  [2dfb63ee] PooledArrays v0.5.3
  [189a3867] Reexport v0.2.0
  [91c51154] SentinelArrays v1.2.9
  [a2af1166] SortingAlgorithms v0.3.1
  [3783bdb8] TableTraits v1.0.0
  [bd369af6] Tables v1.0.4
  [2a0f44e3] Base64 
  [ade2ca70] Dates 
  [8bb1440f] DelimitedFiles 
  [8ba89e20] Distributed 
  [9fa8497b] Future 
  [b77e0a4c] InteractiveUtils 
  [76f85450] LibGit2 
  [8f399da3] Libdl 
  [37e2e46d] LinearAlgebra 
  [56ddb016] Logging 
  [d6f4376e] Markdown 
  [a63ad114] Mmap 
  [44cfe95a] Pkg 
  [de0858da] Printf 
  [3fa0cd96] REPL 
  [9a3f8284] Random 
  [ea8e919c] SHA 
  [9e88b42a] Serialization 
  [1a1011a3] SharedArrays 
  [6462fe0b] Sockets 
  [2f01184e] SparseArrays 
  [10745b16] Statistics 
  [8dfed614] Test 
  [cf7118a7] UUIDs 
  [4ec0a83e] Unicode 

julia> versioninfo()
Julia Version 1.4.2
Commit 44fa15b150* (2020-05-23 18:35 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, haswell)

Could you please provide more information? Operating system, versions of the packages involved (preferably in a clean environment where you install only CSV and HTTP), etc…

(@v1.4) pkg> st
Status `~/.julia/environments/v1.4/Project.toml`
  [336ed68f] CSV v0.7.4
  [a93c6f00] DataFrames v0.21.2
  [cd3eb016] HTTP v0.8.16

and

julia> versioninfo()
Julia Version 1.4.0
Commit b8e9a9ecc6 (2020-03-21 16:36 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, broadwell)

Actually, I don’t think this is a problem with CSV, because the following works:

julia>  download("https://covidtracking.com/api/v1/states/daily.csv") |> CSV.File
7745-element CSV.File{false}:

so maybe a problem with how HTTP.get is downloading data?

CSV is likely not involved at all, mbedtls is used by HTTP. I guess that you can reproduce the error with HTTP.get Only

You didn’t answer to this question :point_up:

The OS is

[affans@hpc contact_tracing]$ lsb_release -d
Description:    CentOS Linux release 7.6.1810 (Core)

and yes, I also realized it’s not CSV. I can reproduce by just using HTTP.

julia> using HTTP

julia> HTTP.get("https://covidtracking.com/api/v1/states/daily.csv").body
julia: symbol lookup error: /home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedtls.so: undefined symbol: mbedtls_x509_crt_verify_restartable
[affans@hpc ]$

After loading HTTP, what’s the output of

using Libdl
filter!(lib -> occursin("mbed", lib), dllist())

?

julia> filter!(lib -> occursin("mbed", lib), dllist())
5-element Array{String,1}:
 "/home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedcrypto.so"
 "/home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedtls.so"
 "/usr/lib64/libmbedx509.so.0"
 "/usr/lib64/libmbedcrypto.so.2"
 "/home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedx509.so"

These are probably interfering. How did you install Julia?

Downloaded the binary from the website and is currently sitting in my home directory. This is a cluster envrionment and I don’t have access to sudo. I also noticed when downloaded CSV to 0.7, it was downloading a lot of artifacts, after which my code stopped working (which is why I initially thought it was a CSV problem).

From the error, it seems that it’s not using the system library though but the one local to me, so wouldn’t that be a problem?

What’s the output of the same commands in a fresh session? I’m not sure why those two libraries are being pulled in.

That was from a fresh session, but I did it again:

[affans@hpc contact_tracing]$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.4.0 (2020-03-21)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using HTTP

julia> using Libdl

julia> filter!(lib -> occursin("mbed", lib), dllist())
5-element Array{String,1}:
 "/home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedcrypto.so"
 "/home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedtls.so"
 "/usr/lib64/libmbedx509.so.0"
 "/usr/lib64/libmbedcrypto.so.2"
 "/home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedx509.so"

Sorry, by fresh session I meant a fresh session, without using HTTP :slightly_smiling_face:

julia> using Libdl

julia> filter!(lib -> occursin("mbed", lib), dllist())
0-element Array{String,1}

Wondering if we should move this to chat, but might have more eyes on discourse.

To a chat where the knowledge would be lost forever? :slightly_smiling_face:

Try the following. Do ]dev MbedTLS_jll, and edit the file src/wrappers/x86_64-linux-gnu.jl: move dlopening of libmbedx509 before dlopen-ing of libmbedtls

Okay, so my changes

    global libmbedx509_path = normpath(joinpath(artifact_dir, libmbedx509_splitpath...))

    # Manually `dlopen()` this right now so that future invocations
    # of `ccall` with its `SONAME` will find this path immediately.
    global libmbedx509_handle = dlopen(libmbedx509_path)
    push!(LIBPATH_list, dirname(libmbedx509_path))

    
    # Manually `dlopen()` this right now so that future invocations
    # of `ccall` with its `SONAME` will find this path immediately.
    global libmbedtls_handle = dlopen(libmbedtls_path)
    push!(LIBPATH_list, dirname(libmbedtls_path))

Now,

julia> using HTTP
[ Info: Precompiling HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3]

julia> using Libdl

julia> filter!(lib -> occursin("mbed", lib), dllist())
3-element Array{String,1}:
 "/home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedcrypto.so"
 "/home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedx509.so"
 "/home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedtls.so"

and

julia> HTTP.get("https://covidtracking.com/api/v1/states/daily.csv").body
1687178-element Array{UInt8,1}:

Success!

Two questions

  1. Does this mean I get to submit a PR? I’ve never done a PR before except for a very small doc change.

  2. Will I have to use the dev'ed version until the library is fixed?

  3. Could you just ELIF what we did here, what was going wrong, and why moving that code fixed things?

Thanks!!!

No, those files are automatically generated, a manual PR would be overwritten.

For a while, yes.

Not sure what “ELIF” means here. If you’re asking for an explanation, libmdetls.so depends on libmbedx509.so, as you can see with ldd ~/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedtls.so. In the JLL wrapper libmdetls.so is being opened before libmbedx509.so. Not finding libmbedx509.so already available, when trying to dlopen libmbedtls.so your dynamic loader decided to open the system libmbedx509.so, instead of the one from the artifacts (which is what happens in my case). By inverting the opening order you make sure that the dynamic loader doesn’t decide to open the system library.

For the record, your problem should be fixed in a more robust way by addressing this issue: https://github.com/JuliaPackaging/BinaryBuilder.jl/issues/669, but it isn’t trivial.

Okay. last question, somewhat non-related. I realize I don’t infact need to use HTTP.get() for my purpose, and download(url) |> CSV.File works for me. So how do I un-dev so that I go back to the “system” default.

And thanks so much for the lesson!

]rm MbedTLS_jll and then you can also safely delete the directory ~/.julia/dev/MbedTLS_jll if you really want to.