# \`HTTP.get()\` crashes Julia completely

**URL:** <https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506>\
**Category:** General Usage\
**Created:** [July 22, 2020, 6:30pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506 "2020-07-22T18:30:48Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [July 22, 2020, 6:30pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/1 "2020-07-22T18:30:48Z")

</div>

It should be reproducible.

```julia
using CSV
using HTTP

f = CSV.File(HTTP.get("https://covidtracking.com/api/v1/states/daily.csv").body) 

julia: symbol lookup error: /home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedtls.so: undefined symbol: mbedtls_x509_crt_verify_restartable
[affans@hpc covid19abm]$

```

Just upgraded to CSV 0.7 and this started happening. So seems that maybe CSV upgraded an artifact or something? I saw lots of artifacts and libraries being downloaded when `CSV` was installing/building.

Version info, environment, and OS are giving in posts below.

**EDIT:**  
It has nothing to do with CSV. I can reproduce with `HTTP` alone.

```julia
julia> using HTTP

julia> HTTP.get("https://covidtracking.com/api/v1/states/daily.csv").body
julia: symbol lookup error: /home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedtls.so: undefined symbol: mbedtls_x509_crt_verify_restartable
[affans@hpc contact_tracing]$

```

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 22, 2020, 6:39pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/2 "2020-07-22T18:39:07Z")

</div>

I can’t reproduce your issue in this environment:

```julia
(tmp) pkg> status -m
Status `/tmp/Manifest.toml`
  [336ed68f] CSV v0.7.4
  [324d7699] CategoricalArrays v0.8.1
  [34da2185] Compat v3.13.0
  [9a962f9c] DataAPI v1.3.0
  [a93c6f00] DataFrames v0.21.4
  [864edb3b] DataStructures v0.17.19
  [e2d170a0] DataValueInterfaces v1.0.0
  [cd3eb016] HTTP v0.8.16
  [83e8ac13] IniFile v0.5.0
  [41ab1584] InvertedIndices v1.0.0
  [82899510] IteratorInterfaceExtensions v1.0.0
  [682c06a0] JSON v0.21.0
  [739be429] MbedTLS v1.0.2
  [c8ffd9c3] MbedTLS_jll v2.16.6+1
  [e1d29d7a] Missings v0.4.3
  [bac558e1] OrderedCollections v1.3.0
  [69de0a69] Parsers v1.0.7
  [2dfb63ee] PooledArrays v0.5.3
  [189a3867] Reexport v0.2.0
  [91c51154] SentinelArrays v1.2.9
  [a2af1166] SortingAlgorithms v0.3.1
  [3783bdb8] TableTraits v1.0.0
  [bd369af6] Tables v1.0.4
  [2a0f44e3] Base64 
  [ade2ca70] Dates 
  [8bb1440f] DelimitedFiles 
  [8ba89e20] Distributed 
  [9fa8497b] Future 
  [b77e0a4c] InteractiveUtils 
  [76f85450] LibGit2 
  [8f399da3] Libdl 
  [37e2e46d] LinearAlgebra 
  [56ddb016] Logging 
  [d6f4376e] Markdown 
  [a63ad114] Mmap 
  [44cfe95a] Pkg 
  [de0858da] Printf 
  [3fa0cd96] REPL 
  [9a3f8284] Random 
  [ea8e919c] SHA 
  [9e88b42a] Serialization 
  [1a1011a3] SharedArrays 
  [6462fe0b] Sockets 
  [2f01184e] SparseArrays 
  [10745b16] Statistics 
  [8dfed614] Test 
  [cf7118a7] UUIDs 
  [4ec0a83e] Unicode 

julia> versioninfo()
Julia Version 1.4.2
Commit 44fa15b150* (2020-05-23 18:35 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, haswell)

```

Could you please provide more information? Operating system, versions of the packages involved (preferably in a clean environment where you install only `CSV` and `HTTP`), etc…

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [July 22, 2020, 6:43pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/3 "2020-07-22T18:43:48Z")

</div>

```julia
(@v1.4) pkg> st
Status `~/.julia/environments/v1.4/Project.toml`
  [336ed68f] CSV v0.7.4
  [a93c6f00] DataFrames v0.21.2
  [cd3eb016] HTTP v0.8.16

```

and

```julia
julia> versioninfo()
Julia Version 1.4.0
Commit b8e9a9ecc6 (2020-03-21 16:36 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, broadwell)

```

Actually, I don’t think this is a problem with `CSV`, because the following works:

```julia
julia> download("https://covidtracking.com/api/v1/states/daily.csv") |> CSV.File
7745-element CSV.File{false}:

```

so maybe a problem with how `HTTP.get` is downloading data?

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 22, 2020, 6:44pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/4 "2020-07-22T18:44:46Z")

</div>

> [@affans](#):
>
> Edit: Seems to work for files on disk:

`CSV` is likely not involved at all, mbedtls is used by `HTTP`. I guess that you can reproduce the error with `HTTP.get` Only

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 22, 2020, 6:45pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/5 "2020-07-22T18:45:48Z")

</div>

> [@giordano](#):
>
> Operating system

You didn’t answer to this question ☝

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [July 22, 2020, 6:48pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/6 "2020-07-22T18:48:35Z")

</div>

The OS is

```julia
[affans@hpc contact_tracing]$ lsb_release -d
Description: CentOS Linux release 7.6.1810 (Core)

```

and yes, I also realized it’s not CSV. I can reproduce by just using HTTP.

```julia
julia> using HTTP

julia> HTTP.get("https://covidtracking.com/api/v1/states/daily.csv").body
julia: symbol lookup error: /home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedtls.so: undefined symbol: mbedtls_x509_crt_verify_restartable
[affans@hpc]$

```

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 22, 2020, 7:05pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/7 "2020-07-22T19:05:08Z")

</div>

After loading `HTTP`, what’s the output of

```julia
using Libdl
filter!(lib -> occursin("mbed", lib), dllist())

```

?

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [July 22, 2020, 7:07pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/8 "2020-07-22T19:07:34Z")

</div>

```julia
julia> filter!(lib -> occursin("mbed", lib), dllist())
5-element Array{String,1}:
 "/home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedcrypto.so"
 "/home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedtls.so"
 "/usr/lib64/libmbedx509.so.0"
 "/usr/lib64/libmbedcrypto.so.2"
 "/home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedx509.so"

```

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 22, 2020, 7:08pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/9 "2020-07-22T19:08:17Z")

</div>

> [@affans](#):
>
> ```julia
> "/usr/lib64/libmbedx509.so.0"
> "/usr/lib64/libmbedcrypto.so.2"
> 
> ```

These are probably interfering. How did you install Julia?

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [July 22, 2020, 7:08pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/10 "2020-07-22T19:08:51Z")

</div>

Downloaded the binary from the website and is currently sitting in my home directory. This is a cluster envrionment and I don’t have access to sudo. I also noticed when downloaded `CSV` to 0.7, it was downloading a lot of artifacts, after which my code stopped working (which is why I initially thought it was a CSV problem).

From the error, it seems that it’s not using the system library though but the one local to me, so wouldn’t that be a problem?

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 22, 2020, 7:11pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/11 "2020-07-22T19:11:01Z")

</div>

What’s the output of the same commands in a fresh session? I’m not sure why those two libraries are being pulled in.

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [July 22, 2020, 7:13pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/12 "2020-07-22T19:13:01Z")

</div>

That was from a fresh session, but I did it again:

```julia
[affans@hpc contact_tracing]$ julia
               _
   _ _ _(_)_ | Documentation: https://docs.julialang.org
  (_) | (_) (_) |
   _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` | |
  | | |_| | | | (_| | | Version 1.4.0 (2020-03-21)
 _/ |\ __'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |

julia> using HTTP

julia> using Libdl

julia> filter!(lib -> occursin("mbed", lib), dllist())
5-element Array{String,1}:
 "/home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedcrypto.so"
 "/home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedtls.so"
 "/usr/lib64/libmbedx509.so.0"
 "/usr/lib64/libmbedcrypto.so.2"
 "/home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedx509.so"

```

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 22, 2020, 7:13pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/13 "2020-07-22T19:13:50Z")

</div>

Sorry, by fresh session I meant a fresh session, without `using HTTP` 🙂

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [July 22, 2020, 7:15pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/14 "2020-07-22T19:15:10Z")

</div>

```julia
julia> using Libdl

julia> filter!(lib -> occursin("mbed", lib), dllist())
0-element Array{String,1}

```

Wondering if we should move this to chat, but might have more eyes on discourse.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 22, 2020, 7:19pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/15 "2020-07-22T19:19:17Z")

</div>

> [@affans](#):
>
> Wondering if we should move this to chat

To a chat where the knowledge would be lost forever? 🙂

Try the following. Do `]dev MbedTLS_jll`, and edit the file `src/wrappers/x86_64-linux-gnu.jl`: move [`dlopen`ing of `libmbedx509`](https://github.com/JuliaBinaryWrappers/MbedTLS_jll.jl/blob/7a6982129a4c9021bfd0cfbfddade533945eec7a/src/wrappers/x86_64-linux-gnu.jl#L74-L79) before [`dlopen`-ing of `libmbedtls`](https://github.com/JuliaBinaryWrappers/MbedTLS_jll.jl/blob/7a6982129a4c9021bfd0cfbfddade533945eec7a/src/wrappers/x86_64-linux-gnu.jl#L67-L72)

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [July 22, 2020, 7:25pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/16 "2020-07-22T19:25:11Z")

</div>

Okay, so my changes

```julia
    global libmbedx509_path = normpath(joinpath(artifact_dir, libmbedx509_splitpath...))

    # Manually `dlopen()` this right now so that future invocations
    # of `ccall` with its `SONAME` will find this path immediately.
    global libmbedx509_handle = dlopen(libmbedx509_path)
    push!(LIBPATH_list, dirname(libmbedx509_path))

    
    # Manually `dlopen()` this right now so that future invocations
    # of `ccall` with its `SONAME` will find this path immediately.
    global libmbedtls_handle = dlopen(libmbedtls_path)
    push!(LIBPATH_list, dirname(libmbedtls_path))

```

Now,

```julia
julia> using HTTP
[Info: Precompiling HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3]

julia> using Libdl

julia> filter!(lib -> occursin("mbed", lib), dllist())
3-element Array{String,1}:
 "/home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedcrypto.so"
 "/home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedx509.so"
 "/home/affans/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedtls.so"

```

and

```julia
julia> HTTP.get("https://covidtracking.com/api/v1/states/daily.csv").body
1687178-element Array{UInt8,1}:

```

Success!

Two questions

1. Does this mean I get to submit a PR? I’ve never done a PR before except for a very small doc change.

2. Will I have to use the `dev`’ed version until the library is fixed?

3. Could you just ELIF what we did here, what was going wrong, and why moving that code fixed things?

Thanks!!!

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 22, 2020, 7:32pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/17 "2020-07-22T19:32:06Z")

</div>

> [@affans](#):
>
> Does this mean I get to submit a PR?

No, those files are automatically generated, a manual PR would be overwritten.

> [@affans](#):
>
> Will I have to use the `dev` 'ed version until the library is fixed?

For a while, yes.

> [@affans](#):
>
> Could you just ELIF what we did here, what was going wrong, and why moving that code fixed things?

Not sure what “ELIF” means here. If you’re asking for an explanation, `libmdetls.so` depends on `libmbedx509.so`, as you can see with `ldd ~/.julia/artifacts/746f3085962108a6a143ae685781235717cec381/lib/libmbedtls.so`. In the JLL wrapper `libmdetls.so` is being opened before `libmbedx509.so`. Not finding `libmbedx509.so` already available, when trying to dlopen `libmbedtls.so` your dynamic loader decided to open the system `libmbedx509.so`, instead of the one from the artifacts (which is what happens in my case). By inverting the opening order you make sure that the dynamic loader doesn’t decide to open the system library.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 22, 2020, 7:36pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/18 "2020-07-22T19:36:44Z")

</div>

For the record, your problem should be fixed in a more robust way by addressing this issue: [https://github.com/JuliaPackaging/BinaryBuilder.jl/issues/669](https://github.com/JuliaPackaging/BinaryBuilder.jl/issues/669), but it isn’t trivial.

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [July 22, 2020, 7:43pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/19 "2020-07-22T19:43:39Z")

</div>

Okay. last question, somewhat non-related. I realize I don’t infact need to use `HTTP.get()` for my purpose, and `download(url) |> CSV.File` works for me. So how do I un-`dev` so that I go back to the “system” default.

And thanks so much for the lesson!

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 22, 2020, 8:15pm UTC](https://discourse.julialang.org/t/http-get-crashes-julia-completely/43506/20 "2020-07-22T20:15:40Z")

</div>

> [@affans](#):
>
> So how do I un- `dev` so that I go back to the “system” default.

`]rm MbedTLS_jll` and then you can also safely delete the directory `~/.julia/dev/MbedTLS_jll` if you really want to.