# Slow HTTP.jl requests when SSL is enabled

**URL:** <https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183>\
**Category:** Web Stack\
**Created:** [March 19, 2020, 11:07am UTC](https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183 "2020-03-19T11:07:34Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![karelh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/karelh/32/13146_2.png) [@karelh](https://discourse.julialang.org/u/karelh)\
**Post date:** [March 19, 2020, 11:07am UTC](https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183/1 "2020-03-19T11:07:34Z")

</div>

I am using HTTP.jl library to download large files over an encrypted connection (via AWSS3 to connect to an S3-enabled datastore). I have found out that the performance is greatly degraded when encrypted connection is used. Consider the following code (I use http-server Node.js package to serve a ~400MB file):

```julia
# HTTPS
@time HTTP.request("GET", "https://localhost:8080/download.gz" ; require_ssl_verification=false) |> (x -> ())
  4.922933 seconds (177.54 k allocations: 795.587 MiB, 1.06% gc time)

# HTTP
@time HTTP.request("GET", "http://localhost:8080/download.gz") |> (x -> ())
  0.765877 seconds (63.79 k allocations: 777.194 MiB, 0.61% gc time)

```

In comparison, if I perform the same requests via curl, I get 0.785s for the HTTPS request and 0.498s for the HTTP request.

Could you please suggest me how to deal with this performance issue? Thank you in advance!

---

<div class="post-metadata">

**Author:** ![logankilpatrick](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/logankilpatrick/32/38123_2.png) [@logankilpatrick](https://discourse.julialang.org/u/logankilpatrick)\
**Post date:** [March 19, 2020, 11:36am UTC](https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183/2 "2020-03-19T11:36:48Z")

</div>

Hey @karelh, this does not just seem to be a HTTP.jl situation but rather a behavior of HTTPS in general. See this thread for more details: [HTTP vs HTTPS performance - Stack Overflow](https://stackoverflow.com/questions/149274/http-vs-https-performance)

It seems like the solutions that are mentioned have to do with the actual device in question which in this case is owned by Amazon so you don’t have any control over it.

---

<div class="post-metadata">

**Author:** ![logankilpatrick](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/logankilpatrick/32/38123_2.png) [@logankilpatrick](https://discourse.julialang.org/u/logankilpatrick)\
**Post date:** [March 19, 2020, 11:37am UTC](https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183/3 "2020-03-19T11:37:21Z")

</div>

Also, Welcome to the Julia Discourse! 🥳

---

<div class="post-metadata">

**Author:** ![karelh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/karelh/32/13146_2.png) [@karelh](https://discourse.julialang.org/u/karelh)\
**Post date:** [March 19, 2020, 12:04pm UTC](https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183/4 "2020-03-19T12:04:49Z")

</div>

Thank you for the reply! I am fully aware that HTTPS will be slower than plain unencrypted HTTP. My main concern regarding the example is that the HTTPS request via HTTP.jl is more than 6x slower than the same request executed using curl. It seems to be an issue with HTTPS since in the case of unencrypted connection, HTTP.jl was just ~1.5x slower than curl.

Thank you!

---

<div class="post-metadata">

**Author:** ![Vukasin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vukasin/32/38671_2.png) [@Vukasin](https://discourse.julialang.org/u/Vukasin)\
**Post date:** [August 9, 2022, 8:10am UTC](https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183/5 "2022-08-09T08:10:20Z")

</div>

Hello,

did anyone have any revelation regarding this slowness?  
I second karelh’s argument that this slowness of http vs https is much bigger in Julia than in other languages.  
My initial profiling show that MbedTLS library takes a lot of CPU to perform `mbedtls_gcm_update`.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [August 9, 2022, 11:46am UTC](https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183/6 "2022-08-09T11:46:58Z")

</div>

It’s a compilation issue/overhead (which someone can look into, and likely eliminate), actually (for me on Julia 1.7.3), https if faster (0.1666 sec. vs. 0.3339 sec., 1.87x faster for best case) after first run:

```julia
julia> @time using HTTP
  0.296008 seconds (124.49 k allocations: 9.604 MiB, 63.46% compilation time)

julia> @time HTTP.request("GET", "http://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183" ; require_ssl_verification=false) |> (x -> ())
 12.075923 seconds (13.90 M allocations: 727.456 MiB, 4.09% gc time, 91.38% compilation time)
()

julia> @time HTTP.request("GET", "http://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183" ; require_ssl_verification=false) |> (x -> ())
  0.454217 seconds (1.36 k allocations: 221.203 KiB, 0.11% compilation time)
()

[..]

julia> @time HTTP.request("GET", "http://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183") |> (x -> ())
  2.327364 seconds (1.17 M allocations: 61.813 MiB, 1.14% gc time, 47.11% compilation time)
()

julia> @time HTTP.request("GET", "http://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183") |> (x -> ())
  0.443472 seconds (1.37 k allocations: 222.422 KiB, 0.11% compilation time)
()

# Why did this next one suddenly get way more allocations?

julia> @time HTTP.request("GET", "http://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183") |> (x -> ())
  0.457293 seconds (29.66 k allocations: 1.738 MiB, 18.99% compilation time)
()

julia> @time HTTP.request("GET", "http://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183") |> (x -> ())
  0.339599 seconds (1.38 k allocations: 221.641 KiB, 0.19% compilation time)
()

julia> @time HTTP.request("GET", "http://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183") |> (x -> ())
  0.335106 seconds (1.38 k allocations: 221.641 KiB, 0.17% compilation time)
()

```

In new session:

```julia
julia> @time HTTP.request("GET", "https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183" ; require_ssl_verification=false) |> (x -> ())
 11.720031 seconds (13.76 M allocations: 719.623 MiB, 4.22% gc time, 91.48% compilation time)
()

julia> @time HTTP.request("GET", "https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183" ; require_ssl_verification=false) |> (x -> ())
  0.279456 seconds (1.20 k allocations: 347.609 KiB, 0.10% compilation time)
()

julia> @time HTTP.request("GET", "https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183" ; require_ssl_verification=false) |> (x -> ())
  0.203439 seconds (1.22 k allocations: 211.641 KiB, 0.25% compilation time)
()

julia> @time HTTP.request("GET", "https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183" ; require_ssl_verification=false) |> (x -> ())
  0.172431 seconds (1.21 k allocations: 211.516 KiB, 0.16% compilation time)
()

```

Actually for me beating curl’s 0.905 sec. (and its http 0.343 sec, if excluding time for `using HTTP` and other one-time overhead):

```julia
$ time curl https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183 >/dev/null

```

The original complaint downloaded a much larger “~400MB” file (see allocation numbers), so this could be a scalability issue, and at some point https gets slower? It seems to allocate 2x the size of the file, assuming the file is actually 777/2 = 388.5 MB or less. With https then it also allocates 2.3% more, but I doubt that’s an issue.

---

<div class="post-metadata">

**Author:** ![rikh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rikh/32/204104_2.png) [@rikh](https://discourse.julialang.org/u/rikh)\
**Post date:** [August 9, 2022, 12:21pm UTC](https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183/7 "2022-08-09T12:21:24Z")

</div>

`HTTP.jl` also appears slower for large files here than `curl` and Firefox. Firefox appears to take a few seconds, `curl` takes 1.7 seconds and `HTTP.jl` takes 18 seconds after compilation.

**Details**

```julia
pkg> activate --temp

pkg> add HTTP

julia> using HTTP

julia> url = "https://julialang-s3.julialang.org/bin/linux/x64/1.7/julia-1.7.3-linux-x86_64.tar.gz";

julia> @time @eval HTTP.request("GET", url);
 48.622656 seconds (12.99 M allocations: 886.074 MiB, 2.31% gc time, 60.97% compilation time)

julia> @time @eval HTTP.request("GET", url);
 18.295575 seconds (53.13 k allocations: 235.733 MiB)

```

```julia
$ time curl -H "Cache-Control: no-cache" --location http://julialang-s3.julialang.org/bin/linux/x64/1.7/julia-1.7.3-linux-x86_64.tar.gz --output tmp.txt
 [...]
 Executed in 1.74 secs fish external
   usr time 488.51 millis 0.46 millis 488.05 millis
   sys time 809.93 millis 1.73 millis 808.20 millis

```

@karelh seems to have correctly determined that the root of the problem is the SSL implementation. This is the bottom of the `PProf.jl` flamegraph with `--threads=1`:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/0/1/01ff177f9d4f59c41de7bff6aa47c257832183b1.png)

For an indication, 95% of the time is spent in `ssl_unsafe_read` which is defined at [MbedTLS.jl/ssl.jl at 22f5393176a1a470317fa831c2db2392a9ccfb1e · JuliaLang/MbedTLS.jl · GitHub](https://github.com/JuliaLang/MbedTLS.jl/blob/22f5393176a1a470317fa831c2db2392a9ccfb1e/src/ssl.jl#L344). This is with the latest HTTP.jl and MbedTLS.jl versions; respectively 1.2.0 and 1.1.3.

---

<div class="post-metadata">

**Author:** ![Vukasin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vukasin/32/38671_2.png) [@Vukasin](https://discourse.julialang.org/u/Vukasin)\
**Post date:** [August 9, 2022, 1:04pm UTC](https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183/8 "2022-08-09T13:04:18Z")

</div>

Indeed, I was also experimenting with larger downloads (e.g. 1 MB files, but plenty of them in parallel, using 8 threads). It’s interesting to know that it actually has lower latency for smaller files, thanks @Palli

I have just opened an issue at MbedTLS to ask about this: [possible performance issue: mbedtls\_gcm\_update CPU utilization · Issue #254 · JuliaLang/MbedTLS.jl · GitHub](https://github.com/JuliaLang/MbedTLS.jl/issues/254)

Though for me, it was different decryption scheme (GCM), so this info from @rikh means it’s not specific to GCM scheme.

---

<div class="post-metadata">

**Author:** ![Vukasin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vukasin/32/38671_2.png) [@Vukasin](https://discourse.julialang.org/u/Vukasin)\
**Post date:** [August 9, 2022, 1:26pm UTC](https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183/9 "2022-08-09T13:26:00Z")

</div>

@Palli , I see that you have opened an issue: [Drop mbedTLS (and support BoringSSL (and HTTP/3)) · Issue #45856 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/45856), to consider using BoringSSL. If this package existed in Julia, I think HTTP.jl would benefit from it.

Here’s some perf comparison between MbedTLS and BoringSSL that I stumbled upon, it looks pretty significant: [mbedTLS vs BoringSSL on ARM](https://www.amongbytes.com/post/201804-comparing-mbedtls-to-boringssl/)

---

<div class="post-metadata">

**Author:** ![jd-foster](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jd-foster/32/35824_2.png) [@jd-foster](https://discourse.julialang.org/u/jd-foster)\
**Post date:** [August 9, 2022, 2:00pm UTC](https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183/10 "2022-08-09T14:00:12Z")

</div>

FWIW, the maintainer of HTTP.jl is aware of MbedTLS limitations:

> [@TLS Support Questions](https://discourse.julialang.org/t/tls-support-questions/84004):
>
> Currently HTTP.jl hard-codes a dependency on MbedTLS.jl for TLS support; it has served us decently over the years, but there have always been a lingering set of errors that crop up that are very difficult to debug; either because mbedtls library is a “black box” or the bindings around it (in MbedTLS.jl) are also tricky to get right. What do people generally think around TLS in Julia going forward? It doesn’t seem like there’s an appetite to do something native (though go and rust both have impl…