Slow HTTP.jl requests when SSL is enabled

[karelh]

I am using HTTP.jl library to download large files over an encrypted connection (via AWSS3 to connect to an S3-enabled datastore). I have found out that the performance is greatly degraded when encrypted connection is used. Consider the following code (I use http-server Node.js package to serve a ~400MB file):

# HTTPS
@time HTTP.request("GET", "https://localhost:8080/download.gz" ; require_ssl_verification=false) |> (x -> ())
  4.922933 seconds (177.54 k allocations: 795.587 MiB, 1.06% gc time)

# HTTP
@time HTTP.request("GET", "http://localhost:8080/download.gz") |> (x -> ())
  0.765877 seconds (63.79 k allocations: 777.194 MiB, 0.61% gc time)

In comparison, if I perform the same requests via curl, I get 0.785s for the HTTPS request and 0.498s for the HTTP request.

Could you please suggest me how to deal with this performance issue? Thank you in advance!

[logankilpatrick]

Hey @karelh, this does not just seem to be a HTTP.jl situation but rather a behavior of HTTPS in general. See this thread for more details: HTTP vs HTTPS performance - Stack Overflow

It seems like the solutions that are mentioned have to do with the actual device in question which in this case is owned by Amazon so you don’t have any control over it.

[logankilpatrick]

Also, Welcome to the Julia Discourse!

[karelh]

Thank you for the reply! I am fully aware that HTTPS will be slower than plain unencrypted HTTP. My main concern regarding the example is that the HTTPS request via HTTP.jl is more than 6x slower than the same request executed using curl. It seems to be an issue with HTTPS since in the case of unencrypted connection, HTTP.jl was just ~1.5x slower than curl.

Thank you!

[Vukasin]

Hello,

did anyone have any revelation regarding this slowness?
I second karelh’s argument that this slowness of http vs https is much bigger in Julia than in other languages.
My initial profiling show that MbedTLS library takes a lot of CPU to perform mbedtls_gcm_update.

[Palli]

It’s a compilation issue/overhead (which someone can look into, and likely eliminate), actually (for me on Julia 1.7.3), https if faster (0.1666 sec. vs. 0.3339 sec., 1.87x faster for best case) after first run:

julia> @time using HTTP
  0.296008 seconds (124.49 k allocations: 9.604 MiB, 63.46% compilation time)

julia> @time HTTP.request("GET", "http://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183" ; require_ssl_verification=false) |> (x -> ())
 12.075923 seconds (13.90 M allocations: 727.456 MiB, 4.09% gc time, 91.38% compilation time)
()

julia> @time HTTP.request("GET", "http://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183" ; require_ssl_verification=false) |> (x -> ())
  0.454217 seconds (1.36 k allocations: 221.203 KiB, 0.11% compilation time)
()

[..]

julia> @time HTTP.request("GET", "http://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183") |> (x -> ())
  2.327364 seconds (1.17 M allocations: 61.813 MiB, 1.14% gc time, 47.11% compilation time)
()

julia> @time HTTP.request("GET", "http://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183") |> (x -> ())
  0.443472 seconds (1.37 k allocations: 222.422 KiB, 0.11% compilation time)
()

# Why did this next one suddenly get way more allocations?

julia> @time HTTP.request("GET", "http://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183") |> (x -> ())
  0.457293 seconds (29.66 k allocations: 1.738 MiB, 18.99% compilation time)
()

julia> @time HTTP.request("GET", "http://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183") |> (x -> ())
  0.339599 seconds (1.38 k allocations: 221.641 KiB, 0.19% compilation time)
()

julia> @time HTTP.request("GET", "http://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183") |> (x -> ())
  0.335106 seconds (1.38 k allocations: 221.641 KiB, 0.17% compilation time)
()

In new session:

julia> @time HTTP.request("GET", "https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183" ; require_ssl_verification=false) |> (x -> ())
 11.720031 seconds (13.76 M allocations: 719.623 MiB, 4.22% gc time, 91.48% compilation time)
()

julia> @time HTTP.request("GET", "https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183" ; require_ssl_verification=false) |> (x -> ())
  0.279456 seconds (1.20 k allocations: 347.609 KiB, 0.10% compilation time)
()

julia> @time HTTP.request("GET", "https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183" ; require_ssl_verification=false) |> (x -> ())
  0.203439 seconds (1.22 k allocations: 211.641 KiB, 0.25% compilation time)
()

julia> @time HTTP.request("GET", "https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183" ; require_ssl_verification=false) |> (x -> ())
  0.172431 seconds (1.21 k allocations: 211.516 KiB, 0.16% compilation time)
()

Actually for me beating curl’s 0.905 sec. (and its http 0.343 sec, if excluding time for using HTTP and other one-time overhead):

$ time curl https://discourse.julialang.org/t/slow-http-jl-requests-when-ssl-is-enabled/36183 >/dev/null

The original complaint downloaded a much larger “~400MB” file (see allocation numbers), so this could be a scalability issue, and at some point https gets slower? It seems to allocate 2x the size of the file, assuming the file is actually 777/2 = 388.5 MB or less. With https then it also allocates 2.3% more, but I doubt that’s an issue.

[rikh]

HTTP.jl also appears slower for large files here than curl and Firefox. Firefox appears to take a few seconds, curl takes 1.7 seconds and HTTP.jl takes 18 seconds after compilation.

Details

pkg> activate --temp

pkg> add HTTP

julia> using HTTP

julia> url = "https://julialang-s3.julialang.org/bin/linux/x64/1.7/julia-1.7.3-linux-x86_64.tar.gz";

julia> @time @eval HTTP.request("GET", url);
 48.622656 seconds (12.99 M allocations: 886.074 MiB, 2.31% gc time, 60.97% compilation time)

julia> @time @eval HTTP.request("GET", url);
 18.295575 seconds (53.13 k allocations: 235.733 MiB)

$ time curl -H "Cache-Control: no-cache" --location http://julialang-s3.julialang.org/bin/linux/x64/1.7/julia-1.7.3-linux-x86_64.tar.gz --output tmp.txt
 [...]
 Executed in    1.74 secs      fish           external
   usr time  488.51 millis    0.46 millis  488.05 millis
   sys time  809.93 millis    1.73 millis  808.20 millis

@karelh seems to have correctly determined that the root of the problem is the SSL implementation. This is the bottom of the PProf.jl flamegraph with --threads=1:

For an indication, 95% of the time is spent in ssl_unsafe_read which is defined at MbedTLS.jl/ssl.jl at 22f5393176a1a470317fa831c2db2392a9ccfb1e · JuliaLang/MbedTLS.jl · GitHub. This is with the latest HTTP.jl and MbedTLS.jl versions; respectively 1.2.0 and 1.1.3.

[Vukasin]

Indeed, I was also experimenting with larger downloads (e.g. 1 MB files, but plenty of them in parallel, using 8 threads). It’s interesting to know that it actually has lower latency for smaller files, thanks @Palli

I have just opened an issue at MbedTLS to ask about this: possible performance issue: mbedtls_gcm_update CPU utilization · Issue #254 · JuliaLang/MbedTLS.jl · GitHub

Though for me, it was different decryption scheme (GCM), so this info from @rikh means it’s not specific to GCM scheme.

[Vukasin]

@Palli , I see that you have opened an issue: Drop mbedTLS (and support BoringSSL (and HTTP/3)) · Issue #45856 · JuliaLang/julia · GitHub, to consider using BoringSSL. If this package existed in Julia, I think HTTP.jl would benefit from it.

Here’s some perf comparison between MbedTLS and BoringSSL that I stumbled upon, it looks pretty significant: mbedTLS vs BoringSSL on ARM

[jd-foster]

FWIW, the maintainer of HTTP.jl is aware of MbedTLS limitations:

TLS Support Questions

Currently HTTP.jl hard-codes a dependency on MbedTLS.jl for TLS support; it has served us decently over the years, but there have always been a lingering set of errors that crop up that are very difficult to debug; either because mbedtls library is a “black box” or the bindings around it (in MbedTLS.jl) are also tricky to get right. What do people generally think around TLS in Julia going forward? It doesn’t seem like there’s an appetite to do something native (though go and rust both have impl…