HTTP hangs indefinitely for certain sites

Specific Domains Web Stack
1

I’m using HTTP to talk to the Mastodon API, which involves connecting to many sites, many of which do not actually run the API (any more). For a very few sites (but not zero), HTTP.get() hangs, apparently indefinitely. For example:

using HTTP
response = HTTP.get("https://laurin-funkwhale.loophole.site", 
   connect_timeout=8, read_timeout=5, retry=false)

If I attempt to connect to that site with a browser, there is a warning about a self-signed certificate. If I go on to connect with the browser, subsequent attempts from within Julia will complete promptly with an error.

Is there any way of getting HTTP to recognise this problem? Or of invoking it within a wrapper that will kill the request if it hasn’t completed within a given time (the built-in timeouts don’t seem to deal with this situation)?

UPDATE: It hangs for 1 hour

@time try HTTP.get("https://laurin-funkwhale.loophole.site", connect_timeout=8, read_timeout=5, retry=false); catch why; println(why); end;
 HTTP.Exceptions.ConnectError("https://laurin-funkwhale.loophole.site", EOFError())                                                                                     
3633.672266 seconds (129.83 k allocations: 6.798 MiB, 0.80% compilation time)
2

What version of HTTP are you on? There were bugs with not respecting the connect_timeout in the past that have since been fixed.

3

I get the same 1-hr timeout with HTTP v0.9.17 on an old installation, and v1.8.0 (which I think is the most recent) on another machine (Julia 1.8.5).

FWIW, wget with default options takes 15 mins to give up on this URL, curl takes 5 mins (with an “SSL connection timeout” message).

4

Can you try the latest HTTP.jl version? 1.9.4? We’ve had a number of fixes/changes, so I’d like to know if this is still an issue in the current release. If so, let’s file an issue and I can dig into it.

1 Like
5

It works. It times out in about 15 seconds, using the call above.

However, if I GET it a second time (with a session variable tagged on at the end of the URL to avoid caching) it hangs again for an hour. (I don’t have a good reason to call it a second time, but I found the behaviour odd.)

Code and output below:

using Pkg
Pkg.activate("httptest")
Pkg.status("HTTP")

using HTTP
@time try HTTP.get("https://laurin-funkwhale.loophole.site/?nc=81234",
                   connect_timeout=8, read_timeout=5, retry=false);
catch why; println(why); end;

@time try HTTP.get("https://laurin-funkwhale.loophole.site/?nc=91234",
                   connect_timeout=8, read_timeout=5, retry=false);
catch why; println(why); end;


Status `~/work/mastodon/magnit/test/httptest/Project.toml`
  [cd3eb016] HTTP v1.9.5
HTTP.Exceptions.ConnectError("https://laurin-funkwhale.loophole.site/?nc=81234", OpenSSL.OpenSSLError("self-signed certificate"))
 17.443414 seconds (14.18 M allocations: 707.679 MiB, 5.67% gc time, 69.64% compilation time)
HTTP.Exceptions.ConnectError("https://laurin-funkwhale.loophole.site/?nc=91234", EOFError())
3600.262096 seconds (12.34 k allocations: 674.812 KiB, 0.00% compilation time)
  Activating project at `~/work/mastodon/magnit/test/httptest`