HTTP.jl requests fail from time to time

Specific Domains Web Stack
1

I found infrequent failures when using HTTP.jl library pinging for a website. It usually happens when the code did not make any queries for a while. I tried to use retry and timeout config to mitigate the issue but found them ineffective. It feels like each time when the program did not run for a while the query code would need to be recompiled. Does anyone else have a similar experience?
I would interrupt the code using Ctrl C and would receive errors like the following, even though the code has already run for more than 2 seconds ( set as the time for connection time out and readtimeout ).

ERROR: ReadTimeoutError: Connection closed after 2 seconds
Stacktrace:

request(::Type{TimeoutLayer{StreamLayer{Union{}}}}, ::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, ::Request, ::Array{UInt8,1}; readtimeout::Int64, kw::Base.Iterators.Pairs{Symbol,Union{Nothing, Bool},Tuple{Symbol,Symbol,Symbol},NamedTuple{(:iofunction, :reached_redirect_limit, :retry),Tuple{Nothing,Bool,Bool}}}) at /home/fan/.julia/packages/HTTP/LXxGT/src/TimeoutRequest.jl

(::HTTP.var"#request##kw")(::NamedTuple{(:iofunction, :reached_redirect_limit, :readtimeout, :retry),Tuple{Nothing,Bool,Int64,Bool}}, ::typeof(HTTP.request), ::Type{TimeoutLayer{StreamLayer{Union{}}}}, ::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, ::Request, ::Array{UInt8,1}) at /home/fan/.julia/packages/HTTP/LXxGT/src/TimeoutRequest.jl

request(::Type{ConnectionPoolLayer{TimeoutLayer{StreamLayer{Union{}}}}}, ::URIs.URI, ::Request, ::Array{UInt8,1}; proxy::Nothing, socket_type::Type{T} where T, reuse_limit::Int64, kw::Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:iofunction, :reached_redirect_limit, :readtimeout, :retry),Tuple{Nothing,Bool,Int64,Bool}}}) at /home/fan/.julia/packages/HTTP/LXxGT/src/ConnectionRequest.jl

(::HTTP.var"#request##kw")(::NamedTuple{(:iofunction, :reached_redirect_limit, :readtimeout, :retry),Tuple{Nothing,Bool,Int64,Bool}}, ::typeof(HTTP.request), ::Type{ConnectionPoolLayer{TimeoutLayer{StreamLayer{Union{}}}}}, ::URIs.URI, ::Request, ::Array{UInt8,1}) at /home/fan/.julia/packages/HTTP/LXxGT/src/ConnectionRequest.jl

request(::Type{ExceptionLayer{ConnectionPoolLayer{TimeoutLayer{StreamLayer{Union{}}}}}}, ::URIs.URI, ::Vararg{Any,N} where N; kw::Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:iofunction, :reached_redirect_limit, :readtimeout, :retry),Tuple{Nothing,Bool,Int64,Bool}}}) at /home/fan/.julia/packages/HTTP/LXxGT/src/ExceptionRequest.jl

request at /home/fan/.julia/packages/HTTP/LXxGT/src/ExceptionRequest.jl

request(::Type{MessageLayer{ExceptionLayer{ConnectionPoolLayer{TimeoutLayer{StreamLayer{Union{}}}}}}}, ::String, ::URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}; http_version::VersionNumber, target::String, parent::Nothing, iofunction::Nothing, kw::Base.Iterators.Pairs{Symbol,Integer,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:reached_redirect_limit, :readtimeout, :retry),Tuple{Bool,Int64,Bool}}}) at /home/fan/.julia/packages/HTTP/LXxGT/src/MessageRequest.jl

request at /home/fan/.julia/packages/HTTP/LXxGT/src/MessageRequest.jl

request(::Type{BasicAuthLayer{MessageLayer{ExceptionLayer{ConnectionPoolLayer{TimeoutLayer{StreamLayer{Union{}}}}}}}}, ::String, ::URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}; kw::Base.Iterators.Pairs{Symbol,Integer,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:reached_redirect_limit, :readtimeout, :retry),Tuple{Bool,Int64,Bool}}}) at /home/fan/.julia/packages/HTTP/LXxGT/src/BasicAuthRequest.jl

(::HTTP.var"#request##kw")(::NamedTuple{(:reached_redirect_limit, :readtimeout, :retry),Tuple{Bool,Int64,Bool}}, ::typeof(HTTP.request), ::Type{BasicAuthLayer{MessageLayer{ExceptionLayer{ConnectionPoolLayer{TimeoutLayer{StreamLayer{Union{}}}}}}}}, ::String, ::URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at /home/fan/.julia/packages/HTTP/LXxGT/src/BasicAuthRequest.jl
...
2

If I haven’t run code using HTTP.jl module for about 5 - 10 minutes the following would fail.

conf = (connect_timeout=5, readtimeout=5, retry = false)
HTTP.get("https://julialang.org/"; conf...);

however rerun the code in repl for a second time, the code would run. I guess maybe I need to start the code, have a timeout, and rerun it? so I made this

conf = (connect_timeout=5, readtimeout=5, retry = false)
http_fetched = false
max_runtime = 2
while ~http_fetched
    global ticker_res_task = @async HTTP.get("https://julialang.org/"; conf...);
    timeout_status = timedwait(() -> istaskdone(ticker_res_task), max_runtime)
    if timeout_status == :ok
        http_fetched = true
    else
        println("request failed, do one more time")
    end
end

however, I got

request failed, do one more time
request failed, do one more time
request failed, do one more time
request failed, do one more time
...
3

After trials and errors, it seems the problem is caused by Revise.jl.