HTTP Package

General Usage
,
1

I’m trying to write a script that uses the HTTP package to download a bunch of files off a website. I seem to be getting a ECONNRESET error when I try to download the file. Any chance someone might know what is going on here? I’m new to the HTTP package, so I could be missing something super basic.

ERROR: LoadError: On worker 2:
HTTP.IOExtras.IOError(read: connection reset by peer (ECONNRESET), "during request(https://nccs-data.urban.org/data/core/2015/coreco.core2015co.csv)")
try_yieldto at .\event.jl:189
wait at .\event.jl:234
wait at .\event.jl:27
wait_readnb at .\stream.jl:296
eof at .\stream.jl:56
eof at C:\Users\[...]\.julia\v0.6\MbedTLS\src\ssl.jl:292
eof at C:\Users\[...]\.julia\v0.6\HTTP\src\ConnectionPool.jl:156
eof at C:\Users\[...]\.julia\v0.6\HTTP\src\Streams.jl:187
write at .\io.jl:356
readbody at C:\Users\[...]\.julia\v0.6\HTTP\src\StreamRequest.jl:132
macro expansion at C:\Users\[...]\.julia\v0.6\HTTP\src\StreamRequest.jl:59 [inlined]
macro expansion at .\task.jl:302 [inlined]
#request#1 at C:\Users\[...]\.julia\v0.6\HTTP\src\StreamRequest.jl:53
#request at .\<missing>:0
#request#1 at C:\Users\[...]\.julia\v0.6\HTTP\src\ConnectionRequest.jl:45
#request at .\<missing>:0
#request#1 at C:\Users\[...]\.julia\v0.6\HTTP\src\ExceptionRequest.jl:19
#request at .\<missing>:0
##49#50 at .\error.jl:139
##49#51 at .\<missing>:0
#request#1 at C:\Users\[...]\.julia\v0.6\HTTP\src\RetryRequest.jl:44
#request at .\<missing>:0
#request#1 at C:\Users\[...]\.julia\v0.6\HTTP\src\MessageRequest.jl:45
#request at .\<missing>:0
#request#1 at C:\Users\[...]\.julia\v0.6\HTTP\src\RedirectRequest.jl:24
#request at .\<missing>:0
#request#7 at C:\Users\[...]\.julia\v0.6\HTTP\src\HTTP.jl:289
#request at .\<missing>:0
#request#8 at C:\Users\[...]\.julia\v0.6\HTTP\src\HTTP.jl:303
#request at .\<missing>:0
#getFile#6 at ...Project\data\EndowmentGetNCCS.jl:9
#getFiles#7 at C:\Users\[...]\Dropbox\Projects\Endowment Project\data\EndowmentGetNCCS.jl:33
#getFiles at .\<missing>:0
#10 at .\distributed\macros.jl:25
#103 at .\distributed\process_messages.jl:264 [inlined]
run_work_thunk at .\distributed\process_messages.jl:56
run_work_thunk at .\distributed\process_messages.jl:65 [inlined]
#96 at .\event.jl:73
Stacktrace:
 [1] #remotecall_fetch#141(::Array{Any,1}, ::Function, ::Function, ::Base.Distributed.Worker, ::Base.Distributed.RRID, ::Vararg{Any,N} where N) at .\distributed\remotecall.jl:359
 [2] remotecall_fetch(::Function, ::Base.Distributed.Worker, ::Base.Distributed.RRID, ::Vararg{Any,N} where N) at .\distributed\remotecall.jl:351
 [3] #remotecall_fetch#144(::Array{Any,1}, ::Function, ::Function, ::Int64, ::Base.Distributed.RRID, ::Vararg{Any,N} where N) at .\distributed\remotecall.jl:372
 [4] call_on_owner(::Function, ::Future) at .\distributed\remotecall.jl:445
 [5] fetch(::Future) at .\distributed\remotecall.jl:465
 [6] broadcast_t(::Function, ::Type{Any}, ::Tuple{Base.OneTo{Int64}}, ::CartesianRange{CartesianIndex{1}}, ::Array{Future,1}) at .\broadcast.jl:258
 [7] broadcast_c at .\broadcast.jl:321 [inlined]
 [8] broadcast at .\broadcast.jl:455 [inlined]
 [9] #getFilesPar#8(::Array{String,1}, ::Float64, ::Function, ::Array{String,1}, ::Array{String,1}, ::String) at C:\Users\[...]\Dropbox\Projects\Endowment Project\data\EndowmentGetNCCS.jl:71
 [10] (::Endowment.#kw##getFilesPar)(::Array{Any,1}, ::Endowment.#getFilesPar, ::Array{String,1}, ::Array{String,1}, ::String) at .\<missing>:0
 [11] #getFiles#7(::Array{String,1}, ::Float64, ::Function, ::Array{String,1}, ::Array{String,1}, ::String, ::Symbol) at C:\Users\[...]\Dropbox\Projects\Endowment Project\data\EndowmentGetNCCS.jl:23
 [12] (::Endowment.#kw##getFiles)(::Array{Any,1}, ::Endowment.#getFiles, ::Array{String,1}, ::Array{String,1}, ::String, ::Symbol) at .\<missing>:0
 [13] #getFiles#13(::Symbol, ::String, ::String, ::Float64, ::Function, ::String) at C:\Users\[...]\Dropbox\Projects\Endowment Project\data\EndowmentGetNCCS.jl:88
 [14] preProcessData at C:\Users\[...]\Dropbox\Projects\Endowment Project\data\EndowmentData.jl:5 [inlined]
 [15] endowmentTasks() at C:\Users\[...]\Dropbox\Projects\Endowment Project\EndowmentScripts.jl:20
 [16] include_string(::String, ::String) at .\loading.jl:522
 [17] include_string(::Module, ::String, ::String) at C:\Users\[...]\.julia\v0.6\Compat\src\Compat.jl:88
 [18] (::Atom.##112#116{String,String})() at C:\Users\[...]\.julia\v0.6\Atom\src\eval.jl:109
 [19] withpath(::Atom.##112#116{String,String}, ::String) at C:\Users\[...]\.julia\v0.6\CodeTools\src\utils.jl:30
 [20] withpath(::Function, ::String) at C:\Users\[...]\.julia\v0.6\Atom\src\eval.jl:38
 [21] hideprompt(::Atom.##111#115{String,String}) at C:\Users\[...]\.julia\v0.6\Atom\src\repl.jl:67
 [22] macro expansion at C:\Users\[...]\.julia\v0.6\Atom\src\eval.jl:106 [inlined]
 [23] (::Atom.##110#114{Dict{String,Any}})() at .\task.jl:80
while loading C:\Users\[...]\Dropbox\Projects\Endowment Project\EndowmentScripts.jl, in expression starting on line 237

Here is a snippit of the code which produces the error, which I tried to base off of the HTTP documentation for streaming to a file.

  oStream::GZipStream = GZip.open("$outPath\\$outName.gz", "w")
  rHandle::HTTP.Request = HTTP.request("GET", "$url/$fileName",
    response_stream=oStream, retries=4)
  GZip.close(oStream)

Thanks!

2

I would typically run this like:

oStream::GZipStream = GZip.open("$outPath\\$outName.gz", "w")
HTTP.open("GET", "$url/$fileName", retries=4) do io
    while !eof(io)
        write(oStream, readavailable(io))
    end
end

You can also run HTTP.request(...; verbose=3) to see more verbose logging of the actual request getting sent over the wire (and response coming back if any).

1 Like
3

Writing it that way worked like a charm. Thanks!