using HTTP, TerminalLoggers, Logging
# Set logger to a Terminal logger which thus supports progress bars
global_logger(TerminalLogger(right_justify=120))
url = "http://ipv4.download.thinkbroadband.com/10MB.zip"
HTTP.download(url)
This might be useful to simplify your implementation, if you were so inclined.
You are almost correct, the second version should look like
res = DataFrame(CSV.File(IOBuffer(urldownload(url, parser = identity))))
and there will be no difference in the outcome. Itβs just a matter of personal taste which form to use.
Regarding FileIO.jl, it was my initial approach, unfortunately, FileIO.jl (or perhaps libraries that use it) are not very friendly to IOBuffer data types. For example, this bug was fixed only recently, there is no support for JSON files and so on. In the end, it was easier to avoid FileIO.jl altogether. Maybe at some point, Iβll return FileIO but it requires lots of testing.
Guys, a related question, do you know how can I test if a link is valid? I have a list of URLS and I want to download them all to disk, but I cant make a loop, since if the link isbroken it sends an error.
using UrlDownload
using Sockets
using DataFrames
urls = ["https://badurl", "https://www.stats.govt.nz/assets/Uploads/Electronic-card-transactions/Electronic-card-transactions-May-2020/Download-data/electronic-card-transactions-may-2020-csv.zip"]
res = DataFrame[]
for url in urls
try
push!(res, urldownload(url, true) |> DataFrame)
catch er
if er isa Sockets.DNSError
println("url $url does not exists")
else
rethrow()
end
end
end
You should go through catch part and find out which errors you want to skip and which you want to raise error to.