# \[ANN\] UrlDownload.jl

**URL:** <https://discourse.julialang.org/t/ann-urldownload-jl/37841>\
**Category:** Package Announcements\
**Tags:** package, announcement\
**Created:** [April 19, 2020, 9:23am UTC](https://discourse.julialang.org/t/ann-urldownload-jl/37841 "2020-04-19T09:23:22Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [April 19, 2020, 9:23am UTC](https://discourse.julialang.org/t/ann-urldownload-jl/37841/1 "2020-04-19T09:23:22Z")

</div>

[UrlDownload.jl](https://github.com/Arkoniak/UrlDownload.jl) is a small packaged aimed to simplify the process of data download and conversion to the necessary format.

**Features**

1. In-memory data processing, no intermediate files stored.
2. Supports csv, json, feather, various image formats with basic autodetection ability.
3. Allow usage of custom parsers for all other data formats.
4. Use `ProgressMeter.jl` to show the status of the download process.

**Examples**

1. Basic usage

```julia
using UrlDownload

url = "https://raw.githubusercontent.com/Arkoniak/UrlDownload.jl/master/data/test.feather"
df = urldownload(url)

# 2×2 DataFrame
# │ Row │ x │ y │
# │ │ Int64 │ Int64 │
# ├─────┼───────┼───────┤
# │ 1 │ 1 │ 2 │
# │ 2 │ 3 │ 4 │

```

1. Progress meter

```julia
using UrlDownload

url = "https://www.stats.govt.nz/assets/Uploads/Business-price-indexes/Business-price-indexes-December-2019-quarter/Download-data/business-price-indexes-december-2019-quarter-csv.csv"

urldownload(url, true)
# Progress: 45%|████████████████████ | Time: 0:00:01

```

1. Custom parsers

```julia
using UrlDownload
using DataFrames
using CSV

url = "https://raw.githubusercontent.com/Arkoniak/UrlDownload.jl/master/data/ext.csv"
res = urldownload(url, parser = x -> DataFrame(CSV.File(IOBuffer(x))))
# 2×2 DataFrame
# │ Row │ x │ y │
# │ │ Int64 │ Int64 │
# ├─────┼───────┼───────┤
# │ 1 │ 1 │ 2 │
# │ 2 │ 3 │ 4 │

```

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [April 19, 2020, 9:50am UTC](https://discourse.julialang.org/t/ann-urldownload-jl/37841/2 "2020-04-19T09:50:02Z")

</div>

You might also be interested in [`HTTP.download`](https://github.com/JuliaWeb/HTTP.jl/blob/master/src/download.jl#L92) and [TerminalLoggers Progress Bars](https://c42f.github.io/TerminalLoggers.jl/stable/#Progress-bars-1)

```julia
using HTTP, TerminalLoggers, Logging

# Set logger to a Terminal logger which thus supports progress bars
global_logger(TerminalLogger(right_justify=120))

url = "http://ipv4.download.thinkbroadband.com/10MB.zip"
HTTP.download(url)

```

[![asciicast](https://asciinema.org/a/XGF38e7MEMLf2eSsb7ZOTMlzB.svg)](https://asciinema.org/a/XGF38e7MEMLf2eSsb7ZOTMlzB?t=10)

This might be useful to simplify your implementation, if you were so inclined.

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [April 19, 2020, 9:51am UTC](https://discourse.julialang.org/t/ann-urldownload-jl/37841/3 "2020-04-19T09:51:53Z")

</div>

Can you tell me what the difference betweeen:

```julia
res = urldownload(url, parser = x -> DataFrame(CSV.File(IOBuffer(x))))

```

and:

```julia
res = DataFrame(CSV.File(IOBuffer(urldownload(url))))

```

is?

Is the point to avoid triggering the autodetection?

* * *

You might also be interested in [FileIO.jl](https://github.com/JuliaIO/FileIO.jl) which is all about autodetection.  
Possibly you could use this to support autodetection on a ton more types?

---

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [April 19, 2020, 10:09am UTC](https://discourse.julialang.org/t/ann-urldownload-jl/37841/4 "2020-04-19T10:09:51Z")

</div>

You are almost correct, the second version should look like

```julia
res = DataFrame(CSV.File(IOBuffer(urldownload(url, parser = identity))))

```

and there will be no difference in the outcome. It’s just a matter of personal taste which form to use.

Regarding `FileIO.jl`, it was my initial approach, unfortunately, `FileIO.jl` (or perhaps libraries that use it) are not very friendly to `IOBuffer` data types. For example, this [bug](https://github.com/JuliaIO/FileIO.jl/issues/174) was fixed only recently, there is no support for `JSON` files and so on. In the end, it was easier to avoid `FileIO.jl` altogether. Maybe at some point, I’ll return `FileIO` but it requires lots of testing.

---

<div class="post-metadata">

**Author:** ![singularitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/singularitti/32/17678_2.png) [@singularitti](https://discourse.julialang.org/u/singularitti)\
**Post date:** [April 19, 2020, 10:44am UTC](https://discourse.julialang.org/t/ann-urldownload-jl/37841/5 "2020-04-19T10:44:02Z")

</div>

`urldownload` differs from [`Base.download`](https://docs.julialang.org/en/v1/base/file/#Base.download) by storing everything in memory, apart from parsers, progress bars, etc. ?

Does it rely on `curl`, `wget`, `fetch`, etc., like `Base.download`?

---

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [April 19, 2020, 12:15pm UTC](https://discourse.julialang.org/t/ann-urldownload-jl/37841/6 "2020-04-19T12:15:07Z")

</div>

It relies currently on `HTTP.jl`, which as far as I understand is pure Julia without any of `curl`, `wget` and so on.

---

<div class="post-metadata">

**Author:** ![carlomontec](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carlomontec/32/2011_2.png) [@carlomontec](https://discourse.julialang.org/u/carlomontec)\
**Post date:** [July 3, 2020, 8:25am UTC](https://discourse.julialang.org/t/ann-urldownload-jl/37841/7 "2020-07-03T08:25:58Z")

</div>

Guys, a related question, do you know how can I test if a link is valid? I have a list of URLS and I want to download them all to disk, but I cant make a loop, since if the link isbroken it sends an error.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [July 3, 2020, 8:28am UTC](https://discourse.julialang.org/t/ann-urldownload-jl/37841/8 "2020-07-03T08:28:57Z")

</div>

Couldn’t you just do standard error handling with `try catch`?

---

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [July 3, 2020, 8:46am UTC](https://discourse.julialang.org/t/ann-urldownload-jl/37841/9 "2020-07-03T08:46:35Z")

</div>

You should use `try catch` block. For example

```julia
using UrlDownload
using Sockets
using DataFrames

urls = ["https://badurl", "https://www.stats.govt.nz/assets/Uploads/Electronic-card-transactions/Electronic-card-transactions-May-2020/Download-data/electronic-card-transactions-may-2020-csv.zip"]
res = DataFrame[]
for url in urls
    try
        push!(res, urldownload(url, true) |> DataFrame)
    catch er
        if er isa Sockets.DNSError
            println("url $url does not exists")
        else
            rethrow()
        end
    end
end

```

You should go through `catch` part and find out which errors you want to skip and which you want to raise error to.

---

<div class="post-metadata">

**Author:** ![carlomontec](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carlomontec/32/2011_2.png) [@carlomontec](https://discourse.julialang.org/u/carlomontec)\
**Post date:** [July 3, 2020, 8:55am UTC](https://discourse.julialang.org/t/ann-urldownload-jl/37841/10 "2020-07-03T08:55:32Z")

</div>

Thank you all ! That was fast . @nilshg @Skoffer