# \[ANN\] RemoteFiles.jl: Download files from the Internet and keep them up-to-date

**URL:** <https://discourse.julialang.org/t/ann-remotefiles-jl-download-files-from-the-internet-and-keep-them-up-to-date/5142>\
**Category:** Community\
**Created:** [July 30, 2017, 7:28pm UTC](https://discourse.julialang.org/t/ann-remotefiles-jl-download-files-from-the-internet-and-keep-them-up-to-date/5142 "2017-07-30T19:28:39Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![helgee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/helgee/32/2022_2.png) [@helgee](https://discourse.julialang.org/u/helgee)\
**Post date:** [July 30, 2017, 7:28pm UTC](https://discourse.julialang.org/t/ann-remotefiles-jl-download-files-from-the-internet-and-keep-them-up-to-date/5142/1 "2017-07-30T19:28:39Z")

</div>

The title says it all 😉

Remote files are declared through the `@RemoteFile` macro:

```julia
using RemoteFiles

@RemoteFile(JULIA_BINARY, "https://status.julialang.org/download/win64",
    file="julia-nightly-x64.exe", updates=:daily)

# Download the file if it is out-of-date
download(JULIA_BINARY)

# Check whether the file has been downloaded
isfile(JULIA_BINARY)

# Get the path
path(JULIA_BINARY)

```

By default the file is downloaded to `Pkg.dir(CURRENT_PACKAGE)/data`.  
This can be customized with the `dir` keyword argument to the `@RemoteFile` macro.

`RemoteFile`s can be grouped together in a `RemoteFileSet`:

```julia
@RemoteFileSet BINARIES "Julia Binaries" begin
    win = @RemoteFile "https://julialang-s3.julialang.org/bin/winnt/x64/0.6/julia-0.6.0-win64.exe"
    osx = @RemoteFile "https://julialang-s3.julialang.org/bin/osx/x64/0.6/julia-0.6.0-osx10.7+.dmg"
end

# Download all of them

download(BINARIES)

```

Have a look [here](https://github.com/helgee/RemoteFiles.jl).

---

<div class="post-metadata">

**Author:** ![anon94023334](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@anon94023334](https://discourse.julialang.org/u/anon94023334)\
**Post date:** [July 31, 2017, 3:35am UTC](https://discourse.julialang.org/t/ann-remotefiles-jl-download-files-from-the-internet-and-keep-them-up-to-date/5142/2 "2017-07-31T03:35:46Z")

</div>

Can you explain more about how `updates=:daily` works?

---

<div class="post-metadata">

**Author:** ![helgee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/helgee/32/2022_2.png) [@helgee](https://discourse.julialang.org/u/helgee)\
**Post date:** [July 31, 2017, 5:19am UTC](https://discourse.julialang.org/t/ann-remotefiles-jl-download-files-from-the-internet-and-keep-them-up-to-date/5142/3 "2017-07-31T05:19:17Z")

</div>

Before starting the download it checks this condition: `(last, now) -> Date(now) > Date(last)` where `last` is the time when the file was last modified and `now` is the current time.

These are the other possible values:

- `:never`  
- `:daily`  
- `:monthly`  
- `:yearly`  
- `:mondays`/`:weekly`, `:tuesdays`, etc.

---

<div class="post-metadata">

**Author:** ![Per](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/per/32/10387_2.png) [@Per](https://discourse.julialang.org/u/Per)\
**Post date:** [July 31, 2017, 8:05am UTC](https://discourse.julialang.org/t/ann-remotefiles-jl-download-files-from-the-internet-and-keep-them-up-to-date/5142/4 "2017-07-31T08:05:04Z")

</div>

Looking at the code, I can’t see that you’re actually checking if the file needs to be updated. Your `isoutdated` function only looks at update frequency - not the file modification date on the server.

You should check the remote date, or use the `If-Modified-Since` header (the `-z` flag with `curl`) to reduce strain on the server.

---

<div class="post-metadata">

**Author:** ![helgee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/helgee/32/2022_2.png) [@helgee](https://discourse.julialang.org/u/helgee)\
**Post date:** [July 31, 2017, 8:19am UTC](https://discourse.julialang.org/t/ann-remotefiles-jl-download-files-from-the-internet-and-keep-them-up-to-date/5142/5 "2017-07-31T08:19:41Z")

</div>

PR? 😉

---

<div class="post-metadata">

**Author:** ![Per](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/per/32/10387_2.png) [@Per](https://discourse.julialang.org/u/Per)\
**Post date:** [July 31, 2017, 8:37am UTC](https://discourse.julialang.org/t/ann-remotefiles-jl-download-files-from-the-internet-and-keep-them-up-to-date/5142/6 "2017-07-31T08:37:52Z")

</div>

Sorry, I don’t have time to do a PR right now, but I suspect all you have to do for the non-windows case is add ` -z` after `curl`.

---

<div class="post-metadata">

**Author:** ![helgee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/helgee/32/2022_2.png) [@helgee](https://discourse.julialang.org/u/helgee)\
**Post date:** [July 31, 2017, 8:49am UTC](https://discourse.julialang.org/t/ann-remotefiles-jl-download-files-from-the-internet-and-keep-them-up-to-date/5142/7 "2017-07-31T08:49:15Z")

</div>

I think a proper solutions is a bit more involved. Actually checking on the server whether the file was modified would make the `updates` parameter obsolete, would it not?

A cross-platform-compatible solution should also probably use [LibCURL.jl](https://github.com/JuliaWeb/LibCURL.jl). But it might take some time until I am able to tackle that.

---

<div class="post-metadata">

**Author:** ![Per](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/per/32/10387_2.png) [@Per](https://discourse.julialang.org/u/Per)\
**Post date:** [July 31, 2017, 9:11am UTC](https://discourse.julialang.org/t/ann-remotefiles-jl-download-files-from-the-internet-and-keep-them-up-to-date/5142/8 "2017-07-31T09:11:34Z")

</div>

> [@helgee](#):
>
> Actually checking on the server whether the file was modified would make the updates parameter obsolete, would it not?

No. The `updates` parameter would still determine how frequently the server is contacted. (Checking headers can also overload a server unless the rate is limited.) The `If-Modified-Since` header is to avoid downloading the exact same file twice. Both mechanisms are necessary.

If you publish a package that downloads large files frequently and unnecessarily, and that package gets popular, then people who pay for server bandwidth will get annoyed with you. For example, if 10 000 people download a 100 megabyte file each day, even though the file has not changed, then that’s about 100 dollars a day in extra bandwidth cost.