I am frustrated with adding packages which requires downloading large binaries, such as OpenBLAS_jll. I have a very slow internet connection and the downloading typically takes hours to complete (>10MB). And the failing probability is quite high due to unstable internet connection. Can I download the binary using some other software which can resume downloading. After that I can move the downloaded binary to some known location and invoke the installation?
Sorry if this is off mark, but bit-torrent is perfect for that.
How do I use bit-torrent?
Again, this might be totally off-topic, but I mean using a torrenting program (Google “torrent program”) to download anything. Those programs usually handle slow connection and interruptions very well. The problem might be to find a torrent file for the content you need (e.g. OpenBLAS_jll)…
I am not sure about this — torrent protocols send and receive, and while robust, they may actually require more bandwidth.
About the original question: I am not sure if there is a convenient way to download and provide artifacts manually.
That’s exactly what I want. Now I have the OpenBLAS.v0.3.7.x86_64-apple-darwin14-libgfortran5.tar.gz locally. Where can I put it so Pkg can use it instead of downloading it automatically from using its internal downloader (may be via git-repo?)
1 Like
I think you can override artifact locations:
https://julialang.github.io/Pkg.jl/dev/artifacts/#Overriding-artifact-locations-1
1 Like
@dilumaluthge called my attention to this on Slack and I just wanted to post here that it would be possible to support resuming interrupted downloads if the partial download is saved somewhere persistent and the client and server side both support HTTP range requests. We don’t currently allow this, but it’s a good idea, so I’ve created an issue to track it: support resuming downloads · Issue #1731 · JuliaLang/Pkg.jl · GitHub.
3 Likes
The government deliberately constrains the downloading speed of anything from GitHub in China. you may want to try some Chinese mirrors of Julia packages.
You can try using the new Pkg server protocol.
Set the environment variable JULIA_PKG_SERVER to pkg.julialang.org. For example, in Bash:
export JULIA_PKG_SERVER="pkg.julialang.org"
To be noticed, I have used the USTC mirror for installing packages that don’t depend on pre-compiled binaries. However, the mirror seems lack of support of binaries.
Thanks. I will give it a try.
How about slimming down the artifacts? For example, are static libraries really needed?
Kicking out static libraries and the log file (OpenBLAS.log is 5897 kB !) from OpenBLAS_jll would reduce the size-on-disk on Windows from 123.7 MB to 33.6 MB. That should translate to faster downloads.
Other examples on my system
- OpenSSL_jll has 86.8 MB of documentation (!), 8.4 MB static libs
- ImageMagick_jll has 34.1 MB static libs, 12.9 MB docs
packages-directory a lot of potential for size reduction as well.
5 Likes
Along those lines, could somebody explain to me why OpenBLAS_jll is even necessary to begin with? Julia ships with a perfectly functional libopenblas64_.so; why do we need an additional package to provide the same library?
Seems like good ideas, perhaps you can open an issue at GitHub - JuliaPackaging/Yggdrasil: Collection of builder repositories for BinaryBuilder.jl with your suggestions?
May not be the same version.
For most scenarios, the cost of tracking down bugs due to subtle version mismatches, or maintaining software for parallel versions, dominates downloading a few 10MBs of binary blobs.
The solution is to make it easy for the users to provide these manually (in an environment w/o net access), or a more robust downloading protocol.