Julia Artifact Download Problem: How to tell Julia to follow 301-redirected downloads?

General Usage
1

Hi,
as some other fellow Julia users I want to use Julia behind a corporate firewall which in some cases intercepts downloads, analyzes the content and then downloads the files. Therefore in the beginning I was not able to download any package I wanted. I set up a proxy.sh files and also set the CURL_CA_BUNDLE variable to point to our corporate.cert ssl verify file. This solved 50 percent of the problems, however I am still not able to download some important Packages like “Plots”. I tried to manually download artifacts with curl. As expected this didn’t work, however, when I included the -L flag to follow 301-redirected downloads, i.e. “curl -L …” everything works fine. I am now interested in how I can teach Julia to also use the -L flag.

Edit:
I am using Julias latest Docker image v1.7.3. I have tried setting: export BINARYPROVIDER_DOWNLOAD_ENGINE=“curl -L” to no avail.

2

What version of Julia are you using?

3

I am using the latest julia docker image which should be 1.7.3.

4

That version of Julia uses libcurl for all downloads and always sets the equivalent of the -L flag. Can you share the error output you get? You should probably be setting SSL_CERT_FILE variable, not CURL_CA_BUNDLE.

5

SSL_CERT_FILE and CURL_CA_BUNDLE are both set to the same file. Here is the output for QT5Base with the DebugArtifacs package (as you can see, I already tried all solutions that worked for other users):

debug_artifact("Qt5Base")
[ Info: Platform: Linux x86_64 {cxxstring_abi=cxx11, julia_version=1.7.3, libc=glibc, libgfortran_version=4.0.0, libstdcxx_version=3.4.29}
Julia Version 1.7.3
Commit 742b9abb4d (2022-05-06 12:58 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-4617 0 @ 2.90GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, sandybridge)
Environment:
  JULIA_GPG = 3673DF529D9049477F76B37566E3C7DC03D6E495
  JULIA_VERSION = 1.7.3
  JULIA_PATH = /usr/local/julia
  JULIA_PKG_SERVER = pkg.julialang.org
  JULIA_SSL_NO_VERIFY = **

[ Info: Downloading Artifacts.toml to /tmp/jl_fOWxMa/Artifacts.toml...
[ Info: Extracting artifact info for platform x86_64-linux-gnu-libgfortran4-cxx11-libstdcxx29-julia_version+1.7.3...
[ Info: Found meta object with git-tree-sha1 988066cc974e30c89774b0c471f47201975a4423, attempting download...
ERROR: HTTP/1.0 200 OK (Operation too slow. Less than 1 bytes/sec transferred the last 20 seconds) while requesting https://github.com/JuliaBinaryWrappers/Qt5Base_jll.jl/releases/download/Qt5Base-v5.15.3+1/Qt5Base.v5.15.3.x86_64-linux-gnu-cxx11.tar.gz
Stacktrace:
  [1] (::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Downloads.var"#24#27"{Pkg.PlatformEngines.var"#15#17"{Base.TTY}}, Bool, Nothing, Bool, String, Bool, Bool})(easy::Downloads.Curl.Easy)
    @ Downloads /usr/local/julia/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:387
  [2] with_handle(f::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Downloads.var"#24#27"{Pkg.PlatformEngines.var"#15#17"{Base.TTY}}, Bool, Nothing, Bool, String, Bool, Bool}, handle::Downloads.Curl.Easy)
    @ Downloads.Curl /usr/local/julia/share/julia/stdlib/v1.7/Downloads/src/Curl/Curl.jl:88
  [3] #8
    @ /usr/local/julia/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:328 [inlined]
  [4] arg_write(f::Downloads.var"#8#17"{Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Downloads.var"#24#27"{Pkg.PlatformEngines.var"#15#17"{Base.TTY}}, Bool, Nothing, Bool, String, Bool, Bool}, arg::IOStream)
    @ ArgTools /usr/local/julia/share/julia/stdlib/v1.7/ArgTools/src/ArgTools.jl:112
  [5] #7
    @ /usr/local/julia/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:327 [inlined]
  [6] arg_read(f::Downloads.var"#7#16"{IOStream, Nothing, Vector{Pair{String, String}}, Float64, Downloads.var"#24#27"{Pkg.PlatformEngines.var"#15#17"{Base.TTY}}, Bool, Nothing, Bool, String, Bool, Bool}, arg::Base.DevNull)
    @ ArgTools /usr/local/julia/share/julia/stdlib/v1.7/ArgTools/src/ArgTools.jl:61
  [7] request(url::String; input::Nothing, output::IOStream, method::Nothing, headers::Vector{Pair{String, String}}, timeout::Float64, progress::Pkg.PlatformEngines.var"#15#17"{Base.TTY}, verbose::Bool, debug::Nothing, throw::Bool, downloader::Nothing)
    @ Downloads /usr/local/julia/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:326
  [8] #3
    @ /usr/local/julia/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:231 [inlined]
  [9] open(f::Downloads.var"#3#4"{Nothing, Vector{Pair{String, String}}, Float64, Pkg.PlatformEngines.var"#15#17"{Base.TTY}, Bool, Nothing, Nothing, String}, args::String; kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:write,), Tuple{Bool}}})
    @ Base ./io.jl:330
 [10] arg_write(f::Function, arg::String)
    @ ArgTools /usr/local/julia/share/julia/stdlib/v1.7/ArgTools/src/ArgTools.jl:86
 [11] #download#2
    @ /usr/local/julia/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:230 [inlined]
 [12] download(url::String, dest::String; verbose::Bool, headers::Vector{Pair{String, String}}, auth_header::Nothing, io::Base.TTY)
    @ Pkg.PlatformEngines /usr/local/julia/share/julia/stdlib/v1.7/Pkg/src/PlatformEngines.jl:282
 [13] download_verify(url::String, hash::String, dest::String; verbose::Bool, force::Bool, quiet_download::Bool)
    @ Pkg.PlatformEngines /usr/local/julia/share/julia/stdlib/v1.7/Pkg/src/PlatformEngines.jl:345
 [14] download_verify_unpack(url::String, hash::String, dest::String; tarball_path::Nothing, ignore_existence::Bool, force::Bool, verbose::Bool, quiet_download::Bool, io::Base.TTY)
    @ Pkg.PlatformEngines /usr/local/julia/share/julia/stdlib/v1.7/Pkg/src/PlatformEngines.jl:500
 [15] (::DebugArtifacts.var"#3#4"{String, Base.BinaryPlatforms.Platform, String})(tmp_dir::String)
    @ DebugArtifacts ~/.julia/packages/DebugArtifacts/2VJet/src/DebugArtifacts.jl:67
 [16] mktempdir(fn::DebugArtifacts.var"#3#4"{String, Base.BinaryPlatforms.Platform, String}, parent::String; prefix::String)
    @ Base.Filesystem ./file.jl:750
 [17] mktempdir (repeats 2 times)
    @ ./file.jl:748 [inlined]
 [18] debug_artifact(artifact_name::String, platform::Base.BinaryPlatforms.Platform)
    @ DebugArtifacts ~/.julia/packages/DebugArtifacts/2VJet/src/DebugArtifacts.jl:55
 [19] debug_artifact(artifact_name::String)
    @ DebugArtifacts ~/.julia/packages/DebugArtifacts/2VJet/src/DebugArtifacts.jl:43
 [20] top-level scope
    @ REPL[6]:1
6

Looks like the client isn’t getting any data for a long time—no data in over 20 seconds. It might be that the proxy waits until the entire file is received before sending any data to you. You could try doing the following before trying debug_artifact("Qt5Base") again:

using Downloads
Downloads.EASY_HOOK[] = (easy, info) -> begin
    Curl.setopt(easy, Curl.CURLOPT_LOW_SPEED_TIME, 0)
end
7

Now I get an error that Curl is not defined:

ERROR: UndefVarError: Curl not defined
8

Sorry, about that, the correct code is this:

using Downloads
Downloads.DOWNLOADER[] = nothing
Downloads.EASY_HOOK[] = (easy, info) -> begin
    Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_LOW_SPEED_TIME, 0)
end
1 Like
9

Thanks, that worked, had to wait 3 min, but then it started downloading and after that everything went super fast.

10

Ok, that seems to verify that the issue is the MITM proxy buffering entire downloads. I wonder what happens with a really huge download :flushed:. Does the proxy buffer an entire 50GB download before returning anything? For most users having a timeout if the server doesn’t send any data for a while is a much better experience since a normal server will reply to such a request incrementally, but clearly not in this case. You can put that hook in your startup script to disable the low speed timeout always.