Unable to automatically install 'Zstd'

New to Julia
1

I am trying to install “Parquet” package in Julia 1.7 on company PC. It is behind Proxy. I have added the following environment variables in the startup script

ENV["HTTP_PROXY"] = "http://xxxx:xxxxx"
ENV["HTTPS_PROXY"] = "http://xxxx:xxxxx"
ENV["JULIA_PKG_SERVER"]= "https://us-west.pkg.julialang.org"
ENV["BINARYPROVIDER_DOWNLOAD_ENGINE"] = "curl"

But I get an error with Zstd. The error reads:

Installed Zstd_jll ──────────────────── v1.5.0+0
Downloaded artifact: Zstd
Downloaded artifact: Zstd
ERROR: Unable to automatically install ‘Zstd’ from ‘/home/xxxxxx/.julia/packages/Zstd_jll/UffgW/Artifacts.toml’
Stacktrace:
…
…

I tried many online suggestions, but could not get around this. Any idea? If it is github and proxy relevant, why it does not occur with many other installations?

Thanks in advance

2

You removed what might be the most useful part of the Stacktrace - the part which says what the actual error was.

3

It is the error:

[1] error(s::String)
@ Base ./error.jl:33
[2] ensure_artifact_installed(name::String, meta::Dict{String, Any}, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, verbose::Bool, quiet_download::Bool, io::Base.TTY)
@ Pkg.Artifacts ~/Julia/julia-1.7.1/share/julia/stdlib/v1.7/Pkg/src/Artifacts.jl:441
[3] download_artifacts(env::Pkg.Types.EnvCache; platform::Base.BinaryPlatforms.Platform, julia_version::VersionNumber, verbose::Bool, io::Base.TTY)
@ Pkg.Operations ~/Julia/julia-1.7.1/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:617
[4] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
@ Pkg.Operations ~/Julia/julia-1.7.1/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:1182
[5] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
@ Pkg.API ~/Julia/julia-1.7.1/share/julia/stdlib/v1.7/Pkg/src/API.jl:268
[6] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Pkg.API ~/Julia/julia-1.7.1/share/julia/stdlib/v1.7/Pkg/src/API.jl:149
[7] add(pkgs::Vector{Pkg.Types.PackageSpec})
@ Pkg.API ~/Julia/julia-1.7.1/share/julia/stdlib/v1.7/Pkg/src/API.jl:144
[8] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL)
@ Pkg.REPLMode ~/Julia/julia-1.7.1/share/julia/stdlib/v1.7/Pkg/src/REPLMode/REPLMode.jl:407
[9] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool)
@ Pkg.REPLMode ~/Julia/julia-1.7.1/share/julia/stdlib/v1.7/Pkg/src/REPLMode/REPLMode.jl:385
[10] do_cmd
@ ~/Julia/julia-1.7.1/share/julia/stdlib/v1.7/Pkg/src/REPLMode/REPLMode.jl:376 [inlined]
[11] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuffer, ok::Bool)
@ Pkg.REPLMode ~/Julia/julia-1.7.1/share/julia/stdlib/v1.7/Pkg/src/REPLMode/REPLMode.jl:549
[12] #invokelatest#2
@ ./essentials.jl:716 [inlined]
[13] invokelatest
@ ./essentials.jl:714 [inlined]
[14] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
@ REPL.LineEdit ~/Julia/julia-1.7.1/share/julia/stdlib/v1.7/REPL/src/LineEdit.jl:2493
[15] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
@ REPL ~/Julia/julia-1.7.1/share/julia/stdlib/v1.7/REPL/src/REPL.jl:1230
[16] (::REPL.var"#49#54"{REPL.LineEditREPL, REPL.REPLBackendRef})()
@ REPL ./task.jl:423

Thanks for support.

4

Oh dear, that is quite useless isn’t it? I see why you didn’t post it before!

5

Indeed it is useless. It gives no insight. I am new to packaging and how it works in Julia. I assume Julia does pull the packages from Git and even though I have added the Proxys to the ENV, but it is lost somewhere when inner packages are pulled. But in soma packages like DataTables where multiple other packages are also installed, there was no problem.

Could the problem be with Registries? What I did now was as following. I dropped ~/.julia folder and tried to update the registry and the following was the outcome:

shell> mv .julia/ _julia

(@v1.6) pkg> up
Installing known registries into ~/.julia
┌ Warning: could not download https://pkg.julialang.org/registries
└ @ Pkg.Types /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Types.jl:997
┌ Warning: could not download https://pkg.julialang.org/registries
└ @ Pkg.Types /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Types.jl:997
Cloning registry from “GitHub - JuliaRegistries/General: The official registry of general Julia packages”
Fetching: [========================================>] 100.0 %
Added registry General to ~/.julia/registries/General
Updating registry at ~/.julia/registries/General
Updating git-repo https://github.com/JuliaRegistries/General.git
No Changes to ~/.julia/environments/v1.6/Project.toml
No Changes to ~/.julia/environments/v1.6/Manifest.toml

(@v1.6) pkg>

Could the problem be explained by these warnings?

6

If you had your proxy settings in .julia/config/startup.jl, those were lost when you tried this experiment, and it just showed what happens when you use Julia without proxy.

Somewhat oversimplified there are three components to package installation:

  • Registries (by default only the General registry)
  • Packages
  • Artifacts (in most cases binary dependencies). Some packages require artifacts, some don’t.

By default all of these are downloaded with https from a package server. If a download fails for some reason, the package manager falls back to

  • getting the General registry with git from github,
  • getting packages with git, in most cases from github,
  • getting artifacts from their original download addresses, frequently but not always with https from github.

This is usually very robust but if you are behind a proxy and some fallback strategy is blocked, you are more vulnerable to package server outage. Potentially you can get inconsistent results for different packages if a) the package server is temporarily down and/or b) the fallbacks are effective for only some packages or artifacts.

To check whether your package server is operational I believe the following should be sufficient:

using Pkg: pkg_server
using Downloads: download
download(pkg_server() * "/registries")
1 Like
7

My statup file is located under installation directory and not .julia/config/startup.jl, Indeed I do not have similar path. I tried your instructions and got the following error:

julia> using Pkg: pkg_server
julia> using Downloads: download
julia> download(pkg_server() * "/registries")
ERROR: HTTP/1.1 407 Proxy Authentication Required (Received HTTP code 407 from proxy after CONNECT) while requesting https://pkg.julialang.org/registries
Stacktrace:
  [1] (::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Bool, String, Int64, Bool, Bool})(easy::Downloads.Curl.Easy)
    @ Downloads /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:369
  [2] with_handle(f::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}},Float64, Nothing, Bool, Bool, String, Int64, Bool, Bool}, handle::Downloads.Curl.Easy)
    @ Downloads.Curl /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/Curl.jl:64
  [3] #8
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:311 [inlined]
  [4] arg_write(f::Downloads.var"#8#17"{Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Bool, String, Int64, Bool, Bool}, arg::IOStream)
    @ ArgTools /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:112
  [5] #7
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:310 [inlined]
  [6] arg_read
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:61 [inlined]
  [7] request(url::String; input::Nothing, output::IOStream, method::Nothing, headers::Vector{Pair{String, String}}, timeout::Float64, progress::Nothing, verbose::Bool, throw::Bool, downloader::Nothing)
    @ Downloads /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:309
  [8] (::Downloads.var"#3#4"{Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Nothing, String})(output::IOStream)
    @ Downloads /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:222
  [9] arg_write(f::Downloads.var"#3#4"{Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Nothing, String}, arg::Nothing)
    @ ArgTools /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:101
 [10] #download#2
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:221 [inlined]
 [11] download (repeats 2 times)
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:221 [inlined]
 [12] top-level scope
    @ REPL[3]:1
8

This error message clearly says that your proxy authentication is not working and considering that it requests from pkg.julialang.org (the default) rather than from us-west.pkg.julialang.org that you were setting in your startup, I strongly suspect that your startup file isn’t read at all. My recommendation is to double-check that you have it in a valid location and after starting Julia, that the ENV global variable contains the expected fields.

1 Like
9

Thanks for reply. I double checked everything and I am not still able to get around this.

julia --banner=no
This is read from Startup file.
Startup under .julia/config/ loaded.
julia> println(ENV["HTTP_PROXY"], "   ", ENV["JULIA_PKG_SERVER"])
http://proxy1:8080   pkg.julialang.org

julia> using Pkg: pkg_server

julia> using Downloads: download

julia> download(pkg_server() * "/registries")
ERROR: HTTP/1.1 407 Proxy Authentication Required (Received HTTP code 407 from proxy after CONNECT) while requesting https://pkg.julialang.org/registries
Stacktrace:
  [1] (::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Bool, String, Int64, Bool, Bool})(easy::Downloads.Curl.Easy)
    @ Downloads /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:369
  [2] with_handle(f::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Bool, String, Int64, Bool, Bool}, handle::Downloads.Curl.Easy)
    @ Downloads.Curl /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/Curl.jl:64
  [3] #8
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:311 [inlined]
  [4] arg_write(f::Downloads.var"#8#17"{Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Bool, String, Int64, Bool, Bool}, arg::IOStream)
    @ ArgTools /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:112
  [5] #7
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:310 [inlined]
  [6] arg_read
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:61 [inlined]
  [7] request(url::String; input::Nothing, output::IOStream, method::Nothing, headers::Vector{Pair{String, String}}, timeout::Float64, progress::Nothing, verbose::Bool, throw::Bool, downloader::Nothing)
    @ Downloads /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:309
  [8] (::Downloads.var"#3#4"{Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Nothing, String})(output::IOStream)
    @ Downloads /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:222
  [9] arg_write(f::Downloads.var"#3#4"{Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Nothing, String}, arg::Nothing)
    @ ArgTools /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:101
 [10] #download#2
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:221 [inlined]
 [11] download (repeats 2 times)
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:221 [inlined]
 [12] top-level scope
    @ REPL[4]:1

It seems that I am not still there to work with Julia. If it was Python or R in a worse scenario I would have installed from source or downloaded the package and built on my own machine. I am lacking enough knowledge on Julia and how packaging is working.

10

If you telnet into the proxy and execute this HTTP request, you will probably get the same 407 error

which is effectively what Julia is doing

$ telnet proxy1 8080
GET http://cnn.com/ HTTP/1.0

this might help

https://github.com/JuliaLang/julia/issues/20948#issuecomment-287305414

11

The solutions mentioned in the link did not resolve the problem. I removed and added the registery and it seeems it can not manage to download the package part.


julia>  Pkg.Registry.add("General")
┌ Warning: could not download https://pkg.julialang.org/registries
└ @ Pkg.Types /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Types.jl:997
     Cloning registry from "https://github.com/JuliaRegistries/General.git"
       Added registry `General` to `~/.julia/registries/General`

The following also did not work:

$  telnet proxy1 8080
Connected to proxy1.
Escape character is '^]'.
GET http://cnn.com/ HTTP/1.0
Connection closed by foreign host.
12

For a regular HTTP proxy, this is what you should get

$ telnet localhost 8118
Connected to localhost.
Escape character is ‘^]’.
GET http://cnn.com HTTP/1.0

HTTP/1.1 301 Moved Permanently
Connection: close
Content-Length: 0
Server: Varnish
Retry-After: 0
Cache-Control: public, max-age=600
Location: http://www.cnn.com/
Accept-Ranges: bytes
Date: Fri, 07 Jan 2022 08:12:28 GMT
Via: 1.1 varnish
Set-Cookie: countryCode=GB; Domain=.cnn.com; Path=/; SameSite=Lax
Set-Cookie: stateCode=KEC; Domain=.cnn.com; Path=/; SameSite=Lax
Set-Cookie: geoData=kensington and chelsea|KEC|sw3 1aa|GB|EU|0|broadband|51.500|-0.160; Domain=.cnn.com; Path=/; SameSite=Lax
X-Served-By: cache-lon4277-LON
X-Cache: HIT
X-Cache-Hits: 0

13

I have done a bit investigations and found out that I can clone the package under ~/.julia/registries and then Pkg first will look into this location to install the packages (https://github.com/JuliaLang/Pkg.jl/issues/1119). At leat it worked on my private machine with no firewall issues. But the same procedure does not work for Artifacts and Zstd.
My goal is to install Parquet package that has dependencies on Zstd and tries to get it, but fails.

14

I can install many package such as CSV and DataFrames, … . But when I install Arrow or Parquet, the Artifact (Zstd) fails to load. looking at https://github.com/JuliaBinaryWrappers/Zstd_jll.jl I see that my machine is supported. Could be something with Hash key in Artifacts.toml file that reads:

[[Zstd]]
arch = "x86_64"
git-tree-sha1 = "e6ce8cd51fdf23fc9c9e8abde61b8a05d2502ac9"
libc = "glibc"
os = "linux"

    [[Zstd.download]]
    sha256 = "f81d29d10ca06a7b37494a7bad4d3395f5491aecdf3c483f08b868642b7fdf73"
    url = "https://github.com/JuliaBinaryWrappers/Zstd_jll.jl/releases/download/Zstd-v1.5.0+0/Zstd.v1.5.0.x86_64-linux-gnu.tar.gz"
15

I finally managed to resolve this issue after a week of trying to learn how package installation in Julia works. It seems that due to Proxy issue I did not manage to properly download the library. What I did was indeed very trivial.
I opened .julia/packages/Zstd_jll/UffgW/Artifacts.toml and there looked for the artifact link that matches architecture of my machine. I downloaded it on my machine (manually and not with Pkg) and since the Artifacts are immutable, I could safely put the downloaded file under .julia/artifacts/.

Then I needed to tell to my machine where to look for the file! I had to update relevant part of .julia/packages/Zstd_jll/UffgW/Artifacts.toml as follwing:

[[Zstd]]
arch = "x86_64"
git-tree-sha1 = "e6ce8cd51fdf23fc9c9e8abde61b8a05d2502ac9"
libc = "glibc"
os = "linux"

    [[Zstd.download]]
    sha256 = "f81d29d10ca06a7b37494a7bad4d3395f5491aecdf3c483f08b868642b7fdf73"
    url = "file:/home/myUserName/.julia/artifacts/Zstd.v1.5.0.x86_64-linux-gnu.tar.gz"

Now back to Julia REPEl, I could install my main library like Pkg.add("XXX") where it will requires Zstd to be already installed.

It may not be in agreement with philosophy of Julia, but due to my limitations, I had to go this way and indeed it resolved my problem. Happy that I did not give up.

1 Like
16

Well done :star_struck: