New Julia install: Downloading artifacts fails behind proxy

General Usage
1

I am installing Julia 1.4.2 on a new computer (MacOS), which is behind an http/https proxy. I have exported environment variables in the shell to set proxy variables:

HTTP_PROXY=http://<company.server>:3128
HTTPS_PROXY=https://<company.server>:3128

and configured git to also use the proxy:

$ git config -l
http.proxy=http://<company.server>:3128/
https.proxy=https://<company.server>:3128/

(See this post for the bananas need to add the trailing slash in the above urls.
https://github.com/JuliaLang/julia/issues/33111#issuecomment-541224149)

With the system configured as above, I am able to add packages to Julia that do not contain artifacts, and all is good:

(testProj) pkg> add Unitful
   Updating registry at `~/.julia/registries/General`
   Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Resolving package versions...
   Updating `~/Documents/code/julia/testProj/Project.toml`
  [1986cc42] + Unitful v1.3.0
   Updating `~/Documents/code/julia/testProj/Manifest.toml`
  [187b0558] + ConstructionBase v1.0.0
  [1986cc42] + Unitful v1.3.0
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [9a3f8284] + Random
  [9e88b42a] + Serialization

However, I cannot add packages that download artifacts:

(testProj) pkg> add IJulia
  Resolving package versions...
Downloading artifact: MbedTLS
#=#=#
Downloading artifact: MbedTLS
#=#=#
ERROR: Unable to automatically install 'MbedTLS' from '/Users/djb0706/.julia/packages/MbedTLS_jll/txOzO/Artifacts.toml'
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] ensure_artifact_installed(::String, ::Dict{String,Any}, ::String; platform::Pkg.BinaryPlatforms.Platform, verbose::Bool, quiet_download::Bool) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Artifacts.jl:894
 [3] ensure_all_artifacts_installed(::String; platform::Pkg.BinaryPlatforms.Platform, pkg_uuid::Nothing, include_lazy::Bool, verbose::Bool, quiet_download::Bool) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Artifacts.jl:958
 [4] download_artifacts(::Pkg.Types.Context, ::Array{String,1}; platform::Pkg.BinaryPlatforms.MacOS, verbose::Bool) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Operations.jl:616
 [5] download_artifacts(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; platform::Pkg.BinaryPlatforms.MacOS, verbose::Bool) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Operations.jl:595

[…AND A BUNCH MORE LINES IN THE STACKTRACE]

As a workaround suggested in this post, I have tried to use the new Pkg protocol by setting the environment variable JULIA_PKG_SERVER=pkg.julilang.org, and deleting the .julia/registries/General directory. But in this case, the server is not reachable (“could not download”), and Pkg falls back to github:

$ rm -rf ~/.julia/registries/General
$ export JULIA_PKG_SERVER="pkg.julilang.org"
$ julia --project=.

(testProj) pkg> add IJulia
    Cloning default registries into `~/.julia`
┌ Warning: could not download https://pkg.julialang.org/registries
â”” @ Pkg.Types /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Types.jl:889
    Cloning registry from "https://github.com/JuliaRegistries/General.git"

[…AND THEN FAILS SIMILARLY TO THE ABOVE]

Any idea how to 1.) download artifacts behind a firewall when Pkg uses github over https behind a proxy, and/or 2.) get the new Pkg protocol to connect to pkg.julialang.org?

Thanks in advance!

2

I have found the solution to the above issue and am documenting it here for future reference. TLDR: misconfigured proxy environment variables. Changing the vars to the following fixed my problem:
http_proxy=http://<company.server>:3128
https_proxy=http://<company.server>:3128

The first thing is realizing that the above failure to add packages with artifacts, stems from how Pkg selects download engines based on operating system. In my case, Pkg was using curl, so focuing on curl usage from the terminal (elimiating Julia altogether) helped to get to the bottom of this. This is an important realization because the error messages and Stacktrace reported in Julia were not helpful.
Furthermore, I believe that Pkg silently falls back to using an internal git library (for which I had apparently set up the proxies well enough in .gitconfig to allow packages without artifacts to be download-able.)

For troubleshooting, getting this command to work is sufficient to making sure this command does not error:
curl -L https://pkg.julialang.org/meta --verbose

Even after figuring out the need to focus on curl, there were two separate problems, which made debugging difficult:

  1. The https_proxy variable was broken: It needs to point to the server at http://<company.server> (note the lack of https).
  2. The http_proxy variable was broken in a different way: The http_proxy environment variable name needs to be lowercase for curl on MacOS to find it. I still can’t believe capitalization would matter, but it absolutely did. The https_proxy env var can be either lower or uppercase. Bananas. The curl docs indicate that it shouldn’t matter, but [shrugs].

Shoutout to @fredrikekre for pointing me in the right direction.

4 Likes
3

This was by far the most helpful thread I found to correcting my corporate firewall proxy problems. However, despite correctly configuring my proxy env variables, I was still not able to connect curl to pkg.julialang.org as above. Therefore, I added
--insecure
to my .curlrc file, which allowed me to install files and operate Julia properly. I realize that I’ve disabled the security however.