Unable to automatically install 'OpenSpecFun' when trying to add 'Symbolics'

Aleksandr_Mikheev · May 12, 2021, 3:51pm

Yes, it still does:

printenv | grep proxy
http_proxy=http://proxy.my.institu.te:8080/

julia>  Downloads.download("http://pkg.julialang.org/meta", stdout; verbose=true)
* Uses proxy env variable http_proxy == 'http://proxy.my.institu.te:8080/'
*   Trying ***.***.***.***:8080...
* Connected to proxy.my.institu.te (***.***.***.***) port 8080 (#0)
> GET http://pkg.julialang.org/meta HTTP/1.1
Host: pkg.julialang.org
Accept: */*
Proxy-Connection: Keep-Alive
User-Agent: curl/7.73.0 julia/1.6

* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Server: Varnish
< Retry-After: 0
< Location: https://eu-central.pkg.julialang.org/meta
< x-geo-continent: EU
< x-geo-country: DE
< x-geo-region: BW
< Content-Length: 0
< Accept-Ranges: bytes
< Date: Wed, 12 May 2021 15:40:32 GMT
< X-Served-By: cache-fra19171-FRA
< X-Cache: HIT
< X-Cache-Hits: 0
< X-Timer: S1620834033.673052,VS0,VE0
< X-Cache: MISS from proxy1.my.institu.te
< X-Cache-Lookup: MISS from proxy1.my.institu.te:3128
< Via: 1.1 varnish, 1.1 proxy1.my.institu.te (squid/4.6)
< Connection: keep-alive
<
* Connection #0 to host proxy.my.institu.te left intact
* Issue another request to this URL: 'https://eu-central.pkg.julialang.org/meta'
*   Trying 18.157.163.169:443...
* connect to 18.157.163.169 port 443 failed: Connection timed out
* Failed to connect to eu-central.pkg.julialang.org port 443: Connection timed out
* Closing connection 1
ERROR: HTTP/1.1 301 Moved Permanently (Failed to connect to eu-central.pkg.julialang.org port 443: Connection timed out) while requesting http://pkg.julialang.org/meta
Stacktrace:
...

staticfloat · May 12, 2021, 4:18pm

And finally, what happens if you set https_proxy=http://proxy.my.institu.te:8080/, that is, you set https_proxy to point to your proxy, but you don’t set https:// as the scheme?

Aleksandr_Mikheev · May 12, 2021, 7:01pm

If I understood what you meant correctly,

printenv | grep proxy
https_proxy=http://proxy.my.institu.te:8080/

then here is the output of Downloads.download():

Downloads.download("http://pkg.julialang.org/meta", stdout; verbose=true)
*   Trying 151.101.14.217:80...
*   Trying 2a04:4e42:3::729:80...
* Immediate connect fail for 2a04:4e42:3::729: Network is unreachable
* connect to 151.101.14.217 port 80 failed: Connection timed out
* Failed to connect to pkg.julialang.org port 80: Connection timed out
* Closing connection 0
ERROR: Failed to connect to pkg.julialang.org port 80: Connection timed out while requesting http://pkg.julialang.org/meta

staticfloat · May 12, 2021, 8:44pm

Sorry, I actually meant in addition to the http_proxy.

Aleksandr_Mikheev · May 13, 2021, 2:15am

Like so?

printenv | grep proxy
http_proxy=http://proxy.my.institu.te:8080/
https_proxy=http://proxy.my.institu.te:8080/

It all worked now! I was able to add Symbolics, Plots, etc. Thank you very much!

If it’s not too much trouble, could you please also briefly explain what seemed to be the problem? Maybe I should also tell something to my IT department so that they can make changes on our end? After all, I am definitely not the only person in my institute who is and will be using Julia…

staticfloat · May 13, 2021, 4:01am

Yes, I’d be happy to explain what happened.

So first off, it’s important to use your university’s proxy, and we needed to communicate it to Julia (really libcurl being used by Julia). Once we did that however, there was the issue that libcurl, when using schannel (The default TLS implementation on Windows) does not support HTTPS connections to proxies! This is what the libcurl is built without the HTTPS-proxy support. error message meant. Luckily, that doesn’t mean you can’t use your proxy to connect to HTTPS websites, (which all of our package servers are), it just means that the connection between your computer and the proxy will not be encrypted. This is apparently a fundamental restriction of libcurl on Windows when using schannel, which is a shame, but at least there is a workaround.

Aleksandr_Mikheev · May 15, 2021, 11:06pm

Sorry for the delayed response, but I am still slightly confused. Let me try to reiterate your explanation as I (mis)understand it:

From what I understand, the network in my institute is organized such that if I want to access something outside the local network, I have to do it via proxy server. Therefore, in order to use the Internet, one first has to set it up.
However, as we established before, until post #19 my proxy environment variables were not set up for a standard user. But then how could I download Julia itself and certain packages from the Internet? Why did I not experience any problem there?
I am also a bit confused about the Windows remark. My remote machine at the institute (the one on which I use Julia) runs on Debian. I then access it via SSH, and it is my local machine that runs on Windows. But what does my local machine have to do with network issues of the remote machine on which I run Julia?

staticfloat · May 16, 2021, 1:03am

From what I understand, the network in my institute is organized such that if I want to access something outside the local network, I have to do it via proxy server.

I assume this is true from the evidence presented here; but you would know better than I!

But then how could I download Julia itself and certain packages from the Internet? Why did I not experience any problem there?

Likely because the way you set proxy settings for browsers is generally not by providing environment variables; there are standard methods to provide proxy settings for browsers that are independent from what the rest of the system does. Here’s an example for Firefox. Most browsers can look at “system-default” settings, if such a thing exists (which it does on Windows and MacOS at least, unsure about Linux).

I then access it via SSH, and it is my local machine that runs on Windows.

Whoops! Somehow I crossed my wires and thought that you were running on Windows, but I can see from your pasted logs that this is clearly not the case. The effect is the same however, on Linux we use the MbedTLS backend, I believe, which doesn’t look like it supports HTTPS proxy either.

qsi · May 17, 2021, 8:49am

Apologies for the late reply! As you suggested, I downloaded the latest build which shows up as Version 1.6.1 (2021-04-23). Unfortunately, it does not solve my problem as adding Plots still results in an error:

(@v1.6) pkg> add Plots
    Updating registry at `C:\Users\me\.julia\registries\General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
  Downloaded artifact: Libgpg_error
  Downloaded artifact: Libgpg_error
ERROR: Unable to automatically install 'Libgpg_error' from 'C:\Users\me\.julia\packages\Libgpg_error_jll\3LF6G\Artifacts.toml'
Stacktrace:
  [1] error(s::String)
    @ Base .\error.jl:33
  [2] ensure_artifact_installed(name::String, meta::Dict{String, Any}, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, verbose::Bool, quiet_download::Bool, io::Base.TTY)
    @ Pkg.Artifacts C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\Artifacts.jl:443
  [3] ensure_all_artifacts_installed(artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, pkg_uuid::Nothing, include_lazy::Bool, verbose::Bool, quiet_download::Bool, io::Base.TTY)
    @ Pkg.Artifacts C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\Artifacts.jl:508
  [4] download_artifacts(ctx::Pkg.Types.Context, pkg_roots::Vector{String}; platform::Base.BinaryPlatforms.Platform, verbose::Bool, io::Base.TTY)
    @ Pkg.Operations C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\Operations.jl:706
  [5] download_artifacts(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; platform::Base.BinaryPlatforms.Platform, julia_version::VersionNumber, verbose::Bool, io::Base.TTY)
    @ Pkg.Operations C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\Operations.jl:683
  [6] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Vector{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\Operations.jl:1237
  [7] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Iterators.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
    @ Pkg.API C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\API.jl:203
  [8] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\API.jl:79
  [9] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\API.jl:77
 [10] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL)
    @ Pkg.REPLMode C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\REPLMode\REPLMode.jl:408
 [11] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool)
    @ Pkg.REPLMode C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\REPLMode\REPLMode.jl:386
 [12] do_cmd
    @ C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\REPLMode\REPLMode.jl:377 [inlined]
 [13] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuffer, ok::Bool)
    @ Pkg.REPLMode C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\REPLMode\REPLMode.jl:550
 [14] #invokelatest#2
    @ .\essentials.jl:708 [inlined]
 [15] invokelatest
    @ .\essentials.jl:706 [inlined]
 [16] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\REPL\src\LineEdit.jl:2441
 [17] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\REPL\src\REPL.jl:1126
 [18] (::REPL.var"#44#49"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL .\task.jl:411

julia>

The Downloads.download() also still gives the same error. I tried various workarounds to add the proxy certificate to my machine, but this exceeds both my knowledge as well as my administrative rights (corporate environment). So rather than bug IT about this, I read through the thread on issue #108 that you linked, and after some experimentation, I can get it to work by going the insecure route of ENV["JULIA_SSL_NO_VERIFY_HOSTS"] = "**".

This is obviously not ideal but until Julia becomes a standard install at work I think may have to run it like this.

Thank you again for your invaluable help!