Downloading packages fails on work network

Hi, I’ve seen a couple of threads on this but to no avail. Here is my error message:

IOError(Base.IOError(“connect: connection refused (ECONNREFUSED)”, -4078) during
request(https://pkg.juliacomputing.com/package/.yank.toml))

I’ve tried everything from this thread, but I get the same error. Can this be resolved by any means other than getting in touch with IT?

One simple thing you can do as a sanity check is just to see if ping github.com works. If this hangs, it means you can’t talk to github and there’s no way of resolving this without telling your IT to unblock github. It might be possible for them only to block specific ports however, so if ping works, you might still be blocked. At least if it fails, you know you probably won’t have much choice other than dealing with IT.

You can also try nmap github.com, from which you should see the following

PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
443/tcp  open  https
9418/tcp open  git

You may not have nmap by default, in which case you’d have to install it with your package manager (e.g. sudo apt install nmap).

1 Like

Thanks for the prompt reply. Sadly, github doesn’t get pinged and is therefore blocked.

I am not sure, but just pinging and failing does not mean that you can’t access github using e.g. http or https.
In the referenced thread there is also missing, that git maybe have to be set to use http/https protocol instead of the git protocol, i.e. changing git://url/ to https://url/.

In the Julia REPL try

run(`git config --global url."https://".insteadOf git://`)
run(`git config --global http.proxy `http://XX:3128`)

Of course you have to set

XX:3128

to your specific proxy host and port (host:port).
Full syntax is http://proxyUsername:proxyPassword@proxy.server.com:port

You may have to search again for git proxy or julia git proxy. There are quite some answers but which one resolves your issue is a matter of try and error (or asking your administrators).

So your company bans GitHub on work network…?

1 Like

I tried this from the thread I linked to. It isn’t working still. Though after making the change I can push and pull for github, so I really don’t get it.

After testing it a bit, I cannot clone, push or pull without the proxy settings. Once I have the settings up I can easily clone, etc.

Perhaps something on the end of my IDE? I am using JuliaPro (Atom)

Maybe a JuliaPro issue on top, but I don’t know.
Best is to just download the julia stand alone release https://julialang.org/downloads/
just to rule out any other influence.
By the way, is this windows, mac or linux/unix? Because of easyness of installations.

So, I understand that git is now working for you, but only julia packages using add is still not working?
If this is the case, some more error outputs would be helpfull.

Bingo! Julia.exe manages to download and build packages without any hitches! Thank you!

I have two systems, my desk computer (windows 7) and the server (debian) that I do my computations on. I’ve been testing everything on the windows 7 pc, since the server needs admin privileges to install things, etc.

So…any way I can get JuliaPro on board with the proxy?

Can you give me an example, which package is failing with JuliaPro?
Than I can try it tomorrow on a windows machine behind a proxy.

Plots for one

Problem: On Windows Julia and JuliaPro relies on powershell to download sources or binaries but powershell is not proxy aware per default

Scroll down to go directly to a possible solution.

Downloaded JuliaPro Current Stable Release (1.2.0-1) for Windows.
Installed with unchanged default settings on Windows 10 64Bit with non admin user account and network behind proxy.

Clicking on the Julia 1.2.0-1 icon on the desktop opens the IDE (atom).

Entered the REPL inside this IDE.

v1.2) pkg> add Plots
  Updating registry at `C:\Users\oheil\.juliapro\JuliaPro_v1.2.0-1\registries\JuliaPro`
[ Info: Please Authenticate..

in between authenticated via GitHub, successfull.

Expected error output
+ Warning: IOError(Base.IOError("connect: connection timed out (ETIMEDOUT)", -4039) during request(https://pkg.juliacomputing.com/package/.yank.toml))
+ @ Pkg.Operations C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Pkg\src\Operations.jl:485
  Updating git-repo `https://pkg.juliacomputing.com//registry/JuliaPro`
 Resolving package versions...
ERROR: Error when installing package Plots:
IOError(Base.IOError("connect: connection timed out (ETIMEDOUT)", -4039) during request(https://pkg.juliacomputing.com//package/91a5bcdd-55d7-5caf-9e0b-520d859cae80/0d7cf139b508defb467ac4fd655f4bd4d92f30cf.tar.gz))

Stacktrace:
 [1] try_yieldto(::typeof(Base.ensure_rescheduled), ::Base.RefValue{Task}) at .\task.jl:517
 [2] wait() at .\task.jl:592
 [3] wait(::Base.GenericCondition{Base.Threads.SpinLock}) at .\condition.jl:104
 [4] stream_wait(::Sockets.TCPSocket, ::Base.GenericCondition{Base.Threads.SpinLock}) at .\stream.jl:47
 [5] wait_connected(::Sockets.TCPSocket) at .\stream.jl:332
 [6] connect at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Sockets\src\Sockets.jl:470 [inlined]
 [7] connect(::Sockets.IPv4, ::UInt64) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Sockets\src\Sockets.jl:454
 [8] #getconnection#14(::Bool, ::Int64, ::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:require_ssl_verification, :iofunction),Tuple{Bool,getfield(CredentialsHandler, Symbol("##1#3")){String}}}}, ::typeof(HTTP.ConnectionPool.getconnection), ::Type{Sockets.TCPSocket}, ::SubString{String}, ::String) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\HTTP\src\ConnectionPool.jl:566
 [9] #getconnection#11(::Int64, ::Int64, ::Int64, ::Int64, ::Bool, ::Base.Iterators.Pairs{Symbol,getfield(CredentialsHandler, Symbol("##1#3")){String},Tuple{Symbol},NamedTuple{(:iofunction,),Tuple{getfield(CredentialsHandler, Symbol("##1#3")){String}}}}, ::typeof(HTTP.ConnectionPool.getconnection), ::Type{HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}}, ::SubString{String}, ::SubString{String}) at .\none:0
 [10] (::getfield(HTTP.ConnectionPool, Symbol("#kw##getconnection")))(::NamedTuple{(:reuse_limit, :iofunction),Tuple{Int64,getfield(CredentialsHandler, Symbol("##1#3")){String}}}, ::typeof(HTTP.ConnectionPool.getconnection), ::Type{HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}}, ::SubString{String}, ::SubString{String}) at .\none:0
 [11] #request#1(::Nothing, ::Type, ::Int64, ::Base.Iterators.Pairs{Symbol,getfield(CredentialsHandler, Symbol("##1#3")){String},Tuple{Symbol},NamedTuple{(:iofunction,),Tuple{getfield(CredentialsHandler, Symbol("##1#3")){String}}}}, ::typeof(HTTP.request), ::Type{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Nothing) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\HTTP\src\ConnectionRequest.jl:41
 [12] (::getfield(HTTP, Symbol("#kw##request")))(::NamedTuple{(:iofunction,),Tuple{getfield(CredentialsHandler, Symbol("##1#3")){String}}}, ::typeof(HTTP.request), ::Type{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Nothing) at .\none:0
 [13] (::getfield(Base, Symbol("###50#51#53")){ExponentialBackOff,getfield(HTTP.RetryRequest, Symbol("##2#3")){Bool,HTTP.Messages.Request},typeof(HTTP.request)})(::Base.Iterators.Pairs{Symbol,getfield(CredentialsHandler, Symbol("##1#3")){String},Tuple{Symbol},NamedTuple{(:iofunction,),Tuple{getfield(CredentialsHandler, Symbol("##1#3")){String}}}}, ::getfield(Base, Symbol("##50#52")){getfield(Base, Symbol("###50#51#53")){ExponentialBackOff,getfield(HTTP.RetryRequest, Symbol("##2#3")){Bool,HTTP.Messages.Request},typeof(HTTP.request)}}, ::Type, ::Vararg{Any,N} where N) at .\error.jl:262
 [14] ##50#52 at .\none:0 [inlined]
 [15] #request#1 at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\HTTP\src\RetryRequest.jl:44 [inlined]
 [16] (::getfield(HTTP, Symbol("#kw##request")))(::NamedTuple{(:iofunction,),Tuple{getfield(CredentialsHandler, Symbol("##1#3")){String}}}, ::typeof(HTTP.request), ::Type{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Nothing) at .\none:0
 [17] #request#1(::VersionNumber, ::String, ::Nothing, ::Function, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(HTTP.request), ::Type{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Nothing) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\HTTP\src\MessageRequest.jl:47
 [18] (::getfield(HTTP, Symbol("#kw##request")))(::NamedTuple{(:iofunction,),Tuple{getfield(CredentialsHandler, Symbol("##1#3")){String}}}, ::typeof(HTTP.request), ::Type{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Nothing) at .\none:0
 [19] #request#1(::Int64, ::Bool, ::Base.Iterators.Pairs{Symbol,getfield(CredentialsHandler, Symbol("##1#3")){String},Tuple{Symbol},NamedTuple{(:iofunction,),Tuple{getfield(CredentialsHandler, Symbol("##1#3")){String}}}}, ::typeof(HTTP.request), ::Type{HTTP.RedirectRequest.RedirectLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Nothing) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\HTTP\src\RedirectRequest.jl:24
 [20] (::getfield(HTTP, Symbol("#kw##request")))(::NamedTuple{(:iofunction,),Tuple{getfield(CredentialsHandler, Symbol("##1#3")){String}}}, ::typeof(HTTP.request), ::Type{HTTP.RedirectRequest.RedirectLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Nothing) at .\none:0
 [21] #request#5(::Base.Iterators.Pairs{Symbol,getfield(CredentialsHandler, Symbol("##1#3")){String},Tuple{Symbol},NamedTuple{(:iofunction,),Tuple{getfield(CredentialsHandler, Symbol("##1#3")){String}}}}, ::typeof(HTTP.request), ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Nothing) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\HTTP\src\HTTP.jl:300
 [22] #request at .\none:0 [inlined]
 [23] #request#6 at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\HTTP\src\HTTP.jl:314 [inlined]
 [24] #request at .\none:0 [inlined]
 [25] #open#7 at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\HTTP\src\HTTP.jl:339 [inlined]
 [26] open at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\HTTP\src\HTTP.jl:339 [inlined]
 [27] download at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\CredentialsHandler\src\CredentialsHandler.jl:89 [inlined] (repeats 2 times)
 [28] (::getfield(BinaryProvider.JuliaTeam, Symbol("##1#2")){getfield(BinaryProvider, Symbol("##51#76")){getfield(BinaryProvider, Symbol("##46#71")){Cmd,String}}})(::String, ::String)
at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\BinaryProvider\src\JuliaTeam.jl:18
 [29] #download#97(::Bool, ::typeof(BinaryProvider.download), ::String, ::String) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\BinaryProvider\src\PlatformEngines.jl:494
 [30] download at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\BinaryProvider\src\PlatformEngines.jl:493 [inlined]
 [31] install_archive(::Array{String,1}, ::Base.SHA1, ::Base.UUID, ::String) at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Pkg\src\Operations.jl:517
 [32] macro expansion at C:\Users\julia\AppData\Local\Julia-1.2.0\share\julia\stdlib\v1.2\Pkg\src\Operations.jl:659 [inlined]
 [33] (::getfield(Pkg.Operations, Symbol("##26#29")){Symbol,Pkg.Types.Context,Dict{Base.UUID,Base.SHA1},Dict{Base.UUID,Array{String,1}},Channel{Any},Channel{Any}})() at .\task.jl:268

(v1.2) pkg> 

These environment settings have been present:

julia> ENV["HTTP_PROXY"]
"http://our-proxy-host:3128"

julia> ENV["HTTPS_PROXY"]
"http://our-proxy-host:3128"

Running the recommended git commands above in the Julia REPL, e.g.

run(`git config --global http.proxy `http://XX:3128`)

is problematic, because JuliaPro brings its own git installation with a quite cryptic path, so above command should be in my JuliaPro installation:

julia> run(`C:\\Users\\oheil\\AppData\\Local\\JuliaPro-1.2.0-1\\app-1.40.1\\resources\\app.asar.unpacked\\node_modules\\dugite\\git\\mingw64\\bin\\git config --global http.proxy "http://our-proxy-host:3128"`)
julia> run(`C:\\Users\\oheil\\AppData\\Local\\JuliaPro-1.2.0-1\\app-1.40.1\\resources\\app.asar.unpacked\\node_modules\\dugite\\git\\mingw64\\bin\\git config --global https.proxy "http://our-proxy-host:3128"`)
run(`C:\\Users\\oheil\\AppData\\Local\\JuliaPro-1.2.0-1\\app-1.40.1\\resources\\app.asar.unpacked\\node_modules\\dugite\\git\\mingw64\\bin\\git config --global url."https://".insteadOf git://`)

You have to change the username “oheil” to your and check if the path exists in your installation.

These commands write the proper configuration lines in the .gitconfig file which is located in your home:

...
[http]
	proxy = http://OUR-PROXY:3128
[https]
	proxy = http://OUR-PROXY:3128
[url "https://"]
	insteadOf = git://
...

So far, this is not enough to make JuliaPro aware of the proxy server.

After hours of trying and searching I probably found a workaround but for me this workaround needs local administrator rights. This may be not necessary for everybody.

In JuliaPro installing any package from the web not only uses some git api requests but also requires some downloads of binaries. This is done via module BinaryProvider (see file PlatformEngines.jl in C:\Users\oheil\AppData\Local\JuliaPro-1.2.0-1\Julia-1.2.0\share\julia\stdlib\v1.2\BinaryProvider\src). The relevant code is this:

PlatformEngines.jl snippet
function probe_platform_engines!(;verbose::Bool = false)
    global gen_download_cmd, gen_list_tarball_cmd, gen_package_cmd
    global gen_unpack_cmd, parse_tarball_listing, gen_sh_cmd
    global tempdir_symlink_creation
    # First things first, determine whether tempdir() can have symlinks created
    # within it.  This is important for our copyderef workaround for e.g. SMBFS
    if Sys.iswindows()
        tempdir_symlink_creation = false
    else
        tempdir_symlink_creation = probe_symlink_creation(tempdir())
    end
    if verbose
        @info("Symlinks allowed in $(tempdir()): $(tempdir_symlink_creation)")
    end

    agent = "BinaryProvider.jl (https://github.com/JuliaPackaging/BinaryProvider.jl)"
    # download_engines is a list of (test_cmd, download_opts_functor)
    # The probulator will check each of them by attempting to run `$test_cmd`,
    # and if that works, will set the global download functions appropriately.
    download_engines = [
        (`curl --help`, (url, path) -> `curl -H "User-Agent: $agent" -C - -\# -f -o $path -L $url`),
        (`wget --help`, (url, path) -> `wget --tries=5 -U $agent -c -O $path $url`),
        (`fetch --help`, (url, path) -> `fetch --user-agent=$agent -f $path $url`),
        (`busybox wget --help`, (url, path) -> `busybox wget -U $agent -c -O $path $url`),
    ]

    # 7z is rather intensely verbose.  We also want to try running not only
    # `7z` but also a direct path to the `7z.exe` bundled with Julia on
    # windows, so we create generator functions to spit back functors to invoke
    # the correct 7z given the path to the executable:
    unpack_7z = (exe7z) -> begin
        return (tarball_path, out_path) ->
            pipeline(`$exe7z x $(tarball_path) -y -so`,
                     `$exe7z x -si -y -ttar -o$(out_path)`)
    end
    package_7z = (exe7z) -> begin
        return (in_path, tarball_path) ->
            pipeline(`$exe7z a -ttar -so a.tar "$(joinpath(".",in_path,"*"))"`,
                     `$exe7z a -si $(tarball_path)`)
    end
    list_7z = (exe7z) -> begin
        return (path) ->
            pipeline(`$exe7z x $path -so`, `$exe7z l -ttar -y -si`)
    end

    # Tar is rather less verbose, and we don't need to search multiple places
    # for it, so just rely on PATH to have `tar` available for us:

    # compression_engines is a list of (test_cmd, unpack_opts_functor,
    # package_opts_functor, list_opts_functor, parse_functor).  The probulator
    # will check each of them by attempting to run `$test_cmd`, and if that
    # works, will set the global compression functions appropriately.
    gen_7z = (p) -> (unpack_7z(p), package_7z(p), list_7z(p), parse_7z_list)
    compression_engines = Tuple[]

    for tar_cmd in [`tar`, `busybox tar`]
        # Some tar's aren't smart enough to auto-guess decompression method. :(
        unpack_tar = (tarball_path, out_path) -> begin
            Jjz = "z"
            if endswith(tarball_path, ".xz")
                Jjz = "J"
            elseif endswith(tarball_path, ".bz2")
                Jjz = "j"
            end
            return `$tar_cmd -x$(Jjz)f $(tarball_path) --directory=$(out_path)`
        end
        package_tar = (in_path, tarball_path) ->
            `$tar_cmd -czvf $tarball_path -C $(in_path) .`
        list_tar = (in_path) -> `$tar_cmd -tzf $in_path`
        push!(compression_engines, (
            `$tar_cmd --help`,
            unpack_tar,
            package_tar,
            list_tar,
            parse_tar_list,
        ))
    end

    # sh_engines is just a list of Cmds-as-paths
    sh_engines = [
        `sh`,
    ]

    # For windows, we need to tweak a few things, as the tools available differ
    @static if Sys.iswindows()
        # For download engines, we will most likely want to use powershell.
        # Let's generate a functor to return the necessary powershell magics
        # to download a file, given a path to the powershell executable
        psh_download = (psh_path) -> begin
            return (url, path) -> begin
                webclient_code = """
                [System.Net.ServicePointManager]::SecurityProtocol =
                    [System.Net.SecurityProtocolType]::Tls12;
                \$webclient = (New-Object System.Net.Webclient);
                \$webclient.Headers.Add("user-agent", "$agent");
				\$webclient.proxy = New-Object System.Net.WebProxy("www-int2.inet.dkfz-heidelberg.de:3128");
                \$webclient.DownloadFile("$url", "$path")
                """
                replace(webclient_code, "\n" => " ")
                return `$psh_path -NoProfile -Command "$webclient_code"`
            end
        end

        # We want to search both the `PATH`, and the direct path for powershell
        psh_path = joinpath(get(ENV, "SYSTEMROOT", "C:\\Windows"), "System32\\WindowsPowerShell\\v1.0\\powershell.exe")
        prepend!(download_engines, [
            (`$psh_path -Command ""`, psh_download(psh_path))
        ])
        prepend!(download_engines, [
            (`powershell -Command ""`, psh_download(`powershell`))
        ])

        # We greatly prefer `7z` as a compression engine on Windows
        prepend!(compression_engines, [(`7z --help`, gen_7z("7z")...)])

        # On windows, we bundle 7z with Julia, so try invoking that directly
        exe7z = joinpath(Sys.BINDIR, "7z.exe")
        prepend!(compression_engines, [(`$exe7z --help`, gen_7z(exe7z)...)])

        # And finally, we want to look for sh as busybox as well:
        busybox = joinpath(Sys.BINDIR, "busybox.exe")
        prepend!(sh_engines, [(`$busybox sh`)])
    end

    # Allow environment override
    if haskey(ENV, "BINARYPROVIDER_DOWNLOAD_ENGINE")
        engine = ENV["BINARYPROVIDER_DOWNLOAD_ENGINE"]
        es = split(engine)
        dl_ngs = filter(e -> e[1].exec[1:length(es)] == es, download_engines)
        if isempty(dl_ngs)
            all_ngs = join([d[1].exec[1] for d in download_engines], ", ")
            warn_msg  = "Ignoring BINARYPROVIDER_DOWNLOAD_ENGINE as its value "
            warn_msg *= "of `$(engine)` doesn't match any known valid engines."
            warn_msg *= " Try one of `$(all_ngs)`."
            @warn(warn_msg)
        else
            # If BINARYPROVIDER_DOWNLOAD_ENGINE matches one of our download engines,
            # then restrict ourselves to looking only at that engine
            download_engines = dl_ngs
        end
    end

    if haskey(ENV, "BINARYPROVIDER_COMPRESSION_ENGINE")
        engine = ENV["BINARYPROVIDER_COMPRESSION_ENGINE"]
        es = split(engine)
        comp_ngs = filter(e -> e[1].exec[1:length(es)] == es, compression_engines)
        if isempty(comp_ngs)
            all_ngs = join([c[1].exec[1] for c in compression_engines], ", ")
            warn_msg  = "Ignoring BINARYPROVIDER_COMPRESSION_ENGINE as its "
            warn_msg *= "value of `$(engine)` doesn't match any known valid "
            warn_msg *= "engines. Try one of `$(all_ngs)`."
            @warn(warn_msg)
        else
            # If BINARYPROVIDER_COMPRESSION_ENGINE matches one of our download
            # engines, then restrict ourselves to looking only at that engine
            compression_engines = comp_ngs
        end
    end

    download_found = false
    compression_found = false
    sh_found = false

    if verbose
        @info("Probing for download engine...")
    end

    local _gen_download_cmd
    # Search for a download engine
    for (test, dl_func) in download_engines
        if probe_cmd(`$test`; verbose=verbose)
            # Set our download command generator
            _gen_download_cmd = (url, path) -> dl_func(url, path)
            download_found = true

            if verbose
                @info("Found download engine $(test.exec[1])")
            end
            break
        end
    end

    if verbose
        @info("Probing for compression engine...")
    end

    # Search for a compression engine
    for (test, unpack, package, list, parse) in compression_engines
        if probe_cmd(`$test`; verbose=verbose)
            # Set our compression command generators
            gen_unpack_cmd = unpack
            gen_package_cmd = package
            gen_list_tarball_cmd = list
            parse_tarball_listing = parse

            if verbose
                @info("Found compression engine $(test.exec[1])")
            end

            compression_found = true
            break
        end
    end

    if verbose
        @info("Probing for sh engine...")
    end

    for path in sh_engines
        if probe_cmd(`$path --help`; verbose=verbose)
            gen_sh_cmd = (cmd) -> `$path -c $cmd`
            if verbose
                @info("Found sh engine $(path.exec[1])")
            end
            sh_found = true
            break
        end
    end


    # Build informative error messages in case things go sideways
    errmsg = ""
    if !download_found
        errmsg *= "No download engines found. We looked for: "
        errmsg *= join([d[1].exec[1] for d in download_engines], ", ")
        errmsg *= ". Install one and ensure it  is available on the path.\n"
    end

    if !compression_found
        errmsg *= "No compression engines found. We looked for: "
        errmsg *= join([c[1].exec[1] for c in compression_engines], ", ")
        errmsg *= ". Install one and ensure it is available on the path.\n"
    end

    if !sh_found && verbose
        @warn("No sh engines found.  Test suite will fail.")
    end

    # Error out if we couldn't find something
    if !download_found || !compression_found
        error(errmsg)
    end

    gen_download_cmd = (url, path) -> JuliaTeam.redirect_download_engine(_gen_download_cmd)
    return
end

For windows a powershell command is used to download anything.

There is some overwrite:

if haskey(ENV, "BINARYPROVIDER_DOWNLOAD_ENGINE")

and I tried this using a separate and new curl installation, but failed with unknown reason. I didn’t want to solve a curl problem now, when the real problem is Julia and powershell and i want to force powershell to see and use the proxy settings, which finally showed to be possible without changing the Julia stdlib code, which, on the other hand, seems to be quite difficult in JuliaPro. Changing any sources e.g. in C:\Users\oheil\AppData\Local\JuliaPro-1.2.0-1\Julia-1.2.0\share\julia\stdlib doesn’t show any effect and I wasn’t able to find out why.

Solution

In your windows home folder edit or create a file .gitconfig and add the following lines:

[http]
	proxy = http://YOUR-PROXY:3128
[https]
	proxy = http://YOUR-PROXY:3128
[url "https://"]
	insteadOf = git://

Instead of 3128 enter your proxy network port.

If you have installed git you can use the following commands inside the REPL to do this, e.g.:

run(`git config --global http.proxy `http://YOUR-PROXY:3128`)
run(`git config --global https.proxy `http://YOUR-PROXY:3128`)
run(`git config --global url."https://".insteadOf git://`)

For the powershell (and especially the .NET System.Net.WebClient class) to be aware of a proxy we need a powershell profile, where we put the relevant options. With that any new powershell receives the proxy settings.

If you already have a powershell profile you probably know about this. Skip the creation part which follows now.

Open a powershell and enter:

New-Item -path $profile -type file –force

this creates a file and prints the name and the location of the file, e.g.:

PS C:\WINDOWS\System32> New-Item -path $profile -type file -force


    Verzeichnis: C:\Users\oheil\Documents\WindowsPowerShell


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----       07.11.2019     15:26              0 Microsoft.PowerShell_profile.ps1

Now edit the file Microsoft.PowerShell_profile.ps1 and enter:

$MyProxy='http://YOUR-PROXY-HOST:3128'

$proxy = New-Object System.Net.WebProxy $MyProxy

$Wcl = new-object System.Net.WebClient
$Wcl.proxy = $proxy
$Wcl.Proxy.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials

Of course you need your proxy host and change port 3128 to your specific port.
Save the file.

If you have to use special credentials for the proxy you have to do the following:
Create a XML file e.g. proxy_cred.xml with the following command:

Get-Credential -Credential "<your-domain>\<your-username>" | Export-Clixml proxy_cred.xml

You will be prompted for your password.
The above powershell profile is than:

$MyProxy='http://YOUR-PROXY-HOST:3128'
$MyProxyCred = Import-Clixml -Path "<full path to proxy_cred.xml>"

$proxy = New-Object System.Net.WebProxy $MyProxy

$Wcl = new-object System.Net.WebClient
$Wcl.proxy = $proxy
$Wcl.Proxy.Credentials = $MyProxyCred

Now open a new powershell and see if you get error messages not being allowed to run scripts like it is for me, but it works for me with local admin rights.

I didn’t had to create the profile again als local administrator (which is another user for me), it just worked with the profile created as normal user (this is somehow strange).

Now running JuliaPro as local Administrator and installing Plots results in:

(v1.2) pkg> add Plots
   Cloning default registries into `C:\Users\admv310\.juliapro\JuliaPro_v1.2.0-1`
   Cloning registry from "https://pkg.juliacomputing.com/registry/General"
ERROR: Please retrieve a token from "https://pkg.juliacomputing.com/" and store it at "C:\Users\admv310\.juliapro\token.toml"
Stacktrace:
 [1] error(::String) at .\error.jl:33

which looks much better and where I stop now, as getting a token is something else and not related to proxy servers (I hope).

If you do not get any errors about running scripts in powershell is not allowed than you do not need local Admin rights.

I would say, this whole process is a mess and we Windows Julians urgently need some more general and more reliable download mechanism in the REPL.

2 Likes

I don’t know if it’s smart, but I copy pasted the julia packages in my users/.juliapro folder to my work pc and the packages work. Still tedious though—I love using the package manager at home on my pc.

By the way, thanks a lot for going through the bother and testing my problem on your work pc. Really appreciate it.

2 Likes

Sure, this is the easiest workaround but will get quite problematic after a while of working with it. Updates of packages, complex dependencies and so on are likely to make some problems in the future.

The proxy-topic is something which comes regularely in to discussion and typically I can reproduce the issues because I use Julia on Windows and in Linux, at Home and at Work with proxy and without, so I am one of those capable to look into it and find work arounds. Thats why I do it. As a side effect I learn a lot of things about Julia and Windows (like I never knew about powershell profiles) which is also a valuable benefit for me.

I think I will open a new topic why it is not possible (or difficult) to play around with the JuliaPro base and stdlib source code :slight_smile:

2 Likes

Yeah, back to square one for me :worried: something seems to be broken and I can’t download the package again. No juliapro until I get home and redo the thing again.

I would be happy if you could try and confirm that the powershell profile thing is working for you too! :wink:

I just tried it again. It won’t work for me. Something is up though, I can’t replicate julia.exe working by itself to download packages (without Juliapro in the picture) Urgh :frowning: