# Pkg: Proxy Authentication Required (Received HTTP code 407 from proxy after CONNECT)

**URL:** https://discourse.julialang.org/t/pkg-proxy-authentication-required-received-http-code-407-from-proxy-after-connect/79984
**Category:** General Usage
**Tags:** proxy, error, package-manager
**Created:** [April 25, 2022, 12:28pm UTC](https://discourse.julialang.org/t/pkg-proxy-authentication-required-received-http-code-407-from-proxy-after-connect/79984 "2022-04-25T12:28:36Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![Keluaa](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/keluaa/32/50675_2.png) [@Keluaa](https://discourse.julialang.org/u/Keluaa)
#### Post date: [April 25, 2022, 12:28pm UTC](https://discourse.julialang.org/t/pkg-proxy-authentication-required-received-http-code-407-from-proxy-after-connect/79984/1 "2022-04-25T12:28:36Z")

</div>

Hello,

I am currently working with Julia 1.7.2 on a linux machine through ssh which requires a proxy to access the internet.

By configuring the environment variables “http\_proxy” and “https\_proxy” to `http://usr:@some_address:port/` (there is no password), I was able to clone repositories with git in the command line and make requests using wget or curl.

However I was not able to install any Julia packages because of this error:

```julia
(v1.7) pkg> add JSON
    Updating registry at `~/.julia/registries/General.toml`
┌ Warning: could not download https://pkg.julialang.org/registries
│ exception = HTTP/1.1 407 Proxy Authentication Required (Received HTTP code 407 from proxy after CONNECT) while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.7/Pkg/src/Registry/Registry.jl:82
   Resolving package versions...
     Cloning [682c06a0-de6a-54ab-a142-c8b1cf79cde6] JSON from https://github.com/JuliaIO/JSON.jl.git
ERROR: failed to clone from https://github.com/JuliaIO/JSON.jl.git, error: GitError(Code:ERROR, Class:HTTP, proxy authentication required but no callback set)

```

After some research I found this topic: [Can't install packages on corporate pc · Issue #2516 · JuliaLang/Pkg.jl · GitHub](https://github.com/JuliaLang/Pkg.jl/issues/2516)  
Setting `JULIA_PKG_USE_CLI_GIT` to `true` circumvents the issue, but I am still stuck with this problem: “Proxy Authentication Required (Received HTTP code 407 from proxy after CONNECT)” which occurs when trying to update the package registry.

On top of that I am quite sure that this issue is also responsible for this error :

```julia
(v1.7) pkg> add CUDA
...
  Downloaded artifact: LLVMExtra
ERROR: Unable to automatically install 'LLVMExtra' from '/home/.julia/packages/LLVMExtra_jll/y3M2E/Artifacts.toml'
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] ensure_artifact_installed(name::String, meta::Dict{String, Any}, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, verbose::Bool, quiet_download::Bool, io::Base.TTY)
    @ Pkg.Artifacts /home/julia-1.7.2/share/julia/stdlib/v1.7/Pkg/src/Artifacts.jl:441
  [3] download_artifacts(env::Pkg.Types.EnvCache; platform::Base.BinaryPlatforms.Platform, julia_version::VersionNumber, verbose::Bool, io::Base.TTY)
    @ Pkg.Operations /home/julia-1.7.2/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:617
  [4] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /home/julia-1.7.2/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:1182
...

```

I cannot debug the installation of the artifact with “DebugArtifacts” since it is not in the registry currently on the machine (which I copied from another machine with proper internet access):

```julia
julia> Pkg.add("DebugArtifacts")
ERROR: The following package names could not be resolved:
 * DebugArtifacts (not found in project, manifest or registry)

```

I am aware that this is most likely a proxy configuration issue, but I do not know which things to configure in order for Julia to work properly.

---

<div class="post-metadata">

### Author: ![Keluaa](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/keluaa/32/50675_2.png) [@Keluaa](https://discourse.julialang.org/u/Keluaa)
#### Post date: [April 28, 2022, 7:34am UTC](https://discourse.julialang.org/t/pkg-proxy-authentication-required-received-http-code-407-from-proxy-after-connect/79984/2 "2022-04-28T07:34:57Z")

</div>

By further exploring the issue, I tracked down the problem to the fact that Julia only uses the Basic authentication type, while my proxy uses Negotiate ([see the docs](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#authentication_schemes)).

Passing `verbose=true` as an option to Downloads.jl, we can see that I correctly passes a valid header for Basic authentication:

```julia
julia> Downloads.download("https://pkg.julialang.org/registries"; verbose=true)
* Uses proxy env variable no_proxy == 'localhost,127.0.0.0/8,::1'
* Uses proxy env variable https_proxy == 'http://usr:@some_address:port/'
* Couldn't find host pkg.julialang.org in the .netrc file; using defaults
* Trying ***.***. **.*** :port...
* Connected to some_address ( ***.***. **.*** ) port (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to pkg.julialang.org:443
* Proxy auth using Basic with user 'user'
> CONNECT pkg.julialang.org:443 HTTP/1.1
Host: pkg.julialang.org:443
Proxy-Authorization: Basic XXXXX
User-Agent: curl/7.73.0 julia/1.7
Proxy-Connection: Keep-Alive

< HTTP/1.1 407 Proxy Authentication Required
< Server: squid
< Mime-Version: 1.0
< Date: Wed, 27 Apr 2022 14:45:42 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 3642
< X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
< Vary: Accept-Language
< Content-Language: en
< Proxy-Authenticate: Negotiate
< X-Cache: MISS from some_address
< X-Cache-Lookup: NONE from some_address:port
< Via: 1.1 some_address (squid)
< Connection: keep-alive
< 
* Ignore 3642 bytes of response-body
* Received HTTP code 407 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
ERROR: HTTP/1.1 407 Proxy Authentication Required (Received HTTP code 407 from proxy after CONNECT) while requesting https://pkg.julialang.org/registries

```

But the server expects another type of authentication `Proxy-Authenticate: Negotiate`, so the connection is refused.

However when doing the same request with curl in the terminal, it correctly loads my ~/.curlrc config file which specifies to use Negotiate:

```julia
proxy = some_proxy:port
--proxy-negotiate
--proxy-user "user:"
-k

```

Therefore curl is able to perform the request:

```julia
$ curl -v https://pkg.julialang.org/registries
* About to connect() to proxy some_proxy port 'port' (#0)
* Trying ***.***. **.***...
* Connected to some_proxy ( ***.***. **.*** ) port 'port' (#0)
* Establish HTTP proxy tunnel to pkg.julialang.org:443
> CONNECT pkg.julialang.org:443 HTTP/1.1
> Host: pkg.julialang.org:443
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 407 Proxy Authentication Required
< Server: squid
< Mime-Version: 1.0
< Date: Thu, 28 Apr 2022 07:27:14 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 3577
< X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
< Vary: Accept-Language
< Content-Language: en
< Proxy-Authenticate: Negotiate
< X-Cache: MISS from some_address
< X-Cache-Lookup: NONE from some_address:port
< Via: 1.1 some_address (squid)
< Connection: keep-alive
< 
* Ignore 3577 bytes of response-body
* TUNNEL_STATE switched to: 0
* Establish HTTP proxy tunnel to pkg.julialang.org:443
* Proxy auth using GSS-Negotiate with user 'user'
> CONNECT pkg.julialang.org:443 HTTP/1.1
> Host: pkg.julialang.org:443
> Proxy-Authorization: Negotiate auth_dataXXXXXXXXXXXXX
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
...

```

Then, my problem becomes: how to configure Julia to use those options in the ‘~/.curlrc’ file?  
I know that libcurl doesn’t read this file, so then how can I pass those parameters to Julia?  
Does Julia (or Downloads.jl) even support authentication types other than Basic?

---

<div class="post-metadata">

### Author: ![Keluaa](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/keluaa/32/50675_2.png) [@Keluaa](https://discourse.julialang.org/u/Keluaa)
#### Post date: [April 29, 2022, 2:32pm UTC](https://discourse.julialang.org/t/pkg-proxy-authentication-required-received-http-code-407-from-proxy-after-connect/79984/3 "2022-04-29T14:32:33Z")

</div>

From what I could gather from reading the code of Pkg.jl and Download, it seems that there is no support for proxy authentication at all when downloading files.

Therefore I tried overriding `Downloads.download` function, which is used by the package manager for all (I think?) downloads.

Here is my solution, I hope that it could be useful to someone in the future:

```julia
using Downloads
import Downloads: ArgWrite, arg_write, Downloader, request, Response, status_ok, RequestError

function curl_download(
        url :: AbstractString,
        output :: Union{ArgWrite, Nothing} = nothing;
        headers :: Union{AbstractVector, AbstractDict} = Pair{String, String}[],
        progress :: Union{Function, Nothing} = nothing,
        verbose :: Bool = false)

    options = ["-L"] # Follow redirections

    !verbose && push!(options, "-s", "-S") # silent mode, but print errors if any
    verbose && push!(options, "-v") # Verbose mode

    # Pass the headers to curl
    for (header, content) in headers
        push!(options, "-H", header * ":" * content)
    end

    # Simple implementation: only file names are supported
    if !isa(output, AbstractString)
        error("Cannot handle this output file type: ", typeof(output), " as ", output)
    end

    push!(options, "-o", output)
    curl_cmd = `curl $options $url`

    if verbose
        println("Curl cmd: ", curl_cmd)
    end

    # Run the command, if it fails, print it and rethrow the error
    try
        run(curl_cmd)
    catch
        !verbose && println("Curl cmd failed: ", curl_cmd)
        rethrow()
    end

    return output
end

function Downloads.download(
    url :: AbstractString,
    output :: Union{ArgWrite, Nothing} = nothing;
    method :: Union{AbstractString, Nothing} = nothing,
    headers :: Union{AbstractVector, AbstractDict} = Pair{String,String}[],
    timeout :: Real = Inf,
    progress :: Union{Function, Nothing} = nothing,
    verbose :: Bool = false,
    downloader :: Union{Downloader, Nothing} = nothing,
) :: ArgWrite
    return curl_download(url, output; headers, progress, verbose)
end

```

Here we just make a call to the curl via the command line and let it read the ~/.curlrc config file for proper proxy configuration.

Together with `JULIA_PKG_USE_CLI_GIT=true`, I could download and install CUDA.jl, AMDGPU.jl, LoopVectorization.jl… without any problems.