Julia 1.6 libcurl firewall download issue: Windows Schannel certificate revocation check failure

julia> using Pkg

julia> mktempdir() do dir
           url = "https://github.com/JuliaBinaryWrappers/GLFW_jll.jl/releases/download/GLFW-v3.3.3%2B0/GLFW.v3.3.3.x86_64-linux-gnu.tar.gz"
               Pkg.PlatformEngines.download(url, joinpath(dir, "test.tar.gz"); verbose=true)
       end
ERROR: HTTP/1.1 200 Connection established (schannel: next InitializeSecurityContext failed: Unknown error (0x80092012) - The revocation function was unable to check revocation for the certificate.) while requesting https://github.com/JuliaBinaryWrappers/GLFW_jll.jl/releases/download/GLFW-v3.3.3%2B0/GLFW.v3.3.3.x86_64-linux-gnu.tar.gz
Stacktrace:
  [1] (::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Downloads.var"#24#27"{Pkg.PlatformEngines.var"#13#15"{Base.TTY}}, Bool, Bool, String, Int64, Bool, Bool})(easy::Downloads.Curl.Easy)
    @ Downloads C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Downloads\src\Downloads.jl:356
  [2] with_handle(f::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Downloads.var"#24#27"{Pkg.PlatformEngines.var"#13#15"{Base.TTY}}, Bool, Bool, String, Int64, Bool, Bool}, handle::Downloads.Curl.Easy)
    @ Downloads.Curl C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Downloads\src\Curl\Curl.jl:60
  [3] #8
    @ C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Downloads\src\Downloads.jl:298 [inlined]
  [4] arg_write(f::Downloads.var"#8#17"{Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Downloads.var"#24#27"{Pkg.PlatformEngines.var"#13#15"{Base.TTY}}, Bool, Bool, String, Int64, Bool, Bool}, arg::IOStream)
    @ ArgTools C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\ArgTools\src\ArgTools.jl:112
  [5] #7
    @ C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Downloads\src\Downloads.jl:297 [inlined]
  [6] arg_read(f::Downloads.var"#7#16"{IOStream, Nothing, Vector{Pair{String, String}}, Float64, Downloads.var"#24#27"{Pkg.PlatformEngines.var"#13#15"{Base.TTY}}, Bool, Bool, String, Int64, Bool, Bool}, arg::Base.DevNull)
    @ ArgTools C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\ArgTools\src\ArgTools.jl:61
  [7] request(url::String; input::Nothing, output::IOStream, method::Nothing, headers::Vector{Pair{String, String}}, timeout::Float64, progress::Pkg.PlatformEngines.var"#13#15"{Base.TTY}, verbose::Bool, throw::Bool, downloader::Nothing)
    @ Downloads C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Downloads\src\Downloads.jl:296
  [8] (::Downloads.var"#3#4"{Nothing, Vector{Pair{String, String}}, Float64, Pkg.PlatformEngines.var"#13#15"{Base.TTY}, Bool, Nothing, String})(output::IOStream)
    @ Downloads C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Downloads\src\Downloads.jl:209
  [9] open(f::Downloads.var"#3#4"{Nothing, Vector{Pair{String, String}}, Float64, Pkg.PlatformEngines.var"#13#15"{Base.TTY}, Bool, Nothing, String}, 
args::String; kwargs::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:write,), Tuple{Bool}}})
    @ Base .\io.jl:330
 [10] arg_write(f::Function, arg::String)
    @ ArgTools C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\ArgTools\src\ArgTools.jl:86
 [11] #download#2
    @ C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Downloads\src\Downloads.jl:208 [inlined]
 [12] download(url::String, dest::String; verbose::Bool, headers::Vector{Pair{String, String}}, auth_header::Nothing)
    @ Pkg.PlatformEngines C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\PlatformEngines.jl:270
 [13] #7
    @ .\REPL[68]:3 [inlined]
 [14] mktempdir(fn::var"#7#8", parent::String; prefix::String)
    @ Base.Filesystem .\file.jl:729
 [15] mktempdir(fn::Function, parent::String) (repeats 2 times)
    @ Base.Filesystem .\file.jl:727
 [16] top-level scope
    @ REPL[68]:1

julia> 

Setting that environment variable and running your test looks like it works.

mktempdir() do dir
           url = "https://github.com/JuliaBinaryWrappers/GLFW_jll.jl/releases/download/GLFW-v3.3.3%2B0/GLFW.v3.3.3.x86_64-linux-gnu.tar.gz"
               Pkg.PlatformEngines.download(url, joinpath(dir, "test.tar.gz"); verbose=true)
       end
"C:\\Users\\bakerar\\AppData\\Local\\Temp\\1\\jl_GoDV4l\\test.tar.gz"

Setting that environment variable looks like it works for adding Plots to the package manager. Now, as far as security is concerned, is that what I should be doing?

I don’t personally have the whole story here so I don’t want to do any strong recommendations. The configurations are documented in GitHub - JuliaLang/NetworkOptions.jl so you might get some more useful information there.

Thanks, that explains some stuff.

I am also wondering if I need the HTTP_PROXY stuff set anymore? Could that be interfering with the new Julia package manager?

Hey,

I just removed my 1.5.3 julia to install the new 1.6 on a window machine behind a proxy.

For some reasons Pkg cannot download artifacts anymore:

(@v1.6) pkg> add Distributions
   Updating registry at `C:\Users\u009192\.julia\registries\General`
   Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Resolving package versions...
 Downloading artifact: Rmath
    Downloading [>                                        ]  0.0 %  

It hangs and never stop.

I setted up my .curlrc so that curl works correctly under my proxy:

julia> run(`curl https://google.com`)
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
Process(`curl https://google.com`, ProcessExited(0))

julia>

Standard git operations also works correctly, since the General registry can be removed, reinstalled, updated, etc… I can install artifact-less packages without any issue.

Thinking Pkg was using Powershell to download the artifacts, I tried this solution but it did not work, it still hangs on artifacts.

Base.download("https://google.com") has the same hanging issue. By looking around, upon forcing some environment variables that matches my .curlrc before launching julia by:

export BINARYPROVIDER_DOWNLOAD_ENGINE="curl --proxy <my_proxy> --ssl-no-revoke --insecure"

I am able to make Base.download work, but not for https:

julia> Base.download("google.com")
"C:\\Users\\XXX\\AppData\\Local\\Temp\\jl_D511.tmp"

julia> Base.download("https://google.com")
ERROR: HTTP/1.0 200 Connection established (schannel: next InitializeSecurityContext failed: Unknown error (0x80092012)) while requesting https://google.com
Stacktrace:
 [Very long stacktrace, the same one.]
julia> 

Moreover, I tried DebugArtifacts and the same thing occurs:

julia> using DebugArtifacts

julia> debug_artifact("OpenSpecFun")
[ Info: Platform: Windows x86_64 {cxxstring_abi=cxx11, julia_version=1.6.0, libgfortran_version=5.0.0}
Julia Version 1.6.0
Commit f9720dc2eb (2021-03-24 12:55 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)

[ Info: Downloading Artifacts.toml to C:\Users\u009192\AppData\Local\Temp\jl_GfvfmC\Artifacts.toml...
ERROR: HTTP/1.0 200 Connection established (schannel: next InitializeSecurityContext failed: Unknown error (0x80092012)) while requesting https://raw.githubusercontent.com/JuliaBinaryWrappers/OpenSpecFun_jll.jl/master/Artifacts.toml
Stacktrace:
 [Very long stacktrace, the same one]

But with a simple run of curl, everything is still OK.

julia> run(`curl https://google.com`)
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
Process(`curl https://google.com`, ProcessExited(0))

julia> 

How can i tell the libcurl that is inside julia to respond to my .curlrc ?
Can i pass directly some parameters (the same i passed in my .curlrc to Julia’s libcurl ? How ?

See also Julia 1.6 new Package Manager -- curl firewall problems

Looks like setting

ENV["JULIA_SSL_NO_VERIFY_HOSTS"] = "github.com"

On top of everyhting i already did worked. Now the question is if everything else was really needed…

Thanks anyway, you saved my day.

Edit in facts it did not. After restarting it still does not work.

1 Like

Merging threads here since these seem like the same problem. I also filed an issue to track progress: problems with SSL host verification with proxy servers · Issue #108 · JuliaLang/Downloads.jl · GitHub

Well this is still not solved for me, the workaround did not work in both cases. I have the same errors as before.

I’m confused, you said that it worked, then you restarted and it stopped working. Did you set the same environment variables again after you restarted?

Yes sir, i did. I installed Plots.jl (a lot of artifacts), closed Julia, restarted it, resetted the environnement variable and now I have the same hanging problem.

What did you do differently the second time?

1 Like

Well, I cant remember :wink:

Now the situation is the following:

  • Base.download works if the wanted is http://, hangs if https
  • Artifacts downloads hangs as well
  • even debug_artifacts hangs without erroring out.
  • but of course run(curl https://google.com) works…

Setting BINARYPROVIDER_DOWNLOAD_ENGINE with or without the proxy parameters before launching Julia, and setting ENV["JULIA_SSL_NO_VERIFY_HOSTS"] = "github.com" after launching Julia or not does not change anything anymore.

After half hour of testing every potential combination of what I did, the following is necessary and sufficient for the thing to work:

Check that run(curl ...) works. Then in Julia:

ENV["http_proxy"] = "http://<your_proxy>:<port>"
ENV["https_proy"] = "http://<your_proxy>:<port>"
ENV["JULIA_SSL_NO_VERIFY_HOSTS"] = "github.com"

And now it works correctly.

These commands might go to .julia/config/startup.jl for convenience.

7 Likes

Thanks to everyone in this thread! This solution worked for me.
I also have Windows and when trying to install the Plots package after downloading v 1.6 I received the error:
ERROR: Unable to automatically install ‘GLFW’

Simply running in julia REPL
ENV[“JULIA_SSL_NO_VERIFY_HOSTS”] = “github.com

One more thing that was necessary for me:

I had my .curlrc in my home folder, and not in the right %APPDATA% folder, which had no effects on the windows curl that was called from julia.

Setting the proxy parmaeters and the insecure option in the right file made it work perfectly.

Saying that setting the insecure option makes it work perfectly sounds like an oxymoron though :wink:

Indeed, but it is not my problem anymore : this is now the problem of my adminsys that did not provide a proxy that respects the standards, nor tried to do something when I told him :wink:

1 Like

If you are behind a MITM proxy, it is not actually insecure to turn host verification off since the certificate you’d be verifying is fake anyway. If the proxy is not verifying the server’s identity for you by checking the actual certificate, that would be insecure but presumably the proxy is configured securely. Whether that is the case or not, the client has no control over that anyway, so there’s nothing we can do about it.

There are two things required for a client connection behind a MITM proxy to “just work”:

  1. The client must have a CA root installed that allows it to verify fake certificates created by the MITM proxy. That seems to be the case here, since otherwise we’d get a different error.

  2. The proxy should be configured to handle certificate revocation checks from Windows machines. That seems not to be the case here, as the error indicates that revocation checking failed.

The first requirement is necessary for all operating systems. As of Julia 1.6, we use system TLS engines on macOS and Windows, so if the MITM CA root has been added to the system certificate stores, then that step should be fine. On Linux, we look for a PEM file in common places and use the first one we find, so even though there’s no “system TLS engine” we ought to pick up a MITM CA root if one has been installed.

The second requirement only affects Windows because each OS does certificate revocation checking differently. Linux doesn’t do certificate revocation checks at all, which is obviously insecure, but also not our problem to solve. If there’s ever a standard way to handle certificate revocation checks on Linux, we can hook into it. MacOS does offline updates to its certificate revocation list, so this kind of error cannot happen: the system TLS engine checks a certificate against the revocation list that it already has; it doesn’t try to update that list during the host verification process. It may try (and fail) to update the CRL list at some other time, but that doesn’t block individual TLS requests.

Windows, on the other hand, does a synchronous certificate revocation checks while verifying each host’s identity. That means that if it hasn’t recently checked whether the certificate for a given host has been revoked, it will contact a Microsoft server to check that while it is in the process of verifying the validity of the certificate for the host you are trying to connect to. This is where people are hitting problems: they seem to have a MITM CA root installed, but when Windows tries to see if the certificate has been revoked, that check is being blocked or failing.

We could add an option to not do CRL checks while still doing certificate verification, but I’m not sure if this is actually necessary or useful. This problem only seems to occur when behind a MITM firewall, in which case it’s just as secure to skip verification altogether since the certificate you’d be verifying isn’t real anyway. You might as well turn host verification off altogether when you’re behind a MITM firewall.

It’s possible that I’m misunderstanding something here because this stuff is complicated and documentation is both poor and spread all across the Internet.

3 Likes