Julia 1.6 libcurl firewall download issue: Windows Schannel certificate revocation check failure

Yes sir, i did. I installed Plots.jl (a lot of artifacts), closed Julia, restarted it, resetted the environnement variable and now I have the same hanging problem.

What did you do differently the second time?

1 Like

Well, I cant remember :wink:

Now the situation is the following:

  • Base.download works if the wanted is http://, hangs if https
  • Artifacts downloads hangs as well
  • even debug_artifacts hangs without erroring out.
  • but of course run(curl https://google.com) works…

Setting BINARYPROVIDER_DOWNLOAD_ENGINE with or without the proxy parameters before launching Julia, and setting ENV["JULIA_SSL_NO_VERIFY_HOSTS"] = "github.com" after launching Julia or not does not change anything anymore.

After half hour of testing every potential combination of what I did, the following is necessary and sufficient for the thing to work:

Check that run(curl ...) works. Then in Julia:

ENV["http_proxy"] = "http://<your_proxy>:<port>"
ENV["https_proy"] = "http://<your_proxy>:<port>"
ENV["JULIA_SSL_NO_VERIFY_HOSTS"] = "github.com"

And now it works correctly.

These commands might go to .julia/config/startup.jl for convenience.

7 Likes

Thanks to everyone in this thread! This solution worked for me.
I also have Windows and when trying to install the Plots package after downloading v 1.6 I received the error:
ERROR: Unable to automatically install ‘GLFW’

Simply running in julia REPL
ENV[“JULIA_SSL_NO_VERIFY_HOSTS”] = “github.com

One more thing that was necessary for me:

I had my .curlrc in my home folder, and not in the right %APPDATA% folder, which had no effects on the windows curl that was called from julia.

Setting the proxy parmaeters and the insecure option in the right file made it work perfectly.

Saying that setting the insecure option makes it work perfectly sounds like an oxymoron though :wink:

Indeed, but it is not my problem anymore : this is now the problem of my adminsys that did not provide a proxy that respects the standards, nor tried to do something when I told him :wink:

1 Like

If you are behind a MITM proxy, it is not actually insecure to turn host verification off since the certificate you’d be verifying is fake anyway. If the proxy is not verifying the server’s identity for you by checking the actual certificate, that would be insecure but presumably the proxy is configured securely. Whether that is the case or not, the client has no control over that anyway, so there’s nothing we can do about it.

There are two things required for a client connection behind a MITM proxy to “just work”:

  1. The client must have a CA root installed that allows it to verify fake certificates created by the MITM proxy. That seems to be the case here, since otherwise we’d get a different error.

  2. The proxy should be configured to handle certificate revocation checks from Windows machines. That seems not to be the case here, as the error indicates that revocation checking failed.

The first requirement is necessary for all operating systems. As of Julia 1.6, we use system TLS engines on macOS and Windows, so if the MITM CA root has been added to the system certificate stores, then that step should be fine. On Linux, we look for a PEM file in common places and use the first one we find, so even though there’s no “system TLS engine” we ought to pick up a MITM CA root if one has been installed.

The second requirement only affects Windows because each OS does certificate revocation checking differently. Linux doesn’t do certificate revocation checks at all, which is obviously insecure, but also not our problem to solve. If there’s ever a standard way to handle certificate revocation checks on Linux, we can hook into it. MacOS does offline updates to its certificate revocation list, so this kind of error cannot happen: the system TLS engine checks a certificate against the revocation list that it already has; it doesn’t try to update that list during the host verification process. It may try (and fail) to update the CRL list at some other time, but that doesn’t block individual TLS requests.

Windows, on the other hand, does a synchronous certificate revocation checks while verifying each host’s identity. That means that if it hasn’t recently checked whether the certificate for a given host has been revoked, it will contact a Microsoft server to check that while it is in the process of verifying the validity of the certificate for the host you are trying to connect to. This is where people are hitting problems: they seem to have a MITM CA root installed, but when Windows tries to see if the certificate has been revoked, that check is being blocked or failing.

We could add an option to not do CRL checks while still doing certificate verification, but I’m not sure if this is actually necessary or useful. This problem only seems to occur when behind a MITM firewall, in which case it’s just as secure to skip verification altogether since the certificate you’d be verifying isn’t real anyway. You might as well turn host verification off altogether when you’re behind a MITM firewall.

It’s possible that I’m misunderstanding something here because this stuff is complicated and documentation is both poor and spread all across the Internet.

3 Likes

This post gives details on how Windows does CRL checks and how to configure things so that they will work correctly: Troubleshooting network retrieval of CRLs - Browsers | Microsoft Docs.

2 Likes

Well, unless someone inside your MITM firewall is doing yet another MITM attack with a certificate that was not installed on your machine.

2 Likes

Sure, but that’s also out of our control.

Right, I’m saying that’s a case where verifying certificates (presumably installed by your snooping IT department) without verifying CRLs makes sense (because it still blocks MITM inside the firewall)

2 Likes

Oh right, I see what you mean. Someone could be on the local network doing a MITM attack.

2 Likes

9 posts were split to a new topic: Julia 1.6 other libcurl download issues

In an effort to keep this discussion about one problem, I have moved the posts that are about unrelated download problems to a different topic. Let’s keep this discussion specifically about the Windows Schannel certificate revocation check failure that was originally posted. There does seem to be a libcurl option called CURLSSLOPT_REVOKE_BEST_EFFORT that we could set for this (see CURLOPT_SSL_OPTIONS). According to the docs:

Tells libcurl to ignore certificate revocation checks in case of missing or offline distribution points for those SSL backends where such behavior is present. This option is only supported for Schannel (the native Windows SSL library).

So basically specifically for this problem. Setting this by default seems like the best option to me: it will work and check for revoked certificate to the extent possible but not fail on Windows if the CRL server cannot be reached. This is not perfect since an attacker could theoretically block your access to the CRL server and use a very recently revoked certificate, but that seems pretty improbable. Even with this option set, the behavior on Windows is as secure as the default macOS behavior and both are more secure than Linux where certificates are never checked for revocation at all.

Pull request to set this option: set TLS option CURLSSLOPT_REVOKE_BEST_EFFORT by StefanKarpinski · Pull Request #115 · JuliaLang/Downloads.jl · GitHub.

It would be great if someone who is seeing this problem could test this. It’s a little tricky to test, but here are some relatively simple steps that don’t require recompiling Julia:

  1. Use pkg> mode to checkout a dev copy of Downloads.jl:
    pkg> dev Downloads
    
  2. Go into ~/.julia/dev/Downloads and checkout the PR branch:
    git fetch origin
    git checkout -b sk/tls-revoke-best-effort origin/sk/tls-revoke-best-effort
    
  3. Edit the ~/.julia/dev/Downloads/Project.toml file and modify the UUID somehow, e.g. by changing the last digit to some other valid hex digit.
  4. Change directory into the ~/.julia/dev/Downloads
  5. Start Julia 1.6 in that directory with the --project option
  6. Try the problematic download from the Julia REPL:
    using Downloads: download
    url = "https://github.com/JuliaBinaryWrappers/GLFW_jll.jl/releases/download/GLFW-v3.3.3%2B0/GLFW.v3.3.3.x86_64-linux-gnu.tar.gz"
    file = download(url)
    @assert open(Base._crc32c, file) == 0x40e5ef54
    

If that works behind a firewall without error, downloading the file and getting the right CRC checksum, then the problem is fixed by setting this option.

Anyone?

Have a Windows 10 laptop. but am not behind a firewall, sorry.