Indeed, but it is not my problem anymore : this is now the problem of my adminsys that did not provide a proxy that respects the standards, nor tried to do something when I told him 
1 Like
If you are behind a MITM proxy, it is not actually insecure to turn host verification off since the certificate you’d be verifying is fake anyway. If the proxy is not verifying the server’s identity for you by checking the actual certificate, that would be insecure but presumably the proxy is configured securely. Whether that is the case or not, the client has no control over that anyway, so there’s nothing we can do about it.
There are two things required for a client connection behind a MITM proxy to “just work”:
-
The client must have a CA root installed that allows it to verify fake certificates created by the MITM proxy. That seems to be the case here, since otherwise we’d get a different error.
-
The proxy should be configured to handle certificate revocation checks from Windows machines. That seems not to be the case here, as the error indicates that revocation checking failed.
The first requirement is necessary for all operating systems. As of Julia 1.6, we use system TLS engines on macOS and Windows, so if the MITM CA root has been added to the system certificate stores, then that step should be fine. On Linux, we look for a PEM file in common places and use the first one we find, so even though there’s no “system TLS engine” we ought to pick up a MITM CA root if one has been installed.
The second requirement only affects Windows because each OS does certificate revocation checking differently. Linux doesn’t do certificate revocation checks at all, which is obviously insecure, but also not our problem to solve. If there’s ever a standard way to handle certificate revocation checks on Linux, we can hook into it. MacOS does offline updates to its certificate revocation list, so this kind of error cannot happen: the system TLS engine checks a certificate against the revocation list that it already has; it doesn’t try to update that list during the host verification process. It may try (and fail) to update the CRL list at some other time, but that doesn’t block individual TLS requests.
Windows, on the other hand, does a synchronous certificate revocation checks while verifying each host’s identity. That means that if it hasn’t recently checked whether the certificate for a given host has been revoked, it will contact a Microsoft server to check that while it is in the process of verifying the validity of the certificate for the host you are trying to connect to. This is where people are hitting problems: they seem to have a MITM CA root installed, but when Windows tries to see if the certificate has been revoked, that check is being blocked or failing.
We could add an option to not do CRL checks while still doing certificate verification, but I’m not sure if this is actually necessary or useful. This problem only seems to occur when behind a MITM firewall, in which case it’s just as secure to skip verification altogether since the certificate you’d be verifying isn’t real anyway. You might as well turn host verification off altogether when you’re behind a MITM firewall.
It’s possible that I’m misunderstanding something here because this stuff is complicated and documentation is both poor and spread all across the Internet.
3 Likes
This post gives details on how Windows does CRL checks and how to configure things so that they will work correctly: Troubleshooting network retrieval of CRLs - Browsers | Microsoft Docs.
2 Likes
Well, unless someone inside your MITM firewall is doing yet another MITM attack with a certificate that was not installed on your machine.
3 Likes
Sure, but that’s also out of our control.
Right, I’m saying that’s a case where verifying certificates (presumably installed by your snooping IT department) without verifying CRLs makes sense (because it still blocks MITM inside the firewall)
2 Likes
Oh right, I see what you mean. Someone could be on the local network doing a MITM attack.
2 Likes
9 posts were split to a new topic: Julia 1.6 other libcurl download issues
In an effort to keep this discussion about one problem, I have moved the posts that are about unrelated download problems to a different topic. Let’s keep this discussion specifically about the Windows Schannel certificate revocation check failure that was originally posted. There does seem to be a libcurl option called CURLSSLOPT_REVOKE_BEST_EFFORT that we could set for this (see CURLOPT_SSL_OPTIONS). According to the docs:
Tells libcurl to ignore certificate revocation checks in case of missing or offline distribution points for those SSL backends where such behavior is present. This option is only supported for Schannel (the native Windows SSL library).
So basically specifically for this problem. Setting this by default seems like the best option to me: it will work and check for revoked certificate to the extent possible but not fail on Windows if the CRL server cannot be reached. This is not perfect since an attacker could theoretically block your access to the CRL server and use a very recently revoked certificate, but that seems pretty improbable. Even with this option set, the behavior on Windows is as secure as the default macOS behavior and both are more secure than Linux where certificates are never checked for revocation at all.
Pull request to set this option: set TLS option CURLSSLOPT_REVOKE_BEST_EFFORT by StefanKarpinski · Pull Request #115 · JuliaLang/Downloads.jl · GitHub.
It would be great if someone who is seeing this problem could test this. It’s a little tricky to test, but here are some relatively simple steps that don’t require recompiling Julia:
- Use
pkg>mode to checkout a dev copy ofDownloads.jl:pkg> dev Downloads - Go into
~/.julia/dev/Downloadsand checkout the PR branch:git fetch origin git checkout -b sk/tls-revoke-best-effort origin/sk/tls-revoke-best-effort - Edit the
~/.julia/dev/Downloads/Project.tomlfile and modify the UUID somehow, e.g. by changing the last digit to some other valid hex digit. - Change directory into the
~/.julia/dev/Downloads - Start Julia 1.6 in that directory with the
--projectoption - Try the problematic download from the Julia REPL:
using Downloads: download url = "https://github.com/JuliaBinaryWrappers/GLFW_jll.jl/releases/download/GLFW-v3.3.3%2B0/GLFW.v3.3.3.x86_64-linux-gnu.tar.gz" file = download(url) @assert open(Base._crc32c, file) == 0x40e5ef54
If that works behind a firewall without error, downloading the file and getting the right CRC checksum, then the problem is fixed by setting this option.
Anyone?
Have a Windows 10 laptop. but am not behind a firewall, sorry.
[quote=“StefanKarpinski, post:31, topic:58021”]
(@v1.6) pkg> dev Downloads
Cloning git-repo `https://github.com/JuliaLang/Downloads.jl.git`
ERROR: failed to clone from https://github.com/JuliaLang/Downloads.jl.git, error: GitError(Code:ERROR, Class:OS, failed to send request: The connection with the server was terminated abnormally
)
julia> using Downloads
(@v1.6) pkg> dev Downloads
Cloning git-repo `https://github.com/JuliaLang/Downloads.jl.git`
ERROR: failed to clone from https://github.com/JuliaLang/Downloads.jl.git, error: GitError(Code:ERROR, Class:OS, failed to send request: The connection with the server was terminated abnormally
)
julia> names(Downloads)
6-element Vector{Symbol}:
:Downloader
:Downloads
:RequestError
:Response
:download
:request