Problem with curl.exe (Windows) and package installation

oheil · October 6, 2019, 5:47pm

Lets stay in plain julia for this problem, as I don’t have juliaPro installed.

aerdely · October 6, 2019, 5:53pm

Ok. After executing in Windows terminal:

C:\>curl.exe --insecure -s -S -g -L -f -o "C:\Users\aerde\.julia\conda\3\installer.exe" https://repo.continuum.io/miniconda/Miniconda3-4.5.4-Windows-x86_64.exe

which was executed with no error messages, I tried the following in plain Julia terminal:

julia> using IJulia

julia> notebook()
install Jupyter via Conda, y/n? [y]: y
[ Info: Downloading miniconda installer ...
┌ Error: Download failed: curl: (35) schannel: next InitializeSecurityContext failed: Unknown error (0x80092012) - La funci�n de revocaci�n no puede comprobar la revocaci�n para el certificado.
└ @ Base download.jl:43
ERROR: failed process: Process(`'C:\WINDOWS\System32\curl.exe' -s -S -g -L -f -o 'C:\Users\aerde\.julia\conda\3\installer.exe' https://repo.continuum.io/miniconda/Miniconda3-4.5.4-Windows-x86_64.exe`, ProcessExited(35)) [35]

Stacktrace:
 [1] pipeline_error at .\process.jl:813 [inlined]
 [2] download_curl(::String, ::String, ::String) at .\download.jl:44
 [3] download at .\download.jl:52 [inlined]
 [4] _install_conda(::String, ::Bool) at C:\Users\aerde\.julia\packages\Conda\kLXeC\src\Conda.jl:160
 [5] _install_conda(::String) at C:\Users\aerde\.julia\packages\Conda\kLXeC\src\Conda.jl:152
 [6] runconda(::Cmd, ::String) at C:\Users\aerde\.julia\packages\Conda\kLXeC\src\Conda.jl:111
 [7] #add#1(::String, ::typeof(Conda.add), ::String, ::String) at C:\Users\aerde\.julia\packages\Conda\kLXeC\src\Conda.jl:184
 [8] add at C:\Users\aerde\.julia\packages\Conda\kLXeC\src\Conda.jl:183 [inlined] (repeats 2 times)
 [9] find_jupyter_subcommand(::String) at C:\Users\aerde\.julia\packages\IJulia\fRegO\src\jupyter.jl:23
 [10] #notebook#8 at C:\Users\aerde\.julia\packages\IJulia\fRegO\src\jupyter.jl:88 [inlined]
 [11] notebook() at C:\Users\aerde\.julia\packages\IJulia\fRegO\src\jupyter.jl:87
 [12] top-level scope at REPL[2]:1

oheil · October 6, 2019, 6:00pm

Ok, I found the cure (in Windows 7, whats your version?)

Download the following file:
https://curl.haxx.se/ca/cacert.pem
Its from this site:
https://curl.haxx.se/docs/caextract.html

Rename the file cacert.pem to curl-ca-bundle.crt and place it into C:\Windows\System32\

With this the certificate problems with curl should be solved. In the future you may have to do it again, when root certificates do change.

The complete procedure is described here:
https://curl.haxx.se/docs/sslcerts.html
under
Certificate Verification, 4. If you're using the curl command line tool, you can specify your own CA cert

Complete quote:

If you’re using the curl command line tool on Windows, curl will search for a CA cert file named “curl-ca-bundle.crt” in these directories and in this order:
application's directory
current working directory
Windows System directory (e.g. C:\windows\system32)
Windows Directory (e.g. C:\windows)
all directories along %PATH%

aerdely · October 6, 2019, 6:02pm

I’m in Windows 10

oheil · October 6, 2019, 6:03pm

Should work the same way, as your curl is also in the same folder.

aerdely · October 6, 2019, 11:22pm

In Windows terminal:

C:\Windows\System32>dir curl*.*
 El volumen de la unidad C es Windows
 El número de serie del volumen es: 500A-08B1

 Directorio de C:\Windows\System32

06/10/2019  01:03 p. m.           224,085 curl-ca-bundle.crt
10/08/2019  12:05 p. m.           421,376 curl.exe
               2 archivos        645,461 bytes
               0 dirs  2,714,426,331,136 bytes libres

C:\Windows\System32>

In Julia:

julia> using IJulia

julia> notebook()
install Jupyter via Conda, y/n? [y]: y
[ Info: Downloading miniconda installer ...
┌ Error: Download failed: curl: (35) schannel: next InitializeSecurityContext failed: Unknown error (0x80092012) - La funci�n de revocaci�n no puede comprobar la revocaci�n para el certificado.
└ @ Base download.jl:43
ERROR: failed process: Process(`'C:\WINDOWS\System32\curl.exe' -s -S -g -L -f -o 'C:\Users\aerde\.julia\conda\3\installer.exe' https://repo.continuum.io/miniconda/Miniconda3-4.5.4-Windows-x86_64.exe`, ProcessExited(35)) [35]

Stacktrace:
 [1] pipeline_error at .\process.jl:813 [inlined]
 [2] download_curl(::String, ::String, ::String) at .\download.jl:44
 [3] download at .\download.jl:52 [inlined]
 [4] _install_conda(::String, ::Bool) at C:\Users\aerde\.julia\packages\Conda\kLXeC\src\Conda.jl:160
 [5] _install_conda(::String) at C:\Users\aerde\.julia\packages\Conda\kLXeC\src\Conda.jl:152
 [6] runconda(::Cmd, ::String) at C:\Users\aerde\.julia\packages\Conda\kLXeC\src\Conda.jl:111
 [7] #add#1(::String, ::typeof(Conda.add), ::String, ::String) at C:\Users\aerde\.julia\packages\Conda\kLXeC\src\Conda.jl:184
 [8] add at C:\Users\aerde\.julia\packages\Conda\kLXeC\src\Conda.jl:183 [inlined] (repeats 2 times)
 [9] find_jupyter_subcommand(::String) at C:\Users\aerde\.julia\packages\IJulia\fRegO\src\jupyter.jl:23
 [10] #notebook#8 at C:\Users\aerde\.julia\packages\IJulia\fRegO\src\jupyter.jl:88 [inlined]
 [11] notebook() at C:\Users\aerde\.julia\packages\IJulia\fRegO\src\jupyter.jl:87
 [12] top-level scope at REPL[2]:1

julia>

Didn’t work

oheil · October 7, 2019, 12:54pm

This is strange. Just for putting it all together.
I am now on a Windows 10 machine, downloaded this version:
https://curl.haxx.se/windows/dl-7.66.0_2/curl-7.66.0_2-win64-mingw.zip
into some download path, and extracted there, not in any %PATH%.

I just copied the extracted curl.exe only into d:\Temp
and this is the sequence I can reproduce, inbetween I copy the bundled curl-ca-bundle.crt to d:\Temp:

D:\Temp>dir curl*
 Datenträger in Laufwerk D: ist Daten
 Volumeseriennummer: 2177-D388

 Verzeichnis von D:\Temp

11.09.2019  05:49         4.050.552 curl.exe
               1 Datei(en),      4.050.552 Bytes
               0 Verzeichnis(se), 742.450.786.304 Bytes frei

D:\Temp>d:\Temp\curl.exe -s -S -g -L -f -o "D:\Temp\curl-installer.exe" https://repo.continuum.io/miniconda/Miniconda3-4.5.4-Windows-x86_64.exe
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

D:\Temp>dir curl*
 Datenträger in Laufwerk D: ist Daten
 Volumeseriennummer: 2177-D388

 Verzeichnis von D:\Temp

28.08.2019  03:12           224.085 curl-ca-bundle.crt
11.09.2019  05:49         4.050.552 curl.exe
               2 Datei(en),      4.274.637 Bytes
               0 Verzeichnis(se), 742.393.077.760 Bytes frei

D:\Temp>d:\Temp\curl.exe -s -S -g -L -f -o "D:\Temp\curl-installer.exe" https://repo.continuum.io/miniconda/Miniconda3-4.5.4-Windows-x86_64.exe

D:\Temp>dir curl*
 Datenträger in Laufwerk D: ist Daten
 Volumeseriennummer: 2177-D388

 Verzeichnis von D:\Temp

28.08.2019  03:12           224.085 curl-ca-bundle.crt
07.10.2019  13:17        57.482.400 curl-installer.exe
11.09.2019  05:49         4.050.552 curl.exe
               3 Datei(en),     61.757.037 Bytes
               0 Verzeichnis(se), 742.335.594.496 Bytes frei

D:\Temp>

Removing file d:\Temp\curl-ca-bundle.crt brings back the error message again.

In the meantime I have learned that Windows10 has a preinstalled curl.exe in C:\Windows\system32, this was new to me, Windows7 didn’t have it:

D:\Temp>C:\\WINDOWS\\System32\\curl.exe -V
curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
Release-Date: [unreleased]
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL

Trying to download the installer with no cert file in the path (removed from d:\temp) was successful on my machine, no error:
C:\\WINDOWS\\System32\\curl.exe -s -S -g -L -f -o "D:\Temp\curl-installer.exe" https://repo.continuum.io/miniconda/Miniconda3-4.5.4-Windows-x86_64.exe

I am quite clueless now and can only suggest to check things out just out of despair:

Rename c:\windows\system32\curl.exe to something like curl.exe.OLD_VERSION (if possible, my machine refused to rename it)
download current curl version from above link and put the curl.exe into some new path, e.g. D:\Temp
do the same steps as described above and see if it reproduces my outcome
Remove or rename .julia folder in C:\Users\aerde\
start julia again (plain 1.2)
to avoid the powershell I first started a CMD shell, and in there I run C:\Users\oheil\AppData\Local\Julia-1.2.0\bin\julia.exe
do the find_curl() trick in julia as in the other thread. For me it is: Base.find_curl() = “D:\Temp\curl.exe”
try again using IJulia with the new curl version, with and without curl-ca-bundle.crt in D:\Temp\
After add IJulia leave julia via Ctrl-D , start again with the same line, you can just cursor up and the julia.exe should show, return starts it.

These are the steps in julia 1.2:

julia> using Pkg

julia> Base.find_curl() = "D:\\Temp\\curl.exe"

(v1.2) pkg> add IJulia
   Cloning default registries into `C:\Users\oheil\.julia`
   Cloning registry from "https://github.com/JuliaRegistries/General.git"
     Added registry `General` to `C:\Users\oheil\.julia\registries\General`
 Resolving package versions...
 Installed VersionParsing ── v1.1.3
 Installed ZMQ ───────────── v1.0.0
 Installed Conda ─────────── v1.3.0
 Installed Parsers ───────── v0.3.7
 Installed Compat ────────── v2.1.0
 Installed IJulia ────────── v1.20.0
 Installed JSON ──────────── v0.21.0
 Installed MbedTLS ───────── v0.7.0
 Installed SoftGlobalScope ─ v1.0.10
 Installed BinaryProvider ── v0.5.6
  Updating `C:\Users\oheil\.julia\environments\v1.2\Project.toml`
  [7073ff75] + IJulia v1.20.0
  Updating `C:\Users\oheil\.julia\environments\v1.2\Manifest.toml`
  [b99e7846] + BinaryProvider v0.5.6
  [34da2185] + Compat v2.1.0
  [8f4d0f93] + Conda v1.3.0
  [7073ff75] + IJulia v1.20.0
  [682c06a0] + JSON v0.21.0
  [739be429] + MbedTLS v0.7.0
  [69de0a69] + Parsers v0.3.7
  [b85f4697] + SoftGlobalScope v1.0.10
  [81def892] + VersionParsing v1.1.3
  [c2297ded] + ZMQ v1.0.0
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  Building Conda ──→ `C:\Users\oheil\.julia\packages\Conda\kLXeC\deps\build.log`
  Building ZMQ ────→ `C:\Users\oheil\.julia\packages\ZMQ\ABGOx\deps\build.log`
  Building MbedTLS → `C:\Users\oheil\.julia\packages\MbedTLS\a1JFn\deps\build.log`
  Building IJulia ─→ `C:\Users\oheil\.julia\packages\IJulia\fRegO\deps\build.log`

###  CTRL-D NOW OUT OF JULIA #### START JULIA NEW BUT NOT THE CMD SHELL
### THAN:

julia> Base.find_curl() = "D:\\Temp\\curl.exe"

julia> using IJulia
[ Info: Precompiling IJulia [7073ff75-c697-5162-941a-fcdaad2a7d2a]

julia> notebook()
install Jupyter via Conda, y/n? [y]:
[ Info: Downloading miniconda installer ...
┌ Error: Download failed: curl: (60) SSL certificate problem: unable to get local issuer certificate
└ @ Base download.jl:43
ERROR: failed process: Process(`'D:\Temp\curl.exe' -s -S -g -L -f -o 'C:\Users\oheil\.julia\conda\3\installer.exe' https://repo.continuum.io/miniconda/Miniconda3-4.5.4-Windows-x86_64.exe`, ProcessExited(60)) [60]

Stacktrace:
 [1] pipeline_error at .\process.jl:813 [inlined]
 [2] download_curl(::String, ::String, ::String) at .\download.jl:44
 [3] download at .\download.jl:52 [inlined]
 [4] _install_conda(::String, ::Bool) at C:\Users\oheil\.julia\packages\Conda\kLXeC\src\Conda.jl:160
 [5] _install_conda(::String) at C:\Users\oheil\.julia\packages\Conda\kLXeC\src\Conda.jl:152
 [6] runconda(::Cmd, ::String) at C:\Users\oheil\.julia\packages\Conda\kLXeC\src\Conda.jl:111
 [7] #add#1(::String, ::typeof(Conda.add), ::String, ::String) at C:\Users\oheil\.julia\packages\Conda\kLXeC\src\Conda.jl:184
 [8] add at C:\Users\oheil\.julia\packages\Conda\kLXeC\src\Conda.jl:183 [inlined] (repeats 2 times)
 [9] find_jupyter_subcommand(::String) at C:\Users\oheil\.julia\packages\IJulia\fRegO\src\jupyter.jl:23
 [10] #notebook#8 at C:\Users\oheil\.julia\packages\IJulia\fRegO\src\jupyter.jl:88 [inlined]
 [11] notebook() at C:\Users\oheil\.julia\packages\IJulia\fRegO\src\jupyter.jl:87
 [12] top-level scope at REPL[22]:1

### NOW COPY THE FILE curl-ca-bundle.crt INTO D:\TEMP\

julia> notebook()
install Jupyter via Conda, y/n? [y]:
[ Info: Downloading miniconda installer ...
[ Info: Installing miniconda ...

If there is something ambiguous in the above steps, please ask. It is of some importance that it is done like that to make sure that you are really avoiding powershell and the c:\windows\system32\curl.exe !

I believe the problem is the Windows 10 curl.exe together with some outdated certificates and after some more tests I think that the windows 10 preinstalled curl.exe does not look for the file curl-ca-bundle.crt

I can understand if you have enough from this hassle, but as long as I have ideas, I usually don’t give up.

oheil · October 7, 2019, 1:24pm

I have looked at the sources of curl.exe and in the case, that the SSL backend of curl is schannel, it will not look for the file curl-ca-bundle.crt

This is the code in tool_operate.c:

   if(tls_backend_info->backend != CURLSSLBACKEND_SCHANNEL) {
     char *env;
     env = curlx_getenv("CURL_CA_BUNDLE");
     if(env) {
       config->cacert = strdup(env);
       if(!config->cacert) {
         curl_free(env);
         helpf(global->errors, "out of memory\n");
         return CURLE_OUT_OF_MEMORY;
       }
     }
     else {
       env = curlx_getenv("SSL_CERT_DIR");
       if(env) {
         config->capath = strdup(env);
         if(!config->capath) {
           curl_free(env);
           helpf(global->errors, "out of memory\n");
           return CURLE_OUT_OF_MEMORY;
         }
         capath_from_env = true;
       }
       else {
         env = curlx_getenv("SSL_CERT_FILE");
         if(env) {
           config->cacert = strdup(env);
           if(!config->cacert) {
             curl_free(env);
             helpf(global->errors, "out of memory\n");
             return CURLE_OUT_OF_MEMORY;
           }
         }
       }
     }

     if(env)
       curl_free(env);
#ifdef WIN32
     else {
       result = FindWin32CACert(config, tls_backend_info->backend,
                                "curl-ca-bundle.crt");
     }
#endif
   }
   curl_easy_cleanup(curltls);
 }

Your error message:

Error: Download failed: curl: (35) schannel: ...

seems to be explaining now, why it does not work with c.\windows\system32\curl.exe

So, my hopes still rely on the downloaded curl version, but I understand, that if this works, it is not yet a final solution for the future for you.

Tamas_Papp · October 7, 2019, 2:15pm

If @aerdely or someone affected is willing to invest in tracking this down, it could be worth opening an issue. The first thing to check is whether this is fixed on master, eg by

EDIT: I overlooked the fact that the above ended up in 1.2. Still, making an MWE on master would be the first step.

oheil · October 7, 2019, 3:42pm

I plan to open an issue and a complete problem and solution discription when the underlying problem is identified. Currently it is still a trial and error, what’s really going on.

Currently the theory is, that the Windows 10 pre installed curl.exe in C:\Windows\System32 together with outdated root certificates will cause download problems, if you are not using the powershell download capability. But this is still open to be confirmed.

The current workaround would be, to download an alternative windows curl (which not relies on schannel for TLS) and provide the file curl-ca-bundle.crt to this curl.exe, but this is probably not the end solution. I think we have to fight with Windows Trusted Root Certificates for the real solution, but we are not yet there.

The Certicifate of https://repo.continuum.io/ is issued by “COMODO ECC Domain Validation Secure Server CA 2” which is NOT in the list of Trusted Certs on my Windows 7 machine and starts at 1. July 2019. But it is part of the mentioned curl-ca-bundle.crt.

Another indication that we are on the right track. I am sorry @aerdely that this seems so tedious but I am just trying and learning without being able to reproduce the problem completely. I guess we will end up with an easy solution and everybody why it took so long.

oheil · October 7, 2019, 4:35pm

@aerdely Another thing we should rule out:

Also of relevancy is that sometimes in enterprise environments there are man-in-middle proxies that use certificates trusted only by domain machines for which the specific certificates were installed. Sometimes this leads to issues when a machine within that environment accesses the network, but had never had the specific certificates installed.

aerdely · October 7, 2019, 11:28pm

But even in my personal computer at home it failed. Thanks a lot for your time and effort I really appreciate it and tells a lot about the good will of the Julia community.

oheil · October 8, 2019, 9:36am

I tried also on a Windows 10 machine without any network restrictions, and was able to reproduce an error with the windows 10 preinstalled curl version and workaround with a newer curl download which is not using schannel as TLS backend. So I am still in good faith

Tamas_Papp · October 8, 2019, 9:54am

I am not familiar with Windows (last used it in 1997), but would it be possible to package this workaround, eg as a script that detects this problem and then downloads a more recent version of curl and puts it in the path?

oheil · October 8, 2019, 9:58am

Maybe, but before I start on this I would like to have confirmed that we are on the right track.

Anyway, the easiest and best controllable approach for Julia would be, to always download the newest official windows curl version (without schannel), so no need for any tests.

Tamas_Papp · October 8, 2019, 10:05am

I am now confused: I thought Julia included a libcurl. Why is an external binary needed at all?

oheil · October 8, 2019, 11:23am

It is an install script, which is called when installing IJulia and calling notebook(), but I didn’t looked into exact details. The issue for the OP is, that powershell downloading can’t be used, so somewhere Julia or some installer falls back to curl, which is then found as c:\Windows\system32\curl.exe. So, it is still not clear, how all comes together in detail. Therefor creating some solution as you said is too early.

joa-quim · October 8, 2019, 3:44pm

I’ve been following this and got confused as others. For me it (Win10 updated) it works fine to install Plots.jl and its FFMPEG dependency. What doesn’t work so well is if I remove the Plots package the FFMPEG dependency is left behind as garbage. Even ] gc doesn’t clean it.

oheil · October 8, 2019, 7:17pm

I am confused also. There are so many strange things going on. What I was able to reproduce instantly was the usage of c:\Windows\system32\curl.exe (Windows 10) if powershell is not available, together with certificate problems. So I jumped on this.

phgmacedo · October 9, 2019, 3:08am

If the problem is with curl, can you try downloading with some other command?
Like:

function download(url::AbstractString, filename::AbstractString)
    url = download_url(url)
   elseif Sys.iswindows() #don't use curl
        run(`powershell iwr -Uri $url | out-file $filename`)
    elseif Sys.which("wget") !== nothing
        try
            run(`wget -O $filename $url`)
        catch
            rm(filename, force=true)  # wget always creates a file
            rethrow()
        end
    elseif Sys.which("fetch") !== nothing
        run(`fetch -f $filename $url`)
    else
        error("No download agent available; install curl, wget, or fetch.")
    end
    return filename
end