How to make Pkg use specific SSL certificate for verifying servers? (General registry adding issue on work network | Windows)

scimas · June 16, 2021, 2:57pm

I have cleaned up the .julia/registries directory.
I’m accessing julia from powershell and/or bash (that comes with git for windows).

I’ve read some similar threads on discourse, but they didn’t help to solve the problem. So I’m listing out whatever information I have and my observations if that helps solve the problem.

Trying to add the General registry redirects to the github repository. If I instead try to explicitly add https://pkg.julialang.org I get this error:

ERROR: failed to clone from https://pkg.julialang.org, error: GitError(Code:ERROR, Class:Net, cannot redirect from 'pkg.julialang.org' to 'us-west.pkg.julialang.org')

So if I try to add us-west.pkg... I get this error:

ERROR: failed to clone from https://us-west.pkg.julialang.org, error: GitError(Code:ERROR, Class:HTTP, request failed with status code: 404)

Both the https://pkg.ju… and https://us-west.ju… URLs are accessible through the browser. I get the

You have reached a Julia package server. It is not meant for browsing but for serving resources required by Julia clients to install packages. To discover and explore the Julia package ecosystem, see this page for a list of services.

message when accessing through browser.

I tried to curl the pkg.ju… URL and I got two different results.
From powershell I get a 200 response with the “You have reached package server” html page as contents.
From bash, with or without the -L flag, I get

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

So I’m assuming julia is running into a similar problem of not being able to verifying the server or something like that. But I’m not sure how to solve it since those URLs redirect and give responses just fine from the browser and powershell.

scimas · June 16, 2021, 4:22pm

Alright, I’ve narrowed down the problem. My company is using Zscaler for SSL / firewall stuff (not an networking / security expert, so don’t know the exact purpose). Somehow powershell and browsers know how to verify certificates against Zscaler’s certificate.

If I point the bash curl to the zscaler certificate (curl --cacert /path/to/certificate/file URL), then curl is able to get the same “You have reached package server” page.

So now the question is how do I point Julia to this certificate?

scimas · June 16, 2021, 4:58pm

Found out about the SSL_CERT_FILE env variable. But setting that just gives a different error:

Your Julia is built with a SSL/TLS engine that libgit2 doesn't know how to configure to use a file or directory of certificate authority roots,...

Would really appreciate any help solving this.

scimas · June 17, 2021, 2:38pm

Bump

scimas · June 19, 2021, 1:25am

Bump

fredrikekre · June 21, 2021, 6:18pm

Have you tried setting the various variables from NetworkOptions.jl?

scimas · June 21, 2021, 9:53pm

I hadn’t tried that, I didn’t even know there were such options.

I have tried setting ca_roots, ca_roots_path, JULIA_NO_VERIFY_HOSTS now (independently). They do not seem to have any effect at all. I still get the same “cannot redirect” error on “pkg.julialang” and a 404 on “us-west.pkg…”. Even setting JULIA_NO_VERIFY_HOSTS = "**.julialang.org" - verified by calling NetworkOptions.verify_host("https://pkg.julialang.org") == false - does not help, same errors persist.

Also, thank you for replying, I was starting to feel like my account was under some kind of shadow ban after almost a week of no replies.

johnh · June 22, 2021, 2:28am

@scimas I saw your query, but did not have much to offer in an answer.
I now see that I contributed to a thread on a similar problem in 2019. My insight does not tell you anything new, sorry.

scimas · June 22, 2021, 3:24am

That thread doesn’t really have possible solutions, but thank you for the reference.

johnh · June 22, 2021, 6:35am

I realise that - I was trying to reassure you that people do see your problems on here.
Would you perhaps contact Zscaler? Clearly this is an issue and they might be able to patch their software or suggest a workaround.

fredrikekre · June 22, 2021, 8:11am

Reading the first post again, what you are trying to do (add registry that is not a git repository by URL) is not supposed to work.

To remove the package manager from the equation, try with this example instead:

using Downloads
Downloads.download("https://pkg.julialang.org/registries", stdout; verbose=true)

If you can get that to work properly, then the package manager should work too. Can you show the output of that example (e.g. with and without setting JULIA_NO_VERIFY_HOSTS)?

scimas · June 22, 2021, 2:18pm

Without JULIA_NO_VERIFY_HOSTS

*   Trying 151.101.194.217:443...
* Connected to pkg.julialang.org (151.101.194.217) port 443 (#0)
> GET /registries HTTP/1.1
Host: pkg.julialang.org
Accept: */*
User-Agent: curl/7.73.0 julia/1.6

* Mark bundle as not supporting multiuse
< HTTP/1.1 301 NA (west) internal redirect trigger
< Xonnection: close
< Content-Length: 0
< Server: Varnish
< Retry-After: 0
< Location: https://us-west.pkg.julialang.org/registries
< x-geo-continent: NA
< x-geo-country: US
< x-geo-region: CO
< Accept-Ranges: bytes
< Date: Tue, 22 Jun 2021 13:57:41 GMT
< Via: 1.1 varnish
< X-Served-By: cache-den8277-DEN
< X-Cache: HIT
< X-Cache-Hits: 0
< X-Timer: S1624370261.040583,VS0,VE1
< Connection: Keep-Alive
<
* Connection #0 to host pkg.julialang.org left intact
* Issue another request to this URL: 'https://us-west.pkg.julialang.org/registries'
*   Trying 52.24.172.208:443...
* Connected to us-west.pkg.julialang.org (52.24.172.208) port 443 (#1)
> GET /registries HTTP/1.1
Host: us-west.pkg.julialang.org
Accept: */*
User-Agent: curl/7.73.0 julia/1.6

* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.21.0
< Date: Tue, 22 Jun 2021 13:57:41 GMT
< Content-Type: application/octet-stream
< Content-Length: 88
< Last-Modified: Tue, 22 Jun 2021 13:50:24 GMT
< Connection: keep-alive
< ETag: "60d1eaa0-58"
< Content-Type: text/plain
< Accept-Ranges: bytes
<
* Connection #1 to host us-west.pkg.julialang.org left intact
/registry/23338594-aafe-5451-b93e-139f81909106/c8bc0efa717d4b49854b0fd76d896059ef423de2
Base.TTY(Base.Libc.WindowsRawSocket(0x0000000000000310) open, 0 bytes waiting)

With JULIA_NO_VERIFY_HOSTS = "**.julialang.org"

*   Trying 151.101.194.217:443...
* Connected to pkg.julialang.org (151.101.194.217) port 443 (#0)
> GET /registries HTTP/1.1
Host: pkg.julialang.org
Accept: */*
User-Agent: curl/7.73.0 julia/1.6

* Mark bundle as not supporting multiuse
< HTTP/1.1 301 NA (west) internal redirect trigger
< Xonnection: close
< Content-Length: 0
< Server: Varnish
< Retry-After: 0
< Location: https://us-west.pkg.julialang.org/registries
< x-geo-continent: NA
< x-geo-country: US
< x-geo-region: CO
< Accept-Ranges: bytes
< Date: Tue, 22 Jun 2021 14:02:05 GMT
< Via: 1.1 varnish
< X-Served-By: cache-den8220-DEN
< X-Cache: HIT
< X-Cache-Hits: 0
< X-Timer: S1624370525.488132,VS0,VE1
< Connection: Keep-Alive
<
* Connection #0 to host pkg.julialang.org left intact
* Issue another request to this URL: 'https://us-west.pkg.julialang.org/registries'
*   Trying 52.24.172.208:443...
* Connected to us-west.pkg.julialang.org (52.24.172.208) port 443 (#1)
> GET /registries HTTP/1.1
Host: us-west.pkg.julialang.org
Accept: */*
User-Agent: curl/7.73.0 julia/1.6

* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.21.0
< Date: Tue, 22 Jun 2021 14:02:05 GMT
< Content-Type: application/octet-stream
< Content-Length: 88
< Last-Modified: Tue, 22 Jun 2021 13:50:24 GMT
< Connection: keep-alive
< ETag: "60d1eaa0-58"
< Content-Type: text/plain
< Accept-Ranges: bytes
<
* Connection #1 to host us-west.pkg.julialang.org left intact
/registry/23338594-aafe-5451-b93e-139f81909106/c8bc0efa717d4b49854b0fd76d896059ef423de2
Base.TTY(Base.Libc.WindowsRawSocket(0x0000000000000328) open, 0 bytes waiting)

I believe those are the correct and exact same outputs in both cases. And yet it is also true that adding General registry redirects to github. And adding "https://pkg.julialang.org/registries" gives a cannot redirect error.

(@v1.6) pkg> registry add General
     Cloning registry from "https://github.com/JuliaRegistries/General.git"

(@v1.6) pkg> registry add "https://pkg.julialang.org/registries"
     Cloning registry from "https://pkg.julialang.org/registries"
ERROR: failed to clone from https://pkg.julialang.org/registries, error: GitError(Code:ERROR, Class:Net, cannot redirect from 'pkg.julialang.org' to 'us-west.pkg.julialang.org')

Yes, contacting them or even the IT department before them is an option. But since accessing the URL does work from browser and the powershell curl, I’d need more info to convince them it’s their problem rather than julia’s problem.

fredrikekre · June 22, 2021, 3:46pm

And it seems to work too though? Have you set JULIA_PKG_SERVER to the empty string perhaps? It looks like the code for pkg> registry add General doesn’t take the Pkg server code path…

scimas · June 22, 2021, 4:24pm

No, I’m not setting any environment variables. All outputs I have posted are from fresh sessions to make sure there’s no pollution in the environment.

fredrikekre · June 23, 2021, 10:35am

Okay, so perhaps downloading is just a red herring and the actual problem is unpacking the content? Can you try this:

using Pkg, Downloads

# 1. Download file. Does this work and give you a gzipped tarball?
#    The sha256 sum of this file should be:
#        95e32e3b3f797373f5f618786e6fa535949dd5b53d0d6cd72eae098a4342ac02
#    for reference.
file = Downloads.download("https://pkg.julialang.org/registry/23338594-aafe-5451-b93e-139f81909106/60e2264e4b83c8920007abb1bc2440e990d108aa")

# 2. Unpack the tarball. Does this work and give you the expected registry
#    tree in the tmp path returned?
Pkg.PlatformEngines.unpack(file, mktempdir())

Can you also try using the 1.7 beta (Julia v1.7.0-beta2 is now available - #3 by giordano) or the nightly build (Julia Downloads (nightly binaries)). From Julia 1.7 Pkg does not unpack the registry (but does for packages still).

scimas · June 23, 2021, 3:21pm

fredrikekre:

Does this work and give you a gzipped tarball?
#    The sha256 sum of this file should be:
#        95e32e3b3f797373f5f618786e6fa535949dd5b53d0d6cd72eae098a4342ac02
#    for reference.

I do get a .tmp file, but it’s sha256 doesn’t match the one you have given as generated by stdlib SHA (93622d2586e2721b5a8812cce94468835d582e5a487078d896d970f4d8d088b5, probably because the repository has changed since you replied to this topic and I followed the steps).

Yes, I do get the General registry tree in the temp directory.

I cleared the .julia directory and tried to add the General registry from 1.7. It downloaded the registry from somewhere. But looks like julia 1.7 doesn’t show any info about where it’s getting things from (the Cloning registry from ... info logs are absent). Even Pkg.Registry.status() only shows the name of the registry, no URL.

Explicitly trying to add the registry from julia servers gives the same error, cannot redirect from pkg.julialang to us-west.pkg.julialang.

Following the download → unpack with 1.7 also gives same results - the proper registry tree, different sha256 (again, presumably because the registry changed between runs).
875488a268694ff6732d88be4a1485b25f84b83cc4d342878363a466d24fd072

scimas · June 23, 2021, 8:05pm

So I’ve gone down the rabbit hole of navigating the stacktrace and trying to figure out why this keeps failing.

It has led me to question: Is the pkg.julialang server not supposed to be the default registry?

Because, Pkg.jl source code lists the github repo as the default repo. Pkg.jl/Registry.jl at release-1.7 · JuliaLang/Pkg.jl · GitHub

Combined with Pkg.jl/Registry.jl at 81af75b902586919a858f06949b7c9e6121f1629 · JuliaLang/Pkg.jl · GitHub, registry add General just translates to going to github rather than julialang servers.

If instead you do Pkg.Registry.add(RegistrySpec(url="https://pkg.julialang....")), Pkg.jl/Registry.jl at 81af75b902586919a858f06949b7c9e6121f1629 · JuliaLang/Pkg.jl · GitHub and Pkg.jl/Registry.jl at 81af75b902586919a858f06949b7c9e6121f1629 · JuliaLang/Pkg.jl · GitHub ensure that you hit Pkg.jl/Registry.jl at 81af75b902586919a858f06949b7c9e6121f1629 · JuliaLang/Pkg.jl · GitHub (this definitely feels like a bug, there are no comments in the code and it is not clear to me why it’s trying to avoid the download_verify route). Which in turn leads to a ccall to git_clone in libgit2 and my code navigation abilities come to an end.

fredrikekre · June 23, 2021, 8:36pm

The URL asks for a specific revision so you should get the same. Are you sure you didn’t just compute the hash of the filename string: bytes2hex(sha256("run.jl")) vs bytes2hex(open(sha256, "run.jl"))?

But then everything works? Are you still trying to add by URL which is not supposed to work, e.g. from above:

scimas · June 23, 2021, 9:35pm

Ah, yes, I definitely hashed the path rather than the file itself.

Yes, everything seems to be working.

I was only trying to do that because updating registry from github usually takes a long time, it sits on 99% for a good chunk of time before finally getting to the work of adding / updating packages themselves.
The discourse threads talking about this suggested that the pkg servers were supposed to be the standard places to get the registry from. That was the primary motivation behind trying to add the general registry from that url. Now that 1.7 is refusing to show where it’s downloading the registry from, I guess I will just have to trust that it will select the proper place.

Also, could you elaborate why adding a registry with a URL is not supposed to work?

fredrikekre · June 23, 2021, 9:44pm

Okay, but the default behavior (since Julia 1.5) is to not use git but just download the tarball, so that should basically be instant (at least if you are not using Windows). In addition, in Julia 1.7 the tarball is not even unpacked, which fixes the performance problems on Windows too.

Yea, might be good to show this I suppose.

Because the URL API is for URLs pointing to git repositories (the package servers are not git repos).