Bad cert when accessing the General registry

Tooling Package Management
1

I just started getting errors about untrusted SSL cert when accessing the General registry.

If I try to update General:

(@v1.10) pkg> registry update General
    Updating registry at `~/.julia/registries/General.toml`
┌ Error: Some registries failed to update:
│     — /Users/dmatz/.julia/registries/General.toml — failed to download from https://pkg.julialang.org/registry/23338594-aafe-5451-b93e-139f81909106/89c6967ae14017d23abb5eda0c65a7c263ca9aac. Exception: RequestError: HTTP/2 301 (SSL certificate problem: Invalid certificate chain) while requesting https://pkg.julialang.org/registry/23338594-aafe-5451-b93e-139f81909106/89c6967ae14017d23abb5eda0c65a7c263ca9aac
└ @ Pkg.Registry ~/.julia/juliaup/julia-1.10.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Pkg/src/Registry/Registry.jl:510

Another type of error when my CI system tries to get General in a fresh depot:

$ julia -e 'using Pkg; Pkg.Registry.add("General")'
ERROR: could not download https://pkg.julialang.org/registry/23338594-aafe-5451-b93e-139f81909106/89c6967ae14017d23abb5eda0c65a7c263ca9aac
Stacktrace:
  [1] pkgerror(msg::String)
    @ Pkg.Types /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Types.jl:55
  [2] (::Pkg.Types.var"#82#85"{Pkg.Types.Context, String, Pkg.Types.RegistrySpec})(tmp::String)
    @ Pkg.Types /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Types.jl:1035
  [3] mktempdir(fn::Pkg.Types.var"#82#85"{Pkg.Types.Context, String, Pkg.Types.RegistrySpec}, parent::String; prefix::String)
    @ Base.Filesystem ./file.jl:729
  [4] mktempdir (repeats 2 times)
    @ ./file.jl:727 [inlined]
  [5] clone_or_cp_registries(ctx::Pkg.Types.Context, regs::Vector{Pkg.Types.RegistrySpec}, depot::String)
    @ Pkg.Types /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Types.jl:1027
  [6] clone_or_cp_registries
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Types.jl:1020 [inlined]
  [7] #add#7
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Registry.jl:33 [inlined]
  [8] add
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Registry.jl:29 [inlined]
  [9] #add#5
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Registry.jl:26 [inlined]
 [10] add
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Registry.jl:26 [inlined]
 [11] add(regs::Vector{String}; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.Registry /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Registry.jl:25
 [12] add
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Registry.jl:25 [inlined]
 [13] #add#1
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Registry.jl:24 [inlined]
 [14] add(reg::String)
    @ Pkg.Registry /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Registry.jl:24
 [15] top-level scope
    @ none:1
caused by: HTTP/2 301 (Cert verify failed: BADCERT_NOT_TRUSTED) while requesting https://pkg.julialang.org/registry/23338594-aafe-5451-b93e-139f81909106/89c6967ae14017d23abb5eda0c65a7c263ca9aac

Is anyone else seeing this?

2

what Linux distribution (and version) are you on?

3

The top example is on my Mac, and the bottom example is a CI job running on our Linux system, which is CentOS 7.

4

Ha. At CERN we have this problem as well, where the stock COS7 with Julia (which is official binary but the our software environment would overload a bunch of things) would give you cert error, but only if you’re not using the official binary from juliaup.

As for your local macOS laptop, I’m not sure, I just tried it now and it works fine.

5

Some co-workers are also seeing the issue, but only when on our company network. Must be an issue on our side.

6

Is there a proxy/firewall? Those will sometimes MITM your connections. In such cases there’s usually some company-specific certificate authority cert that needs to be added to allow the transparent proxy to perform this MITM. It will read all your traffic in such a case, even over HTTPS connections.

12

If I use my browser to navigate to your us-west URL, it tries to download the registry. If I navigate to us-east, it seems to forward to “storage.julialang.net”, and my browser complains about an invalid certificate.

If I force Julia to use the us-west server, things are working great:

$ JULIA_PKG_SERVER=https://us-west.pkg.julialang.org julia

(To be clear, my original issue was unrelated to HTTP.jl.)

13

The root certificate for https://storage.julialang.net (which https://us-east.pkg.julialang.org redirects to) is ISRG Root X2 and the root certificate for https://us-west.pkg.julialang.org (and https://us-east.pkg.julialang.org too) is ISRG Root X1. Is it possible that the former isn’t trusted (yet) by your system?

See https://letsencrypt.org/certificates/ for some more details. In particular they list a couple of test pages where you can verify the theory above. For example, https://valid-isrgrootx1.letsencrypt.org/ present a valid certificate signed by ISRG Root X1 and https://valid-isrgrootx2.letsencrypt.org/ a valid cert signed by ISRG Root X2.

14

Thanks for that detail, I’ll pass that along. I’ve opened a ticket with my company’s IT department. They must have changed something this week!

15

Not necessarily a change on your end since the redirection to https://storage.julialang.net was deployed yesterday. Regardless, all certs are valid as far as I can tell.

16

You can locally try to curl the two letsencrypt URLs that Fredrik posted and check if the https://valid-isrgrootx2.letsencrypt.org/ one doesn’t work. In that case that would confirm the issue.

17

I tried both https://valid-isrgrootx1.letsencrypt.org/ and https://valid-isrgrootx2.letsencrypt.org/ in both my browser and with curl, and everything worked fine. But https://us-east.pkg.julialang.org/registry/23338594-aafe-5451-b93e-139f81909106/6e59100195a18329cf56e7a7b76c065fa25d3a5c continues to give me issues.

18

Did you mean .org? Because I get redirected to https://storage.julialang.net.

19

I meant .net, sorry. Updated.

20

My issue bounced through the bureaucracy for a week or so, but it finally made it to the network team. The issue appears to be resolved. Unfortunately, they didn’t leave any notes about what they did, so I’m not sure if it was the root cert or something else.

Thank you all again for help on this!

21

You indicate that on April 9th the redirection was to https://storage.julialang.net . What was the redirect from? That information might help me to pinpoint the source of my CERT_TRUST_IS_UNTRUSTED_ROOT error, which has prevented me from downloading packages / updating packages / updating registry for the last 6 weeks and is probably related.

22

By default Pkg connects to https://pkg.julialang.org which redirects you to a regional server (e.g. https://eu-central.pkg.julialang.org if you are in Europe). The regional endpoint serves some data itself (no redirect) and redirects some requests to https://storage.julialang.org. This hasn’t really changed since my post though, and that was two years ago at this point.