Pkg certificate issue on linux

Tooling Package Management
1

Hi,

I have an issue downloading packages with Julia 1.7.2 on Debian Sid. (julia compiled from source)

From a fresh machine (I moved the .julia folder out of the way), I get a certificate error when trying to install a package:

(@v1.7) pkg> add Plots
    Updating registry at `~/.julia/registries/General.toml`
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = Error reading ca cert path /usr/lib/ssl/certs - mbedTLS: (-0x2900) X509 - Read/write of file failed while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry ~/src/julia/usr/share/julia/stdlib/v1.7/Pkg/src/Registry/Registry.jl:82
   Resolving package versions...
     Cloning [91a5bcdd-55d7-5caf-9e0b-520d859cae80] Plots from https://github.com/JuliaPlots/Plots.jl.git
TLS host verification: the identity of the server `github.com` could not be verified. Someone could be trying toman-in-the-middle your connection. It is also possible that the correct server is using an invalid certificate or that your system's certificate authority root store is misconfigured.
ERROR: failed to clone from https://github.com/JuliaPlots/Plots.jl.git, error: GitError(Code:ERROR, Class:HTTP, user rejected certificate for github.com)

(@v1.7) pkg>

It looks similar to this issue and similar other ones I found online.

I’m on my home machine without proxy, and the procedure worked before and seems to work on my laptop, so I don’t think it’s an issue with my network.

Any clues on how to debug this would be welcome

2

I stumbled upon the DebugArtifacts.jl code in a discussion about a similar issue. Cloning the repository and running it:

(@v1.7) pkg> activate .
  Activating project at `~/src/DebugArtifacts.jl`

julia> using DebugArtifacts
[ Info: Precompiling DebugArtifacts [4f05f810-3073-4646-a23d-fd50056b1323]

julia>

julia> debug_artifact("MozillaCACerts")
[ Info: Platform: Linux x86_64 {cxxstring_abi=cxx11, julia_version=1.7.2, libc=glibc, libgfortran_version=5.0.0,libstdcxx_version=3.4.29}
Julia Version 1.7.2
Commit bf53498635 (2022-02-06 15:21 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: AMD Ryzen Threadripper 3960X 24-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, znver2)

[ Info: Downloading Artifacts.toml to /tmp/jl_yLvn75/Artifacts.toml...
ERROR: Error reading ca cert path /usr/lib/ssl/certs - mbedTLS: (-0x2900) X509 - Read/write of file failed whilerequesting https://raw.githubusercontent.com/JuliaBinaryWrappers/MozillaCACerts_jll.jl/master/Artifacts.toml
Stacktrace:
  [1] (::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Downloads.var"#24#27"{Pkg.PlatformEngines.var"#15#17"{Base.TTY}}, Bool, Bool, String, Int64, Bool, Bool})(easy::Downloads.Curl.Easy)
    @ Downloads ~/src/julia/usr/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:369
  [2] with_handle(f::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Downloads.var"#24#27"{Pkg.PlatformEngines.var"#15#17"{Base.TTY}}, Bool, Bool, String, Int64, Bool, Bool}, handle::Downloads.Curl.Easy)
    @ Downloads.Curl ~/src/julia/usr/share/julia/stdlib/v1.7/Downloads/src/Curl/Curl.jl:64
  [3] #8
    @ ~/src/julia/usr/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:311 [inlined]
  [4] arg_write(f::Downloads.var"#8#17"{Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Downloads.var"#24#27"{Pkg.PlatformEngines.var"#15#17"{Base.TTY}}, Bool, Bool, String, Int64, Bool, Bool}, arg::IOStream)
    @ ArgTools ~/src/julia/usr/share/julia/stdlib/v1.7/ArgTools/src/ArgTools.jl:112
  [5] #7
    @ ~/src/julia/usr/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:310 [inlined]
  [6] arg_read(f::Downloads.var"#7#16"{IOStream, Nothing, Vector{Pair{String, String}}, Float64, Downloads.var"#24#27"{Pkg.PlatformEngines.var"#15#17"{Base.TTY}}, Bool, Bool, String, Int64, Bool, Bool}, arg::Base.DevNull)
    @ ArgTools ~/src/julia/usr/share/julia/stdlib/v1.7/ArgTools/src/ArgTools.jl:61
  [7] request(url::String; input::Nothing, output::IOStream, method::Nothing, headers::Vector{Pair{String, String}}, timeout::Float64, progress::Pkg.PlatformEngines.var"#15#17"{Base.TTY}, verbose::Bool, throw::Bool, downloader::Nothing)
    @ Downloads ~/src/julia/usr/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:309
  [8] #3
    @ ~/src/julia/usr/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:222 [inlined]
  [9] open(f::Downloads.var"#3#4"{Nothing, Vector{Pair{String, String}}, Float64, Pkg.PlatformEngines.var"#15#17"{Base.TTY}, Bool, Nothing, String}, args::String; kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:write,), Tuple{Bool}}})
    @ Base ./io.jl:330
 [10] arg_write(f::Function, arg::String)
    @ ArgTools ~/src/julia/usr/share/julia/stdlib/v1.7/ArgTools/src/ArgTools.jl:86
 [11] #download#2
    @ ~/src/julia/usr/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:221 [inlined]
 [12] download(url::String, dest::String; verbose::Bool, headers::Vector{Pair{String, String}}, auth_header::Nothing, io::Base.TTY)
    @ Pkg.PlatformEngines ~/src/julia/usr/share/julia/stdlib/v1.7/Pkg/src/PlatformEngines.jl:282
 [13] (::DebugArtifacts.var"#3#4"{String, Base.BinaryPlatforms.Platform, String})(tmp_dir::String)
    @ DebugArtifacts ~/src/DebugArtifacts.jl/src/DebugArtifacts.jl:59
 [14] mktempdir(fn::DebugArtifacts.var"#3#4"{String, Base.BinaryPlatforms.Platform, String}, parent::String; prefix::String)
    @ Base.Filesystem ./file.jl:750
 [15] mktempdir (repeats 2 times)
    @ ./file.jl:748 [inlined]
 [16] debug_artifact(artifact_name::String, platform::Base.BinaryPlatforms.Platform)
    @ DebugArtifacts ~/src/DebugArtifacts.jl/src/DebugArtifacts.jl:55
 [17] debug_artifact(artifact_name::String)
    @ DebugArtifacts ~/src/DebugArtifacts.jl/src/DebugArtifacts.jl:43
 [18] top-level scope
    @ REPL[3]:1

I don’t know if this gives any more information.

3

Isn’t this information enough?
Have you checked this folder? Can you access it, read it as normal user?

4

I have checked the folder, can access it, can open files as the same user that runs julia. I don’t know what file it’s trying to access in there.

5

It needs the root certificates to check the validity of the servers certificates.

6

looking in more details at that folder, there where some broken symlinks in there (of the form auth.mymachineFQDN.pem pointing to /usr/local/share/ca-certificates/auth.mymachineFQDN.cert).

I moved those out of the way, and it seems to be working better. Not sure where those files where coming from, don’t know much about certificates.

Thanks for the hints anyway.

7

You can try update-ca-certificates as root

8

For anyone that runs into something similar, it looks like the links are made by the update-ca-certificates script in the ca-certificates package, that seems to be an infrastructure for packages to add some certificates. In my case, it seemed to have a stale link to a certificate from a jitsi-meet client I installed and uninstalled, that didn’t clean up correctly.

Not sure why one broken link in there impacts all the rest, though.

1 Like
9

ninja’d :wink:

10

:stuck_out_tongue:, thanks for the help!

1 Like
11

The individual CA files are usually processed into a single PEM file that is used by tools that need to verify certificates. It’s possible that file was missing or broken either because of the broken symlink or in addition to that. I suspect that running update-ca-certificates fixed that file. Unfortunately I can’t tell you the exact name because there is no standard that’s consistent across all distros of Linux or even different versions of the same distro.

1 Like
12

from the man page, under Debian, it seems to be ca-certificates.crt.
Note that simply moving the broken symlinks out of the way fixed the issue for Julia. I’m not sure what the mbedTLS library that seems to be used by Julia needs.
After discovering the origin of those symlinks, I ran the update-ca-certificates script as root. The ca-certificates.crt timestamp seems to have been updated today a little while ago, so at some point it has been recreated, either when I ran the script, or automagically by the mbedTLS lib, not sure exactly which one of my actions was the direct cause of that.

13

MbedTLS does not create that file, it only reads it. It seems very likely that it was created when you ran the update-ca-certificates script.

14

Yep, what I found weird is that it started working as soon as I deleted the broken symlinks. I just tried again, I recreated the symlinks, and it’s failing again. The ca-certificates.crt file didn’t change. So it looks like I had a valid file. And simply the presence of the weird symlink gets it to fail.

For some reproducible julia cert fun:

/etc/ssl/certs/ $ sudo ln -s ../broken.pem auth.broken.pem

in julia (clean installation, without a ~/.julia directory):

> ]
> add Plots

and the error appears.