TLS host certificate problem with private registry

General Usage
,
1

Hello,

My colleague and I are using a private registry via LocalRegistry.jl. We are using ecdsa key to ssh. Everything works on my end. However, it does not work for my colleague . Although my colleague can clone from the repository, she cannot use the registry. When she updates the dependencies of her project, it throws this message:

TLS host verification: the identity of the server gitlab.removed_for_privacy l could not be verified. Someone could be trying to man-in-the-middle your connection. It is also possible that the correct server is using an invalid certificate or that your system’s certificate authority root store is misconfigured.
ERROR: failed to clone from https://removed_for_privacy, error: GitError(Code:ERROR, Class:HTTP, user cancelled certificate check)

I’m not sure what could be causing this issue. Any guidance would be much appreciated.

Thanks!

2

I do remember seeing someone with a similar-sounding problem here.

3

Thank you. Sorry for the noise. I did find that too and intended to ask whether there are any issues with using that approach? My naïve guess is that it disables some type of security feature.

4

Yes - disabling host checks does open you up to man-in-the-middle attacks. Not a good permanent solution. Sometimes it has to do with proxies, firewalls, etc. … and may require a bit of sleuthing to figure out. See if the problem persisits outside of Julia (i.e., try to connect through some different means, such as curl if you are on a Unix-y platform).

5

Perhaps this might be helpful? Tim Holy’s writeup on remote registries

6

Thanks for your reply. Good to know about the issues with changing the environment variable. We tried it anyways, but it did not work. It asked for a token/password.

I forgot to note that we are using Windows, which I suspect increases the likelihood of issues.

Thanks for the link to Tim Holy’s write up. I will check that out and see what I can understand.

7

The error message looks like the private registry was initially cloned over HTTPS. Your colleague may need to remove + re-add the private registry so that git associates the SSH protocol to its url e.g.,

using Pkg
Pkg.Registry.rm("CustomRegistry")
Pkg.Registry.add(Pkg.RegistrySpec(name="CustomRegistry", url="ssh://git@gitlab.customdomain.com/path/to/registry.git"))
2 Likes
8

Thank you. We will try your proposed solution and report back.

9

We tried your proposed solution but it did not work unfortunately. I’m not sure if this is relevant, but it only initialized the registry once the ssh:// was removed from the url keyword.

We also ruled out firewall issues at least on the local machine. Very perplexing.

10

Troubleshooting network crypto is nightmarish.
To help you more we will likely need your verbatim error text.
Also, you can try out some command line tools like those mentioned here, or something like:

openssl s_client -connect localhost:30001

What we really need to isolate here is if it is a networking problem or a Julia networking problem.

11

Thanks. I will checkout those tools.

Unfortunately, I believe what i posted is the full error message. I only changed the urls for privacy concerns. I am confirming with my colleague.

Please let me know if there is a good way to distinguish between networking vs Julia networks problems.

12

I’d say if you can connect with command line tools, etc., some other programming language with TLS, then you can rule out networking in general. After that, you have to start peeling back the layers in your Julia code, perhaps with a debugger?

Also, if you can run something like Wireshark, that may help (but adding crypto to Wireshark makes life more difficult - but there are tutorials).

1 Like
13

It might also be cached credentials. If you are using ssh, for instance, and you connect to a server - it often asks you if you trust the server (text from here):

The authenticity of host 'mint.phcomp.co.uk (78.32.209.33)' can't be established.
RSA key fingerprint is 6a:de:e0:af:56:f8:0c:04:11:5b:ef:4d:49:ad:09:23.
Are you sure you want to continue connecting (yes/no)? no

If you re-generate the server key, and reconnect to the same IP address, you will get an error like this:

error msg: 
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
6a:de:e0:af:56:f8:0c:04:11:5b:ef:4d:49:ad:09:23.
Please contact your system administrator.
Update the SSHFP RR in DNS with the new host key to get rid of this message.
The authenticity of host 'freshmint.phcomp.co.uk (2001:4d48:ad51:2f00::2:2)' can't be established.
RSA key fingerprint is 6a:de:e0:af:56:f8:0c:04:11:5b:ef:4d:49:ad:09:23.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)?

The fix is to clear out the cache of server keys, usually stored in ~/.ssh/ … for Julia it may be somewhere else, like ~/.julia somewhere. You might have to grep around in the source for your modules.

1 Like