Pkg fails to clone with ssh

ethomag · May 15, 2020, 1:45pm

Not sure if this is Julia related, but I have run out of options

In our company we use gerrit for git access, but I have random problems retrieving packets from within julia package manager (sometimes it works just fine):

(@v1.4) pkg> dev ssh://user@gerrit.company.com:29418/julia/MyPackage.git
    Cloning git-repo `ssh://user@gerrit.company.com:29418/julia/MyPackage.git`
ERROR: failed to clone from ssh://user@gerrit.company.com:29418/julia/MyPackage.git, error: GitError(Code:ERROR, Class:SSH, error authenticating: agent list id failed)

Quitting Julia and doing the same operation from the shell, works fine:

$ git clone ssh://user@gerrit.company.com:29418/julia/MyPackage.git
Cloning into 'MyPackage'...
remote: Total 430 (delta 0), reused 430 (delta 0)
Receiving objects: 100% (430/430), 117.36 KiB | 11.74 MiB/s, done.
Resolving deltas: 100% (314/314), done.

From the error code (error authenticating: agent list id failed), it seems that there is a problem with the client ? Does Pkg use something other than plain git

kristoffer.carlsson · May 15, 2020, 2:18pm

Yes, it uses LibGit2.

ethomag · May 15, 2020, 2:20pm

Do you think that error authenticating: agent list id failed is related to the client ?

ethomag · May 17, 2020, 3:16pm

If Pkg is using LibGit2, could it be that there is some configuration is e.g. ~/.ssh that is read by git executable, but LibGit2 does not read ?

GunnarFarneback · May 17, 2020, 3:30pm

The continuation of the story is that LibGit2 gets its ssh functionality from libssh2, which indeed is less capable of taking the ~/.ssh configurations into account than the ssh command line interface used by the git command line interface.

ethomag · May 18, 2020, 9:28am

Stepping through the code in a debugger, I can see that in libssh2, when sending SSH2_AGENTC_REQUEST_IDENTITIES, the expected response is SSH2_AGENT_IDENTITIES_ANSWER (12), but I get the response 5 which I think corresponds to SSH_AGENT_FAILURE.

Capturing the traffic with tshark, I can see a difference between cloning from within Julia package manager (pkg> dev ssh://user@gerrit.company.com:29418/julia/MyPackage):

    5 0.004021196 10.68.82.138 → 10.221.32.23 SSH 79 Client: Protocol (SSH-2.0-libssh2_1.9.0)
    6 0.005154503 10.221.32.23 → 10.68.82.138 SSHv2 126 Server: Protocol (SSH-2.0-GerritCodeReview_2.14.20-179-g6c46f4efba (SSHD-CORE-1.6.0.2))
   10 0.005629378 10.68.82.138 → 10.221.32.23 SSHv2 232 Client: Key Exchange Init
   11 0.006036971 10.221.32.23 → 10.68.82.138 SSHv2 696 Server: Key Exchange Init
   13 0.007231360 10.68.82.138 → 10.221.32.23 SSHv2 80 Client: Diffie-Hellman Group Exchange Request
   14 0.030877970 10.221.32.23 → 10.68.82.138 SSHv2 272 Server: Diffie-Hellman Group Exchange Group
   15 0.034811399 10.68.82.138 → 10.221.32.23 SSHv2 264 Client: Diffie-Hellman Group Exchange Init
   16 0.041050309 10.221.32.23 → 10.68.82.138 SSHv2 568 Server: Diffie-Hellman Group Exchange Reply
   17 0.045080774 10.68.82.138 → 10.221.32.23 SSHv2 72 Client: New Keys
   18 0.046964505 10.221.32.23 → 10.68.82.138 SSHv2 72 Server: New Keys
   19 0.047070611 10.68.82.138 → 10.221.32.23 SSHv2 120 Client: Encrypted packet (len=64)
   20 0.049089173 10.221.32.23 → 10.68.82.138 SSHv2 136 Server: Encrypted packet (len=80)
   21 0.049160209 10.68.82.138 → 10.221.32.23 SSHv2 136 Client: Encrypted packet (len=80)
   22 0.051449152 10.221.32.23 → 10.68.82.138 SSHv2 136 Server: Encrypted packet (len=80)

It stops there. And cloning from the shell ($ git clone ssh://user@gerrit.company.com:29418/julia/MyPackage)

    5 0.004378099 10.68.82.138 → 10.221.32.22 SSH 97 Client: Protocol (SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3)
    7 0.007004763 10.221.32.22 → 10.68.82.138 SSHv2 126 Server: Protocol (SSH-2.0-GerritCodeReview_2.14.20-179-g6c46f4efba (SSHD-CORE-1.6.0.2))
    9 0.007123218 10.221.32.22 → 10.68.82.138 SSHv2 696 Server: Key Exchange Init
   11 0.007402211 10.68.82.138 → 10.221.32.22 SSHv2 1416 Client: Key Exchange Init
   13 0.009055333 10.68.82.138 → 10.221.32.22 SSHv2 136 Client: Elliptic Curve Diffie-Hellman Key Exchange Init
   14 0.012034598 10.221.32.22 → 10.68.82.138 SSHv2 448 Server: Elliptic Curve Diffie-Hellman Key Exchange Reply
   15 0.012480975 10.68.82.138 → 10.221.32.22 SSHv2 72 Client: New Keys
   16 0.014322358 10.221.32.22 → 10.68.82.138 SSHv2 72 Server: New Keys
   18 0.124765585 10.68.82.138 → 10.221.32.22 SSHv2 120 Client: Encrypted packet (len=64)
   19 0.126596552 10.221.32.22 → 10.68.82.138 SSHv2 136 Server: Encrypted packet (len=80)
   21 0.126652342 10.68.82.138 → 10.221.32.22 SSHv2 136 Client: Encrypted packet (len=80)
   22 0.128500266 10.221.32.22 → 10.68.82.138 SSHv2 136 Server: Encrypted packet (len=80)
   23 0.128580788 10.68.82.138 → 10.221.32.22 SSHv2 440 Client: Encrypted packet (len=384)
   24 0.130603233 10.221.32.22 → 10.68.82.138 SSHv2 408 Server: Encrypted packet (len=352)
   25 0.132204724 10.68.82.138 → 10.221.32.22 SSHv2 712 Client: Encrypted packet (len=656)
   26 0.136491084 10.221.32.22 → 10.68.82.138 SSHv2 120 Server: Encrypted packet (len=64)
   27 0.136588551 10.68.82.138 → 10.221.32.22 SSHv2 136 Client: Encrypted packet (len=80)
   28 0.138405907 10.221.32.22 → 10.68.82.138 SSHv2 136 Server: Encrypted packet (len=80)
[ ... ]

But, I don’t know what to do with this information - any pointers on what to try or ideas to trace or log something are welcome!

GunnarFarneback · May 18, 2020, 9:48am

You may get some more detail from server or client logs of the ssh communication.

If you have control of the server end, or a helpful admin, you can try to set LogLevel in /etc/ssh/sshd_config to some debug level and restart the ssh demon. Then you should get lots of traces in /var/log/auth.log(assuming a Linux server, otherwise I don’t know where it ends up).

If you can’t get logs from the server they can be obtained from the client, but this is more tricky. See https://github.com/JuliaLang/Pkg.jl/issues/1516#issuecomment-560794389.

ethomag · May 18, 2020, 11:35am

Thanks Gunnar!

Unfortunately I don’t have access to the server and I doubt there are any helpful admins here .

Following your advice (however, I did not install libmbedtls-dev since I didn’t know what to do with it), I get a long printout on stderr, which starts as:

(@v1.4) pkg> dev ssh://user@gerrit.company.com:29418/julia/MyPackage
    Cloning git-repo `ssh://user@gerrit.company.com:29418/julia/MyPackage.git`
[libssh2] 0.161669 Transport: New session resource allocated
[libssh2] 0.161738 Transport: session_startup for socket 18
[libssh2] 0.161764 Transport: Sending Banner: SSH-2.0-libssh2_1.9.0_DEV
[libssh2] 0.161807 Socket: Sent 27/27 bytes at 0x7face1d8412d+0
[libssh2] 0.161827 Socket: Error recving 1 bytes: 11
[libssh2] 0.163657 Socket: Recved 1 bytes banner
[libssh2] 0.163674 Socket: Recved 1 bytes banner

Many of those - is that a problem ?

Then, some logs that looks “normal” (i e no errors or warnings). And ends with:

[libssh2] 0.210741 Transport: Requesting userauth service
[libssh2] 0.210776 Socket: Sent 64/64 bytes at 0x2f92c38
[libssh2] 0.210780 Transport: Looking for packet of type: 6
[libssh2] 0.212514 Socket: Recved 80/16384 bytes to 0x2f8ebf8+0
[libssh2] 0.212536 Transport: Packet type 6 received, length=17
[libssh2] 0.212540 Transport: Looking for packet of type: 6
[libssh2] 0.212546 Conn: Setting blocking mode ON
[libssh2] 0.212567 Socket: Sent 80/80 bytes at 0x2f92c38
[libssh2] 0.212585 Transport: Looking for packet of type: 52
[libssh2] 0.212589 Transport: Looking for packet of type: 51
[libssh2] 0.212597 Failure Event: -37 - Would block requesting userauth list
[libssh2] 0.214365 Transport: Looking for packet of type: 52
[libssh2] 0.214376 Transport: Looking for packet of type: 51
[libssh2] 0.214383 Socket: Recved 80/16384 bytes to 0x2f8ebf8+0
[libssh2] 0.214398 Transport: Packet type 51 received, length=15
[libssh2] 0.214403 Transport: Looking for packet of type: 52
[libssh2] 0.214410 Transport: Looking for packet of type: 51
[libssh2] 0.214417 Userauth: Permitted auth methods: publickey
[libssh2] 0.230308 Failure Event: -42 - agent list id failed
[libssh2] 0.230368 Transport: Freeing session resource
[libssh2] 0.230384 Transport: Extra packets left 0
[libssh2] 0.230389 Transport: unable to reset socket's blocking state
ERROR: failed to clone from ssh://user@gerrit.company.com:29418/julia/MyPackage.git, error: GitError(Code:ERROR, Class:SSH, error authenticating: agent list id failed)

Still, I’m not competent enough to deduce any useful information.

It just struck me that the following might be useful to someone knowledgeable in ssh (which I cut out in the above):

[libssh2] 0.164176 Key Ex: Agreed on KEX method: ecdh-sha2-nistp256
[libssh2] 0.164180 Key Ex: Agreed on HOSTKEY method: ssh-rsa
[libssh2] 0.164183 Key Ex: Agreed on CRYPT_CS method: aes128-ctr
[libssh2] 0.164187 Key Ex: Agreed on CRYPT_SC method: aes128-ctr
[libssh2] 0.164191 Key Ex: Agreed on MAC_CS method: hmac-sha2-256
[libssh2] 0.164194 Key Ex: Agreed on MAC_SC method: hmac-sha2-256
[libssh2] 0.164197 Key Ex: Agreed on COMP_CS method: none
[libssh2] 0.164200 Key Ex: Agreed on COMP_SC method: none
[libssh2] 0.164305 Key Ex: Initiating ECDH SHA2 NISTP256

Again, any pointers are greatly appreciated!

StefanKarpinski · May 18, 2020, 1:44pm

Not helpful at the moment (sorry), but I’m working on reengineering the way Pkg gets resources (registries, packages, artifacts) and I’m planning on preferentially using a command-line git client if there’s one in your PATH so that this isn’t an issue. It’s so frustrating when git is perfectly able to access something but Pkg can’t because it’s using libgit2 which does things differently.

GunnarFarneback · May 18, 2020, 6:08pm

Sorry, this does not look similar to the cases I’ve been able to diagnose in the past, so can’t be of much assistance.

If you want to look into workarounds, here are some possibilities with varying complexity and drawbacks:

git clone outside of Julia and pkg> add or pkg> dev by path. Simple but fiddly.
If your git server allows https access, switch to such URLs. In some situations git insteadOf configuration can be helpful to automate this.
Use the LocalRegistry package to organize your local packages and the LocalPackageServer package to distribute them with the new PkgServer functionality of Julia 1.4. Disclaimer, I’ve written those packages. Whereas LocalRegistry seems quite stable, LocalPackageServer is very much on the bleeding edge and wasn’t registered until 20 minutes ago.

ethomag · May 19, 2020, 4:39am

Great! Looking forward to your re-engineered Pkg!

FYI: To make things work with our local registry, we put the following lines in .julia/config/startup.jl

import Pkg
Pkg.GitTools.setprotocol!(domain="gerrit.company.com:29418", protocol="ssh", user="user")

So, I would appreciate that any new git-client would honor these settings, or introduce some other mechanism to specify these.

ethomag · May 19, 2020, 4:51am

Thanks again Gunnar for not giving up on me

I think we’ll go for the manual git clone, when this happens.

I already use your super-handy LocalRegistry package to manage our in-company package registry. Thanks for that one, too.

I could set up an internal github-like package server, but I’m afraid I’ll be swamped with uninteresting work (such as this). So I prefer to let our company experts deal with these kind of issues and store our Julia packages at a managed site.

GunnarFarneback · May 19, 2020, 5:55am

That might be a misunderstanding. The Julia package servers are not at all github-like. What they do is basically to cache and serve gzipped tar archives of packages. Moreover the LocalPackageServer package can fetch local packages dynamically from their git sources, using the standard git client, and send them over to Julia’s package manager as tarballs. Actually, if you just run this locally on your own computer it’s effectively rather similar to having Pkg use git instead of LibGit2, in particular if I add an option not to cache tarballs.

ethomag · May 19, 2020, 6:04am

Aha. That really sounds like another option, then. Thanks again, Gunnar!