I’ve just made myself a private registry for my work packages, following the nice instructions here. My organisation uses private github for most of our code, so I’ve put it there. When I try to do a package update, Pkg tells me it “failed to fetch from repo”. But I can successfully fetch if I navigate to the directory in ~/.julia/registries and do git fetch.
Is Pkg not using my ssh credentials? Should I expect this to work?
From looking at the discussions around this topic, it seems like a user guide would be extremely useful for private registries, and some common edge cases. I would volunteer to do all the requisite learning and writing except I’m already at least two levels deep into “I probably shouldn’t spend time on this but it’s interesting” work.
I see. That’s going to a problem, as our security set up doesn’t allow for anonymous access to our private repos. I guess I should raise an issue against Pkg.
We authenticate git commands using ssh keys only. This appears to be what happens when you insist on 2fa for all accounts in your organisation. https always fails, even if you enter username and password correctly. And Pkg never prompts for username and password.
I had read this to mean “Pkg always uses HTTPS access”, implying that ssh wasn’t possible. That’s why I was expecting Pkg to prompt for a username/password.
Your suggestion to use helper = store works for when I’m off-network, but not when I’m on-network. It seems that Pkg isn’t using my ssh config. Whilst I’m on the network, I am able to fetch the registry from the command line whilst in ~/.julia/registries/my-registry/
Do you know if that’s the expected behaviour?
So, I went digging around in the source for Pkg, and followed things all the way down into the libgit2 C library. At this stage, I couldn’t figure out which calls did what, and ran out of time to investigate further.
A bit more detail about the on-network case which is the problem: We use a socks proxy for ssh traffic leaving our network. This is wrapped in a call to netcat, specified in one’s ssh config file. It seems that libgit2, and be extension Pkg, does not look at ssh config by default. A StackOverflow answer, here, says:
Reading config settings from your OpenSSH config file at ~/.ssh/config isn’t supported by libgit2 because it isn’t support by libssh2. If you want to read settings from there, you have to do it yourself.
If this is accurate, then someone will need to add support for ssh config files into Pkg, or maybe LibGit2.jl.
I’d be happy to hear any more ideas about how to achieve this, but it seems likely that the answer is to raise an issue to request this feature, and probably implement it myself if I want it quickly.
In addition to the SSH key, I need to read a ProxyCommand from the ssh config. So that earlier discussion doesn’t quite do enough.
Your suggestion prompted me to look up the specific issue of doing ProxyCommand-type things when using programs which use libssh2. I haven’t found a solution yet, but it might be useful.
I’m also hitting this issue. Building a Docker image with a julia project with reference to private repos. The SSH_KEY_PATH/SSH_PUB_KEY_PATH workaround does not work.
The error message is:
error: GitError(Code:EUSER, Class:Callback, Aborting, user cancelled credential request.)
on
[1] pkgerror(::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Pkg/src/Types.jl:112
[2] #clone#4(::Nothing, ::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:isbare, :credentials),Tuple{Bool,LibGit2.CachedCredentials}}}, ::typeof(Pkg.GitTools.clone), ::String, ::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Pkg/src/GitTools.jl:128
[3] shred!(::getfield(Pkg.Types, Symbol("##79#81")){String,String}, ::LibGit2.CachedCredentials) at ./none:0
using Julia v1.2.
This error occurs only when building a Docker image. On a running server with Jupyter, registering the ssh key with ssh-add makes Pkg.add work adding private repos without asking for user password.
If this happens in docker there should be good chances to reproduce it elsewhere. Do you get the same problem (works outside docker but not inside) if you try to access one of your github repositories with an ssh url? Can you make a minimal Dockerfile that reproduces the problem?
I was able to solve the issue. In this template, PrivateSource.jl is the main project and PrivateDep.jl is a dependency. Both are cloned from a private repo. In this case I used gitlab.com .
The file id_rsa_gitlab was generated by the following command (without password):