Accesing a Registry on Private Github

I’ve just made myself a private registry for my work packages, following the nice instructions here. My organisation uses private github for most of our code, so I’ve put it there. When I try to do a package update, Pkg tells me it “failed to fetch from repo”. But I can successfully fetch if I navigate to the directory in ~/.julia/registries and do git fetch.
Is Pkg not using my ssh credentials? Should I expect this to work?

From looking at the discussions around this topic, it seems like a user guide would be extremely useful for private registries, and some common edge cases. I would volunteer to do all the requisite learning and writing except I’m already at least two levels deep into “I probably shouldn’t spend time on this but it’s interesting” work.

AFAIK Pkg uses HTTPS access, and for Github/ssh you want the git@... protocol. Just change the remote for origin.

I see. That’s going to a problem, as our security set up doesn’t allow for anonymous access to our private repos. I guess I should raise an issue against Pkg.

I don’t see what “anonymous” means in this context. How do you usually authenticate to Github for your own repos (regardless of Julia)?

We authenticate git commands using ssh keys only. This appears to be what happens when you insist on 2fa for all accounts in your organisation. https always fails, even if you enter username and password correctly. And Pkg never prompts for username and password.

If you use ssh keys, why should it need a password?

In any case, I have private repos on Github working fine with Pkg, but perhaps organizational setups are different. My ~/.gitconfig has

[github]
        user = my_github_username
[credential]
        helper = store

I had read this to mean “Pkg always uses HTTPS access”, implying that ssh wasn’t possible. That’s why I was expecting Pkg to prompt for a username/password.

Your suggestion to use helper = store works for when I’m off-network, but not when I’m on-network. It seems that Pkg isn’t using my ssh config. Whilst I’m on the network, I am able to fetch the registry from the command line whilst in ~/.julia/registries/my-registry/
Do you know if that’s the expected behaviour?

I am an not an expert on Pkg internals, but first I would make sure everything works as it should just using the git command line.

1 Like

OK, thanks Tamas. Seems like git is working just fine. Hopefully someone who knows more about Pkg and Registries will have a look!

Whats the remote origin url of the registry? Is it https or git@? Have you tried Pkg.setprotocol!("ssh")?

The remote origin, using git remote -v is git@. However, in Pkg.Registry.status() it’s https.

Pkg.setprotocol! doesn’t seem to make a difference to the result of Pkg.update().

1 Like

So, I went digging around in the source for Pkg, and followed things all the way down into the libgit2 C library. At this stage, I couldn’t figure out which calls did what, and ran out of time to investigate further.

A bit more detail about the on-network case which is the problem: We use a socks proxy for ssh traffic leaving our network. This is wrapped in a call to netcat, specified in one’s ssh config file. It seems that libgit2, and be extension Pkg, does not look at ssh config by default. A StackOverflow answer, here, says:

Reading config settings from your OpenSSH config file at ~/.ssh/config isn’t supported by libgit2 because it isn’t support by libssh2. If you want to read settings from there, you have to do it yourself.

If this is accurate, then someone will need to add support for ssh config files into Pkg, or maybe LibGit2.jl.

I’d be happy to hear any more ideas about how to achieve this, but it seems likely that the answer is to raise an issue to request this feature, and probably implement it myself if I want it quickly.

Is this earlier discussion helpful? How to specify the SSH key for Pkg to use

In addition to the SSH key, I need to read a ProxyCommand from the ssh config. So that earlier discussion doesn’t quite do enough.

Your suggestion prompted me to look up the specific issue of doing ProxyCommand-type things when using programs which use libssh2. I haven’t found a solution yet, but it might be useful.

I’m also hitting this issue. Building a Docker image with a julia project with reference to private repos. The SSH_KEY_PATH/SSH_PUB_KEY_PATH workaround does not work.

The error message is:

error: GitError(Code:EUSER, Class:Callback, Aborting, user cancelled credential request.)

on

[1] pkgerror(::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Pkg/src/Types.jl:112
 [2] #clone#4(::Nothing, ::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:isbare, :credentials),Tuple{Bool,LibGit2.CachedCredentials}}}, ::typeof(Pkg.GitTools.clone), ::String, ::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Pkg/src/GitTools.jl:128
 [3] shred!(::getfield(Pkg.Types, Symbol("##79#81")){String,String}, ::LibGit2.CachedCredentials) at ./none:0

using Julia v1.2.

This error occurs only when building a Docker image. On a running server with Jupyter, registering the ssh key with ssh-add makes Pkg.add work adding private repos without asking for user password.

If this happens in docker there should be good chances to reproduce it elsewhere. Do you get the same problem (works outside docker but not inside) if you try to access one of your github repositories with an ssh url? Can you make a minimal Dockerfile that reproduces the problem?

1 Like

Yes, I was working on that.

I was able to solve the issue. In this template, PrivateSource.jl is the main project and PrivateDep.jl is a dependency. Both are cloned from a private repo. In this case I used gitlab.com .

The file id_rsa_gitlab was generated by the following command (without password):

ssh-keygen -t rsa -b 4096 -C "felipenoris@gmail.com"

The following is the template Dockerfile.


FROM centos:7

MAINTAINER Felipe Noronha <felipenoris@gmail.com>

WORKDIR /root

RUN yum update -y && yum install -y epel-release && yum clean all
RUN yum update -y && yum install -y \
    curl-devel \
    libcurl \
    libcurl-devel \
    openssl \
    openssl098e \
    openssl-devel \
    git \
    wget \
    && yum clean all \
    && rm -rf /var/cache/yum/*

ENV PATH /usr/local/sbin:/usr/local/bin:$PATH
ENV LD_LIBRARY_PATH /usr/local/lib:/usr/local/lib64

# Julia - https://julialang.org/downloads/
ENV JULIA_VER_MAJ 1.2
ENV JULIA_VER_MIN .0
ENV JULIA_VER $JULIA_VER_MAJ$JULIA_VER_MIN

RUN wget https://julialang-s3.julialang.org/bin/linux/x64/$JULIA_VER_MAJ/julia-$JULIA_VER-linux-x86_64.tar.gz \
        && mkdir /usr/local/julia \
        && tar xf julia-$JULIA_VER-linux-x86_64.tar.gz --directory /usr/local/julia --strip-components=1 \
        && ln -s /usr/local/julia/bin/julia /usr/local/bin/julia \
        && rm -f julia-$JULIA_VER-linux-x86_64.tar.gz

ENV TERM xterm

COPY id_rsa_gitlab /root/.ssh/id_rsa_gitlab

RUN chmod 600 /root/.ssh/id_rsa_gitlab

RUN eval $(ssh-agent -s) && \
    ssh-add /root/.ssh/id_rsa_gitlab && \
    ssh-keyscan gitlab.com >> /root/.ssh/known_hosts && \
    git clone --depth=1 --single-branch --branch master git@gitlab.com:felipenoris/PrivateSource.jl.git && \
    julia --project=PrivateSource.jl -e 'using Pkg; Pkg.instantiate(); using PrivateSource'

CMD julia --project=PrivateSource.jl -e 'import PrivateSource; PrivateSource.greet()'

To build the image, I used a AWS server running centos7 with docker (ami-02e5c6d9065871fb7).

The following are the final steps of the build process (docker build -t julia-docker):

Step 12/16 : ENV TERM xterm
 ---> Running in 587facc1249e
Removing intermediate container 587facc1249e
 ---> 72c5491f2e55
Step 13/16 : COPY id_rsa_gitlab /root/.ssh/id_rsa_gitlab
 ---> 30e5d67049c0
Step 14/16 : RUN chmod 600 /root/.ssh/id_rsa_gitlab
 ---> Running in 55e3608f8bc4
Removing intermediate container 55e3608f8bc4
 ---> db1f0943878a
Step 15/16 : RUN eval $(ssh-agent -s) &&     ssh-add /root/.ssh/id_rsa_gitlab &&     ssh-keyscan gitlab.com >> /root/.ssh/known_hosts &&     git clone --depth=1 --single-branch --branch master git@gitlab.com:felipenoris/PrivateSource.jl.git &&     julia --project=PrivateSource.jl -e 'using Pkg; Pkg.instantiate(); using PrivateSource'
 ---> Running in 308d8b59e2f7
Agent pid 8
Identity added: /root/.ssh/id_rsa_gitlab (felipenoris@gmail.com)
# gitlab.com:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
# gitlab.com:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
# gitlab.com:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
Cloning into 'PrivateSource.jl'...
Warning: Permanently added the ECDSA host key for IP address '35.231.145.151' to the list of known hosts.
   Cloning git-repo `git@gitlab.com:felipenoris/PrivateDep.jl.git`
  Updating git-repo `git@gitlab.com:felipenoris/PrivateDep.jl.git`
Removing intermediate container 308d8b59e2f7
 ---> c5d5ea9adc12
Step 16/16 : CMD julia --project=PrivateSource.jl -e 'import PrivateSource; PrivateSource.greet()'
 ---> Running in 2b90aa2372bd
Removing intermediate container 2b90aa2372bd
 ---> a63915d18507
Successfully built a63915d18507
Successfully tagged julia-docker:latest