Use private registry in GitLab CI job

General Usage
1

I have setup a GitLab CI Job to test a package I wrote - it works.
Now I want to test a second package that depends on the first - private - package.
I have made a private regsitry that I can use to add the package on my local machine.

What I have so far

- julia --project=@. -e 'import LibGit2, Pkg;
            user_regs = joinpath(DEPOT_PATH[1], "registries");
            all_registries = Dict(
                "General" => "https://github.com/JuliaRegistries/General.git",
                "CoolRegistry" => "https://<cool-registry>.jl");
            creds_user = ENV["USER"];
            creds_pass = Base.SecretBuffer(ENV["PASS"]);
            Base.shred!(LibGit2.UserPasswordCredential(creds_user, creds_pass)) do creds;
                for (reg, url) in all_registries;
                    path = joinpath(user_regs, reg);
                    Pkg.GitTools.clone(url, path; credentials = creds);
                end;
            end;
            Pkg.test(; coverage = true);'

I managed to to clone both registries but I am not able to use the packages inside my registry.
All repo paths in the registry are defined via an SSH link.
After a few seconds I just get a timeout.

Is there an easier option for the above?
Can I pass the package manager the SSH key file?

2

If I just use

- julia --project=@. -e 'import Pkg;
            Pkg.Registry.add(Pkg.RegistrySpec(url = "ssh://git@<cool-regitstry>.jl.git"));
            Pkg.test(; coverage = true);'

I get a timeout, even though I specified the SSH_KEY_PATH and SSH_PUB_KEY_PATH as environment variables.

3

I wouldn’t worry about registries until you know that you have the credentials working for just adding a package over ssh. It’s hard to give advice and all CI are different, but the following works for me in a Jenkins + Bitbucket environment:

                withCredentials([sshUserPrivateKey(credentialsId: "Deploy",
                                 keyFileVariable: 'keyfile'),
                                 file(credentialsId: "DeployPublic",
                                 variable: 'pubkeyfile')]) {
                    sh '''export SSH_KEY_PATH=${keyfile}
                          export SSH_PUB_KEY_PATH=${pubkeyfile}
                          julia --project=julia/test julia/test/jenkins_tests.jl
                          '''
                }

where jenkins_tests.jl contains

using Pkg
Pkg.Registry.add("General")
Pkg.Registry.add(RegistrySpec(url = "ssh://git@<internal url>"))
Pkg.instantiate()
include("runtests.jl")
4

I already have exported the two env variables: SSH_KEY_PATH and SSH_PUB_KEY_PATH - GitLab does that for me but I have verified that they are there and valid.

If I than use the above code I simply get stuck in the line where I add the private registry.

5

Have you added the second package’s SSH key to the list of deploy keys for the first package?

6

I don’t quite understand what you mean.
The SSH key for the package is in the registry but I have problems adding the registry itself.
In the test folder there is specific .toml with the deps, needed for the test and in the main .toml for the package all deps for the package itself are specified.

7

Oh, sorry, I’m trying to help you with the wrong problem!

Why are you writing the path as “ssh://git@…”. In my CI config, I just have

  - julia -e 'using Pkg; pkg"registry add git@<internal url>:Julia/Registry.git"'
8

If I don’t I just get asked to specify the private key location.

9

When you say you have SSH_KEY_PATH and SSH_PUB_KEY_PATH exported, do you mean you have entries in Settings → CI / CD → Variables with type File for those two variables?

(I have a private registry working on a self-hosted GitLab instance, so I’m just going through and trying to compare to what I have for you. If this isn’t helpful, let me know!)

10

Exactly, I added both as files.

11

Did you add a deploy key to the registry’s repo with the SSH key of the package that is trying to clone it?

12

I have added a global shared deploy key to both the registry and the packages within the registry.

13

Could this be the issue?

14

Have you tried modifying your CI job so it just uses the git CLI to clone the registry repo, to see if that works? You said you cloned the registry, but I wasn’t clear if that was inside a CI job or not…

Other than that, I’m out of ideas. Sorry. I’m happy to check more settings in my GitLab setup if that would be helpful as you continue to debug.

15

I did that using the first script I provided. But that used HTTP.

16

I suggest you use https tokens. Use git’s insteadOf feature to auto-convert all ssh and non-cred urls to the correct credded url. To do this, have your before_script section do something like

    git config --global url."https://$JULIASYSTEMUSER:$JULIAREGTOKEN@gitlab.mycompany.com/".insteadOf "git@gitlab.mycompany.com:"
    git config --global url."https://$JULIASYSTEMUSER:$JULIAREGTOKEN@gitlab.mycompany.com/".insteadOf "https://gitlab.mycompany.com/" --add

This works nice because it doesn’t matter how the url’s are listed in the registry.

2 Likes
17

JULIASYSTEMUSER and JULIAREGTOKEN is the username and password?

18

They are CI env vars for the system user and that user’s token. This user must have the read-rights you need.

1 Like
19

May I ask, what docker image are you using to make the solution in Use private registry in GitLab CI job - #16 by ianfiske work? My naive attempt resulted in

/bin/bash: line 98: git: command not found
20

You can use any docker image that has julia installed (the default ones on docker hub should work fine). Looks like you need to add git. You might need to add this line to your before_script section in your .gitlab-ci.yml:

  - which git || (apt-get update -qq && apt-get install --no-install-recommends -qqqy git)

assuming your docker image is debian or derivative (e.g., ubuntu).

2 Likes