CI for private repo repending on private repo on Gitlab

Tooling
#1

I have two private repos on Gitlab, lets call them Foo and Bar, the latter depends on Foo.

When developing on my local machine, I have Gitlab authentication set up, so I can just did

pkg> activate /path/to/Bar
pkg> add https://gitlab.com/tkpapp/Foo.jl

and everything works fine.

But I also need a solution for Gitlab CI, and so far I have not been able to figure one out. Gitlab offers access tokens, but they become part of the URL, so I would need a different manifest on CI. Submodules kind of work using an internal path on Gitlab (it’s the same user), but I need to overwrite my .gitmodules files for CI because it’s different on my local machine.

3 Likes
#2

Why not add the private packages via git (as in pkg> add git@gitlab.com:USER/PACKAGE.jl.git) so that you can use SSH key to control access?

#3

I could not get that to work (for reasons I gave up on investigating, something to do with Gitlab’s security model), but I found that Git subtrees work fine.

#4

Hmm… it worked for me between multiple private GitLab.com repositories. I just followed https://docs.gitlab.com/ee/ci/ssh_keys/README.html

1 Like
#5

Can you please share the relevant .gitlab-ci.yml file is possible?

#6

To refresh my memory, I set up a demo that shows the configuration https://gitlab.com/tkfm/gitlabdemodownstream.jl/-/merge_requests/1

Just following https://docs.gitlab.com/ee/ci/ssh_keys/README.html

Concrete steps:

  1. Create an SSH key pair with ssh-keygen -t ed25519 -C GitLabDemoDownstream@gitlab-ci -f key
  2. Add the public key as a deploy key at https://gitlab.com/tkfm/gitlabdemoupstream.jl/-/settings/repository#js-deploy-keys-settings
  3. Add the private key as SSH_PRIVATE_KEY environment variable at https://gitlab.com/tkfm/gitlabdemodownstream.jl/-/settings/ci_cd

— https://gitlab.com/tkfm/gitlabdemodownstream.jl/-/merge_requests/1

The CI for the merge request was run before making these repositories public.

I also invited you to the repositories so that you can see the settings.

Edit: Protected switch setting would look like:

image
image971×440 43.3 KB

2 Likes
#7

Thank you so much! I think the issue for me was the Protected switch.

#8

You are welcome! Yeah, I think it might be a common pitfall. I totally forgot about it.

#9

I used the guide posted above in order to setup my CI which worked fine (thanks by the way!) until recently. After Updating to Julia 1.6 I get:

SSH host verification: the server `private.gitlab:22` is not a known host. Please run `ssh-keyscan private.gitlab:22 >> /root/.ssh/known_hosts` in order to add the server to your known hosts file and then try again.

However, manually adding this server in the .yml does not work.

I found https://github.com/JuliaLang/julia/issues/38777 which seems related. At one point it is suggested that adding ssh:// in front of git.private.gitlab might fix the problem. However, when I tried this in the Pkg.registry.add-command, CI could not resolve the url.

As you are using similar setups, did you run into this problem too? Could you solve it?

Edit: I think I found the problem https://github.com/JuliaLang/julia/issues/40297

Adding this solved my problem: ssh-keyscan private.gitlab >> /root/.ssh/known_hosts

#10

I think it is best to wait for the upstream issues to resolve, in the meantime use the workaround.