I have two private repos on Gitlab, lets call them Foo and Bar, the latter depends on Foo.
When developing on my local machine, I have Gitlab authentication set up, so I can just did
pkg> activate /path/to/Bar
pkg> add https://gitlab.com/tkpapp/Foo.jl
and everything works fine.
But I also need a solution for Gitlab CI, and so far I have not been able to figure one out. Gitlab offers access tokens, but they become part of the URL, so I would need a different manifest on CI. Submodules kind of work using an internal path on Gitlab (it’s the same user), but I need to overwrite my
.gitmodules files for CI because it’s different on my local machine.
Why not add the private packages via git (as in
pkg> add firstname.lastname@example.org:USER/PACKAGE.jl.git) so that you can use SSH key to control access?
I could not get that to work (for reasons I gave up on investigating, something to do with Gitlab’s security model), but I found that Git subtrees work fine.
Hmm… it worked for me between multiple private GitLab.com repositories. I just followed https://docs.gitlab.com/ee/ci/ssh_keys/README.html
Can you please share the relevant
.gitlab-ci.yml file is possible?
To refresh my memory, I set up a demo that shows the configuration https://gitlab.com/tkfm/gitlabdemodownstream.jl/-/merge_requests/1
Just following https://docs.gitlab.com/ee/ci/ssh_keys/README.html
- Create an SSH key pair with
ssh-keygen -t ed25519 -C GitLabDemoDownstream@gitlab-ci -f key
- Add the public key as a deploy key at https://gitlab.com/tkfm/gitlabdemoupstream.jl/-/settings/repository#js-deploy-keys-settings
- Add the private key as
SSH_PRIVATE_KEY environment variable at https://gitlab.com/tkfm/gitlabdemodownstream.jl/-/settings/ci_cd
- Turn off
The CI for the merge request was run before making these repositories public.
I also invited you to the repositories so that you can see the settings.
Protected switch setting would look like:
Thank you so much! I think the issue for me was the Protected switch.
You are welcome! Yeah, I think it might be a common pitfall. I totally forgot about it.
I used the guide posted above in order to setup my CI which worked fine (thanks by the way!) until recently. After Updating to Julia 1.6 I get:
SSH host verification: the server `private.gitlab:22` is not a known host. Please run `ssh-keyscan private.gitlab:22 >> /root/.ssh/known_hosts` in order to add the server to your known hosts file and then try again.
However, manually adding this server in the .yml does not work.
I found https://github.com/JuliaLang/julia/issues/38777 which seems related. At one point it is suggested that adding
ssh:// in front of
git.private.gitlab might fix the problem. However, when I tried this in the
Pkg.registry.add-command, CI could not resolve the url.
As you are using similar setups, did you run into this problem too? Could you solve it?
Edit: I think I found the problem https://github.com/JuliaLang/julia/issues/40297
Adding this solved my problem:
ssh-keyscan private.gitlab >> /root/.ssh/known_hosts
I think it is best to wait for the upstream issues to resolve, in the meantime use the workaround.