Github CI with private repository in Julia

davide-f · May 17, 2022, 3:53pm

Dear Community,

I am having problems on a CI of a private repository of mine, the following error arises during the buildkpg phase of the CI:

caused by: GitError(Code:EUSER, Class:Callback, Aborting, user cancelled credential request.)

The reason is due to the fact that in the building phase of the environment, I do have the https link to the private package (https://github.com//.jl), but it seems like in the process, the CI does not recognize that it is run from my package.

How can I solve?

Thank you

The CI is the standard one:

name: CI
on:
  push:
    branches: [main]
  pull_request:
    types: [opened, synchronize, reopened]
jobs:
  test:
    name: Julia ${{ matrix.version }} - ${{ matrix.os }} - ${{ matrix.arch }} - ${{ github.event_name }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        include:
          - version: '1'
            os: ubuntu-latest
            arch: x64
    steps:
      - uses: actions/checkout@v2
      - uses: julia-actions/setup-julia@v1
        with:
          version: ${{ matrix.version }}
          arch: ${{ matrix.arch }}
      - uses: actions/cache@v1
        env:
          cache-name: cache-artifacts
        with:
          path: ~/.julia/artifacts
          key: ${{ runner.os }}-test-${{ env.cache-name }}-${{ hashFiles('**/Project.toml') }}
          restore-keys: |
            ${{ runner.os }}-test-${{ env.cache-name }}-
            ${{ runner.os }}-test-
            ${{ runner.os }}-
      - uses: julia-actions/julia-buildpkg@v1
      - uses: julia-actions/julia-runtest@v1
      #- uses: julia-actions/julia-processcoverage@v1
      #- uses: codecov/codecov-action@v1
      #  with:
      #    file: lcov.info

rikh · May 17, 2022, 4:34pm

What code are you trying to run exactly in that “building phase of the environment”? Can you share?

davide-f · May 17, 2022, 5:43pm

The toml is the following:

name = ...
uuid = ...
authors = ...
version = "0.1.0"

[deps]
MYPRIVATEREPO = ...
CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
...

[compat]
MathOptInterface = "1.0"
julia = "1"

and in the manifest, except the standard values, there is:

[[MYPRIVATEREPO ]]
deps = ...
git-tree-sha1 = ...
repo-rev = "main"
repo-url = "https://github.com/<github_id>/<MYPRIVATEREPO>.jl"
uuid = ...
version = "0.1.0"

rikh · May 17, 2022, 7:08pm

So, you have a package, say, Foo.jl and you also have Foo.jl in the [deps] section of the Project.toml? Or do you have another package that refers to your private package Foo.jl?

rikh · May 17, 2022, 7:14pm

Generally, what you could do in situations where there is a private repository inside the Manifest.toml is to install it before running buildpkg. For example:

  - run: julia --project -e 'using Pkg; Pkg.add(; url="https://...")'
  - uses: julia-actions/julia-buildpkg@v1

getting access to the repository inside the GitHub Action is possible. You can use a GitHub Action like webfactory/ssh-agent@v0.5.4 to set a secrets.DEPLOY_KEY from your private repository to give GitHub Actions access from one private repository to another.

EDIT: To avoid confusion. What I mean by DEPLOY_KEY is that you add a key to the deploy keys of your private repository and put this key as a secret in the repository in which you currently have problems. Then, set that secret via webfactory/ssh-agent before running the Pkg.add.

davide-f · May 17, 2022, 9:59pm

I’ve tried, but I’m still having issues:

Run webfactory/ssh-agent@v0.5.4
Adding GitHub.com keys to /home/runner/.ssh/known_hosts
Starting ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-lN2HPXrnFuVC/agent.1665
SSH_AGENT_PID=1666
Adding private key(s) to agent
Identity added: (stdin) (git@github.com:davide-f/<repo>.jl.git)
Key(s) added:
4096 SHA256:yAtLh9RzQGzTIGFpDat7iegtJKxD1zBF1I+33laqnE8 git@github.com:davide-f/<repo>.jl.git (RSA)
Configuring deployment key(s)
Added deploy-key mapping: Use identity '/home/runner/.ssh/key-...' for GitHub repository davide-f/<repo>.jl
Comment for (public) key '' does not match GitHub URL pattern. Not treating it as a GitHub deploy key.

The new CI.yaml is:

name: CI
on:
  push:
    branches: [main]
  pull_request:
    types: [opened, synchronize, reopened]
jobs:
  test:
    name: Julia ${{ matrix.version }} - ${{ matrix.os }} - ${{ matrix.arch }} - ${{ github.event_name }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        include:
          - version: '1'
            os: ubuntu-latest
            arch: x64
    steps:
      - uses: actions/checkout@v2
      - uses: julia-actions/setup-julia@v1
        with:
          version: ${{ matrix.version }}
          arch: ${{ matrix.arch }}
      - uses: webfactory/ssh-agent@v0.5.4
        with:
            ssh-private-key: ${{ secrets.SSH_SECRET }}
      - uses: actions/cache@v1
        env:
          cache-name: cache-artifacts
        with:
          path: ~/.julia/artifacts
          key: ${{ runner.os }}-test-${{ env.cache-name }}-${{ hashFiles('**/Project.toml') }}
          restore-keys: |
            ${{ runner.os }}-test-${{ env.cache-name }}-
            ${{ runner.os }}-test-
            ${{ runner.os }}-
...

I’ve followed the guide ssh-agent/README.md at master · webfactory/ssh-agent · GitHub

I put the public key into the deploy keys (named github.com) of “other” private github repo that has to be installed from another private repo

ssh-rsa  <...> git@github.com:davide-f/<repo>.jl.git

The private key has been added into the secret named “SSH_SECRET” of the repo2 that should install the repo1

Still, however, I have problems

Any idea?

Thank you for your help

davide-f · May 17, 2022, 10:00pm

Yes: I have two private repos, and I am trying from repo2 to install repo1

rikh · May 18, 2022, 4:15am

Can you be more specific? Still access denied errors?

davide-f · May 18, 2022, 7:40am

Yes, see the message before, basically the error is the last line of the first block:

Comment for (public) key '' does not match GitHub URL pattern. Not treating it as a GitHub deploy key.

It seems like the keys are not recognized somehow, though it seems to me that they should be fine.

This error suggests that the SSH is not successful and as a consequence, the error I was experiencing during the building of the environment is not solved.
The subsequent error is as follows:

ERROR: failed to clone from https://github.com/davide-f/<repo>.jl, error: GitError(Code:ERROR, Class:Net, failed to resolve address for key-cd9edd77eda3f8efb9b9586183a4d3d26afe042eb1e7e8b23e8b41238d3ef161.github.com: Name or service not known)

rikh · May 18, 2022, 9:51am

Ah. Now I see the error. Have you looked at Multiple deploy keys not working · Issue #88 · webfactory/ssh-agent · GitHub? I’ve setup multiple repositories with SSH keys deploying from one to the other. Most recently a few days ago, so it should work. Double check that you set the right keys.

davide-f · May 26, 2022, 4:57pm

Thank you, I solved.
In particular, I cloned the repo first and then installed it separately.
Thank you very much!

dmoored4 · June 23, 2022, 5:34pm

I’m having the same issue and I can’t figure out how to get the webfactory ssh-agent to work for multiple keys to different private repos. Would you mind sharing what your CI.yml looks like now with the fix you mentioned above?

davide-f · June 26, 2022, 1:57pm

Here it is

name: CI
on:
  push:
    branches: [main]
  pull_request:
    types: [opened, synchronize, reopened]
jobs:
  test:
    name: Julia ${{ matrix.version }} - ${{ matrix.os }} - ${{ matrix.arch }} - ${{ github.event_name }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        include:
          - version: '1'
            os: ubuntu-latest
            arch: x64
    steps:
      - uses: actions/checkout@v2
      - uses: julia-actions/setup-julia@v1
        with:
          version: ${{ matrix.version }}
          arch: ${{ matrix.arch }}
      - uses: webfactory/ssh-agent@v0.5.4
        with:
            ssh-private-key: ${{ secrets.SSH_SECRET }}
      - name: Clone Games.jl
        run: git clone https://github.com/davide-f/{MyPrivateRepo}.jl
      - name: Install private repo
        run: julia --project -e 'using Pkg; Pkg.add(; url="{MyPrivateRepo}.jl")'
      - uses: actions/cache@v1
        env:
          cache-name: cache-artifacts
        with:
          path: ~/.julia/artifacts
          key: ${{ runner.os }}-test-${{ env.cache-name }}-${{ hashFiles('**/Project.toml') }}
          restore-keys: |
            ${{ runner.os }}-test-${{ env.cache-name }}-
            ${{ runner.os }}-test-
            ${{ runner.os }}-
      - uses: julia-actions/julia-buildpkg@v1
      - uses: julia-actions/julia-runtest@v1

dmoored4 · June 26, 2022, 2:46pm

Thank you!

Topic		Replies	Views
Julia `Pkg` seems to look in the wrong place when installing private package in Github CI General Usage	8	747	March 28, 2023
Julia Github actions failing due to missing Pkg Package Management	11	902	March 16, 2021
Cannot add private repository from GitHub New to Julia github , ssh	23	1231	May 16, 2023
Prompting for private key location when developing a private repo General Usage pkg , github	2	839	November 30, 2022
Pkg.clone keeps asking for my SSH key General Usage package , github , ssh	21	5186	October 19, 2021

Github CI with private repository in Julia

Related topics