Julia `Pkg` seems to look in the wrong place when installing private package in Github CI

General Usage
1

I am using Github CI for package development. My package is private, and it has dependencies on some other private packages that I developed. While I can successfully clone the private dependency packages from github into the CI server, when CI goes to install those packages, it fails.

I was hoping someone might be able to see where I am making a mistake. Here is the relevant section of the CI.yml file.

      - name: Clone JuliaGendUniv_Types
        run: git clone git@github.com:university-gender-evolution/JuliaGendUniv_Types.jl.git
      - name: Clone JuliaGendUniv
        run: git clone git@github.com:university-gender-evolution/JuliaGendUniv.jl.git
      - name: Check file download was successful
        run: |
          pwd
          ls -lah
      - name: Install JuliaGendUniv private repo
        run: julia --project -e 'using Pkg; Pkg.add(; path="JuliaGendUniv.jl")'
      - name: Install JuliaGendUniv_Types private repo
        run: julia --project -e 'using Pkg; Pkg.add(; path="JuliaGendUniv_Types.jl")'

So I can see that the files are downloaded to CI since I have an instruction to list the directory structure. The image below is from the CI server output.

drwxr-xr-x 9 runner docker 4.0K Mar 28 00:04 .
drwxr-xr-x 3 runner docker 4.0K Mar 28 00:04 ..
drwxr-xr-x 8 runner docker 4.0K Mar 28 00:04 .git
drwxr-xr-x 3 runner docker 4.0K Mar 28 00:04 .github
-rw-r--r-- 1 runner docker  138 Mar 28 00:04 .gitignore
drwxr-xr-x 8 runner docker 4.0K Mar 28 00:04 JuliaGendUniv.jl
drwxr-xr-x 7 runner docker 4.0K Mar 28 00:04 JuliaGendUniv_Types.jl
-rw-r--r-- 1 runner docker 1.1K Mar 28 00:04 LICENSE
-rw-r--r-- 1 runner docker  95K Mar 28 00:04 Manifest.toml
-rw-r--r-- 1 runner docker  731 Mar 28 00:04 Project.toml
-rw-r--r-- 1 runner docker  681 Mar 28 00:04 README.md
drwxr-xr-x 3 runner docker 4.0K Mar 28 00:04 docs
drwxr-xr-x 3 runner docker 4.0K Mar 28 00:04 src
drwxr-xr-x 2 runner docker 4.0K Mar 28 00:04 test

The error I am getting is below. Seems like the CI server is trying to install the package from the wrong location? The message reads:

ERROR: expected package `JuliaGendUniv_Types [603251c8]` to exist at path `/home/runner/.julia/packages/JuliaGendUniv_Types/ZJncY`

Here is the output from the CI server.

Run julia --project -e 'using Pkg; Pkg.add(; path="JuliaGendUniv.jl")'
  Installing known registries into `~/.julia`
     Cloning git-repo `/home/runner/work/JuliaGendUniv_Optimize.jl/JuliaGendUniv_Optimize.jl/JuliaGendUniv.jl`
    Updating git-repo `/home/runner/work/JuliaGendUniv_Optimize.jl/JuliaGendUniv_Optimize.jl/JuliaGendUniv.jl`
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
ERROR: expected package `JuliaGendUniv_Types [603251c8]` to exist at path `/home/runner/.julia/packages/JuliaGendUniv_Types/ZJncY`
Stacktrace:
  [1] pkgerror(msg::String)
    @ Pkg.Types /opt/hostedtoolcache/julia/1.8.5/x86/share/julia/stdlib/v1.8/Pkg/src/Types.jl:67
  [2] collect_fixed!(env::Pkg.Types.EnvCache, pkgs::Vector{Pkg.Types.PackageSpec}, names::Dict{Base.UUID, String})

Seems like the Package manager is looking in /home/runner/.julia/packages/JuliaGendUniv_Types/ZJncY instead of /home/runner/work/JuliaGendUniv_Optimize.jl/JuliaGendUniv_Optimize.jl/JuliaGendUniv.jl.

I am also wondering if this has anything to do with the Manifest.toml file being in the repository? I had to do that since I am pulling in a private repository and I get some strange errors if I don’t include the manifest with the specific github directory referenced.

Any suggestions would be appreciated. I have kinda exhausted my debugging muscles.

2

I suspect that the problem is that JuliaGendUniv_Types is a dependency of JuliaGendUniv and since they are unregistered you need to install the dependency first.

3

Thanks for the response @GunnarFarneback . It is kinda strange. So the JuliaGendUniv package is the older package and I am trying to break it into smaller packages. So the JuliaGendUniv_Types package and JuliaGendUniv don’t depend on each other. I believe that JuliaGendUniv_Types needs JuliaGendUniv for testing, but that is it.

It is odd though, because if I try to install only the Types package, then CI complains that the JuliaGendUniv old package is not present. If I try and install the JuliaGendUniv old package, then CI complains that the Types package is not present. So just went through and checked and could not find any dependencies between projects, but perhaps there is something buried in there somewhere, and I have to keep digging?

Also, do you happen to know whether I need to keep the Manifest.toml in the private repositories. I think I do, but I want to make sure this does not mess anything thing up. When I tried to remove them, Pkg could no longer find the link to the packages to update them. But as I have seen in the past, we don’t usually save the Manifest with the package. So if you happen to know the correct thing to do there, please pass it along. Otherwise, I will keep digging :).

Thanks again.

4

Whether there are any dependencies between the packages should be visible in the respective Project.toml files.

Unregistered packages can be tricky to work with outside your own Julia installation. Arguably the best solution is to register them, and you can do that in a registry of your own with the help of GitHub - GunnarFarneback/LocalRegistry.jl: Create and maintain local registries for Julia packages.

5

Oh interesting. I was just reading your link. I have not registered a package before, so I am not very familiar with the process. But I am still very early in development, so I was trying to keep things private until all the pieces are working. Does it still make sense to register even private packages? Like is there a way to keep the code private and still register it. And then make the package public once everything works?

Is there a video or something on this. It always helps to watch someone do it, if that is available :).

6

The idea would be to make your own personal registry, which you can keep as private as you wish. Registering packages in your own registry has a very low overhead compared to registering a public package in the General registry.

7

Okay cool. I will try and make a simple dummy package, and see if I can get the local registry to work. If I can get a personal registry to work, would I still be able to use that with the Github CI? I am not clear on that part. Or would I need to setup a separate registry server for that.

8

The registry is just a git repository and if you host it on GitHub you should be able to use it from Github CI, provided that you know how to setup tokens or whatever is needed for authentication to your private repositories.

I don’t have any private GitHub repositories myself but there’s some related discussion in Private registry.

9

Sounds good @GunnarFarneback . I will read through the docs and then give it a try. Thanks again for all of your help and all the information.