Travis CI script does not clone Project.toml of dependent packages

General Usage
1

I have two packages developed myself. Say they are MyPkg and MySubPkg. MyPkg uses MySubPkg, and both are pushed to GitHub.

Both packages have .travis.yml defined for running unit tests automatically by Travis CI when new code is pushed to GitHub. The default Travis CI script (which is called when the script: section is not defined in .travis.yml) runs fine for MySubPkg, but it generates the following error for MyPkg:

...
   Testing MyPkg
 Resolving package versions...
ERROR: could not find project file for MySubPkg at /home/travis/.julia/packages/MySubPkg/EgBdX

It was not clear to me what “project file” means in this error message, but I guessed it would be the Project.toml file of MySubPkg. Indeed, Project.toml of a dependent package does not seem cloned in some cases.

For example, if I remove ~/.julia for a fresh start, start REPL in the directory of MyPkg, and perform pkg> activate . (with the period) followed by (MyPkg) pkg> instantiate, then it starts installing the dependent packages as defined in Project.toml of MyPkg. Because MySubPkg is one of the dependent packages, MySubPkg is cloned in the ~/.julia/packages/MySubPkg/(some random string)/ directory. However, this cloned directory does not contain Project.toml of MySubPkg.

In contrast, if I explicitly add MySubPkg in REPL by pkg> add https://github.com/MY_GITHUB_ID/MySubPkg.jl, then Project.toml of MySubPkg is correctly cloned in the ~/.julia/packages/MySubPkg/(some random string)/ directory.

Why is Project.toml not cloned in some cases? How can I make this file cloned while the Travis CI script runs? Or, is this not the cause of the above error? In that case, how can I resolve the above error?

2

It would be much easier to help with this if you linked the actual repositories.

1 Like
3

I workaround this issue by adding

script:
  - julia -e 'using Pkg; pkg"add https://github.com/MY_GITHUB_ID/MySubPkg.jl"; Pkg.build(); Pkg.test(coverage=true);'

to .travis.yaml, but i’d be glad to see a true solution

1 Like
4

For now that is the solution. Might change in the future though, see e.g. Automatically detect unregistered dependencies by 00vareladavid · Pull Request #1088 · JuliaLang/Pkg.jl · GitHub

1 Like
5

Thanks @abulak and @fredrikekre. I tried @abulak’s suggestion, and now I don’t get the error complaining about the missing project file. However, I get a new error:

ERROR: path /home/travis/.julia/packages/MySubPkg/(random string) for package MySubPkg no longer exists. Remove the package or `develop` it at a new path

So I tried to replace pkg"add ..." with pkg"dev ...", but I still get the same error.

Any suggestions? Maybe related to https://github.com/JuliaLang/Pkg.jl/issues/821?

6

More information. In fact, I have three packages: MyPkg and MySubPkg and HisSubPkg. MyPkg uses the other two packages. None of these are registered packages. Now, my Travis CI script looks like

script:
  - julia --color=yes -e 'using Pkg; pkg"add https://github.com/MY_GITHUB_ID/MySubPkg.jl"'
  - julia --color=yes -e 'using Pkg; pkg"add https://github.com/HIS_GITHUB_ID/HisSubPkg.jl"'
  - julia --project --color=yes -e 'using Pkg; Pkg.build(); Pkg.test(coverage=true)'

In fact, this completes the unit tests. However, very strangely, the first line that adds MySubPkg generates an error about HisSubPkg:

ERROR: path /home/travis/.julia/packages/HisSubPkg/(random string) for package HisSubPkg no longer exists. Remove the package or develop it at a new path.

If I swap the first and second line of the scripts to install HisSubPkg first and MySubPkg next, then the new first line that add HisSubPkg generates an error about MySubPkg:

ERROR: path /home/travis/.julia/packages/MySubPkg/(random string) for package MySubPkg no longer exists. Remove the package or develop it at a new path.

It is very strange that the command adding one package complains about not the package it is adding, but about the other package. These two packages MySubPkg and HisSubPkg do not have interdependency: they are separate, independent packages.

To be more specific, these are actual packages:

and you can see the Travis CI job log here: Travis CI - Test and Deploy Your Code with Confidence.

Any help will be greatly appreciated!

7

Ah, you have commited the Manifest.toml I guess. You could just use instantiate then and everything should work normally.

1 Like
8

I still get the same error even if I do Pkg.instantiate() before Pkg.build().

However, I solved this issue by a different way. I noticed that AppVeyor’s default script succeeds without errors. Looking at AppVeyor’s job log, I was able to pick up its default scripts, which are very simply

julia -e 'using Pkg; Pkg.build()'
julia -e 'using Pkg; Pkg.test(coverage=true)'

By specifying these scripts in .travis.yml, I was able to make the Travis CI job succeeds. I’m not sure why the above is not the default scripts of Travis CI.

For other people who experience the same problem, this is the script: section of my .travis.yml:

script:
  - julia --color=yes -e 'using Pkg; Pkg.build()'
  - julia --color=yes -e 'using Pkg; Pkg.test(coverage=true)'

I have Project.toml and Manifest.toml in all the three unregistered packages involved: MyPkg, MySubPkg, and HisSubPkg, and I am testing only in the Julia nightly build (currently 1.4.0-DEV.285).

9

Weird. I just tested Travis CI’s default script by removing the script: section in .travis.yml, and it works. The default script was not working at the point I asked the original question. I think I made some changes that made the default script work, and that is why AppVeyor’s default script started working above.

10

This saved my day!