We’ve been working on improving the new package evaluator (aptly named NewPkgEval.jl) as a tool for testing all registered Julia packages against changes to Julia itself. This functionality has been integrated in Nanosoldier.jl, originally created to do the same but for performance regressions. A simple
@nanosoldier runtests(...) (see the README for more details) can now be used to test Julia against all packages, see here for an example.
Once per day, we also run an evaluation of all packages against the current
master branch of Julia and compare against the previous daily evaluation. The results of this evaluation are put in the NanosoldierReports repository, and are also used to generate badges that you can use in your package’s README. For example:
# change E/Example to the initial and name of your package [pkgeval-img]: https://juliaci.github.io/NanosoldierReports/pkgeval_badges/E/Example.svg [pkgeval-url]: https://juliaci.github.io/NanosoldierReports/pkgeval_badges/report.html [![PkgEval][pkgeval-img]][pkgeval-url]
However, the latest daily evaluation only successfully tested 1845 out of 2922. Over 800 packages failed tests, either because of legitimate test failures or due to issues with the package evaluator. Packages that fail tests cannot be taken into account when evaluating changes to Julia.
It would greatly improve the effectiveness of PkgEval if more packages would pass tests on it!
To make sure your package works with NewPkgEval.jl:
Open the latest report and locate your package. There’s a section for every status (success, fail, skip), and results are further grouped according to more specific reasons.
If your package fails tests, please fix those The NewPkgEval README explains how to replicate the test environment; you only need to install Docker and have permissions to launch containers (try
docker run hello-world).
If your package is missing a dependency, you can just install that using
apt. NewPkgEval uses a plain Ubuntu-based image; refer to the README for more details.
If your package takes too long to test (> 1 hour), you can alter its behavior by checking the environment: NewPkgEval sets the
If you don’t want or can support NewPkgEval, e.g. because of unsatisfiable binary dependencies, you can always blacklist the package.