# ANN: Nanosoldier package evaluation -- with badges!

**URL:** <https://discourse.julialang.org/t/ann-nanosoldier-package-evaluation-with-badges/33339>\
**Category:** Tooling\
**Tags:** announcement, pkgeval\
**Created:** [January 14, 2020, 10:37am UTC](https://discourse.julialang.org/t/ann-nanosoldier-package-evaluation-with-badges/33339 "2020-01-14T10:37:09Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [January 14, 2020, 10:37am UTC](https://discourse.julialang.org/t/ann-nanosoldier-package-evaluation-with-badges/33339/1 "2020-01-14T10:37:09Z")

</div>

Hi all,

We’ve been working on improving the new package evaluator (aptly named [NewPkgEval.jl](https://github.com/JuliaComputing/NewPkgEval.jl)) as a tool for testing all registered Julia packages against changes to Julia itself. This functionality has been integrated in Nanosoldier.jl, originally created to do the same but for performance regressions. A simple `@nanosoldier runtests(...)` (see the [README](https://github.com/JuliaCI/Nanosoldier.jl#pkgevaljob) for more details) can now be used to test Julia against all packages, [see here](https://github.com/JuliaLang/julia/pull/34238#issuecomment-572547430) for an example.

Once per day, we also run an evaluation of all packages against the current `master` branch of Julia and compare against the previous daily evaluation. The results of this evaluation are put in the [NanosoldierReports](https://github.com/JuliaCI/NanosoldierReports) repository, and are also used to generate badges that you can use in your package’s README. For example:

[![PkgEval](https://juliaci.github.io/NanosoldierReports/pkgeval_badges/E/Example.svg)](https://juliaci.github.io/NanosoldierReports/pkgeval_badges/report.html)

```julia
# change E/Example to the initial and name of your package
[pkgeval-img]: https://juliaci.github.io/NanosoldierReports/pkgeval_badges/E/Example.svg
[pkgeval-url]: https://juliaci.github.io/NanosoldierReports/pkgeval_badges/report.html

[![PkgEval][pkgeval-img]][pkgeval-url]

```

However, the [latest daily evaluation](https://github.com/JuliaCI/NanosoldierReports/blob/master/pkgeval/by_date/2020-01/13/report.md) only successfully tested 1845 out of 2922. Over 800 packages failed tests, either because of legitimate test failures or due to issues with the package evaluator. Packages that fail tests cannot be taken into account when evaluating changes to Julia.

It would **greatly improve the effectiveness of PkgEval if more packages would pass tests** on it!

To make sure your package works with NewPkgEval.jl:

- Open the [latest report](https://juliaci.github.io/NanosoldierReports/pkgeval_badges/report.html) and locate your package. There’s a section for every status (success, fail, skip), and results are further grouped according to more specific reasons.

- If your package fails tests, please fix those 🙂 The [NewPkgEval README](https://github.com/JuliaComputing/NewPkgEval.jl#why-does-my-package-fail) explains how to replicate the test environment; you only need to install Docker and have permissions to launch containers (try `docker run hello-world`).

- If your package is missing a dependency, you can just install that using `apt`. NewPkgEval uses a plain Ubuntu-based image; refer to the README for more details.

- If your package takes too long to test (\> 1 hour), you can alter its behavior by checking the environment: NewPkgEval sets the `CI`, `PKGEVAL` and `JULIA_PKGEVAL` variables to `true`.

- If you don’t want or can support NewPkgEval, e.g. because of unsatisfiable binary dependencies, you can always [blacklist](https://github.com/JuliaComputing/NewPkgEval.jl/blob/bbbc4358636a176677af06cb8f3bf7ff853e7fe1/deps/Registries.toml#L4-L6) the package.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [January 14, 2020, 4:08pm UTC](https://discourse.julialang.org/t/ann-nanosoldier-package-evaluation-with-badges/33339/2 "2020-01-14T16:08:46Z")

</div>

> [@maleadt](#):
>
> Open the [latest report](https://juliaci.github.io/NanosoldierReports/pkgeval_badges/report.html) and locate your package.

It would be great if packages were listed as `username/Packagename.jl` and `org-name/Packagename.jl` instead of only the package name, for users or orgs that have many packages to search for.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [January 14, 2020, 5:59pm UTC](https://discourse.julialang.org/t/ann-nanosoldier-package-evaluation-with-badges/33339/3 "2020-01-14T17:59:58Z")

</div>

Awesome stuff, kudos especially for categorizing the reasons for failure.

> [@maleadt](#):
>
> Open the [latest report](https://juliaci.github.io/NanosoldierReports/pkgeval_badges/report.html) and locate your package

It’s a good thing you are doing this now; at the current rate, pretty soon “looking through the list and finding your packages” is not going to be easy without search tools 🙂

Hopefully you can cross 4 off the list: PaddedViews, ImagineFormat, NRRD, and MetaImageFormat. UnalignedVectors is not necessary anymore (due to `ReinterpretArray`), and nothing in the general registry depends on it. If we [move it to JuliaAttic](https://github.com/JuliaArrays/UnalignedVectors.jl/issues/7) is it automatically blacklisted, or do we need to do that anyway?

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [January 15, 2020, 6:48am UTC](https://discourse.julialang.org/t/ann-nanosoldier-package-evaluation-with-badges/33339/4 "2020-01-15T06:48:17Z")

</div>

It’d be nice if there is a method of `NewPkgEval.run` that takes a vector of `PackageSpec`s rather than names. This way, package authors can make sure that their test suite is NewPkgEval-friendly without releasing it.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [January 15, 2020, 9:54am UTC](https://discourse.julialang.org/t/ann-nanosoldier-package-evaluation-with-badges/33339/5 "2020-01-15T09:54:02Z")

</div>

> [@baggepinnen](#):
>
> It would be great if packages were listed as `username/Packagename.jl` and `org-name/Packagename.jl` instead of only the package name, for users or orgs that have many packages to search for.

That information is not generally known by the registry.

> [@tim.holy](#):
>
> Hopefully you can cross 4 off the list: PaddedViews, ImagineFormat, NRRD, and MetaImageFormat. UnalignedVectors is not necessary anymore (due to `ReinterpretArray` ), and nothing in the general registry depends on it. If we [move it to JuliaAttic](https://github.com/JuliaArrays/UnalignedVectors.jl/issues/7) is it automatically blacklisted, or do we need to do that anyway?

No, we currently try to install all packages from General if there’s any release compatible with the Julia version being tested. What about adding an upper-bound on `julia` to those packages? We could always maintain a list of “deprecated” packages manually, of course.

> [@tkf](#):
>
> It’d be nice if there is a method of `NewPkgEval.run` that takes a vector of `PackageSpec` s rather than names. This way, package authors can make sure that their test suite is NewPkgEval-friendly without releasing it.

That’s a good suggestion, I’ll make an issue for it.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [January 15, 2020, 10:11am UTC](https://discourse.julialang.org/t/ann-nanosoldier-package-evaluation-with-badges/33339/6 "2020-01-15T10:11:04Z")

</div>

> [@maleadt](#):
>
> > It would be great if packages were listed as `username/Packagename.jl` and `org-name/Packagename.jl` instead of only the package name, for users or orgs that have many packages to search for.
> 
> That information is not generally known by the registry.

Isn’t that information encoded in the repo url?  
`repo = "https://github.com/username/PackageName.jl.git"`

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [February 7, 2020, 2:56pm UTC](https://discourse.julialang.org/t/ann-nanosoldier-package-evaluation-with-badges/33339/7 "2020-02-07T14:56:28Z")

</div>

I created a package to generate a dashboard with all your badges to give a quick overview

> [@\[ANN\] PkgDashboards](https://discourse.julialang.org/t/ann-pkgdashboards/34302):
>
> [PkgDashboards.jl](https://github.com/baggepinnen/PkgDashboards.jl) helps you create a markdown page or html page with all your badges from travis, Codecov, PkgEval, commit activity, stars and stars over time graphs. The exported functions are dashboard(target\_users; output=:markdown, autoopen=true, stargazers=false, githubci=false, stars=true, activity=false) pkgdashboard(packagenames; kwargs...) Arguments: target\_users: a string or a vector of strings with github usernames or org names output: :markdown or :html autoopen: indicate wh…