The badge likely turned green again because the issue causing the failure got fixed on Julia nightly.
The reason it was failing before is that I didn’t have stdlib dependencies (like
Pkg) listed in
test/Project.toml. Maybe this was changed on Nightly because of the whole discussion around whether stdlib compat bounds need to be declared in the main
Project.toml file for auto-merging in General (to the extent that I followed that discussion).
Just clicking on the badge for a particular package should also direct you straight to the relevant logs.
That’s actually what I was expecting, but instead, clicking on the badges does “Copy markdown link”. I’d very very much like clicking on the badges to take me the logs instead, so consider this a feature request
For packages that haven’t opted out, it would also be good to have a badge that links to the logs for the documentation build. They’re pretty hard to find hiding behind the .
Running PkgEval.jl yourself is probably not necessary.
Well, if it’s testing only the last release, it would be good to check it before I make a release. And I’m finding that
PkgEval is very idiosynchratic (and undocumented) in how it runs the test suite, so “Testing on Nightly” would not catch issues with
PkgEval, see below.
You should catch most issues by just adding Julia nightly to your CI matrix.
I actually don’t think anyone should have Julia nightly in the normal CI matrix. I used to, and it was breaking all the time for reasons completely unrelated to my package. One of the main reasons was that some packages (like JET) exclude nightly from their compat bounds, so I couldn’t even instantiate my
Given that it’s mostly “noise”, it feels like a waste of significant compute power / carbon for the entire ecosystem to routinely test against Nightly. It’s also a problem that it’s hard for Github workflows to not fail the entire pipeline. I wouldn’t want to have “X” marks on all my commits for jobs that should be “optional”/“advisory”. Granted, that’s a Github problem, and maybe I just gave up too soon trying to find a solution for that.
I might still want to run a check for
PkgEval on release branches (but definitely not on every commit on
master, or every PR).
In any case, just testing against Nightly wouldn’t help much with
PkgEval. The way it runs the test suite is different from how I normally run my test suite (instantiate
test/Project.toml, dev-install the current project, include
test/runtests.jl). I think
PkgEval is using something like the “canonical”
Pkg.test() after activating the main
Project.toml file. However, I’ve seen
Pkg.test() run through when
PkgEval fails. As far as I could tell, for DocumenterCitations it was because the “normal”
Pkg.test() runs within the current git checkout (so instantiating
Documenter.Document() doesn’t fail from not knowing the git remote of the current project), whereas
PkgEval seems to run in a directory that is not a git checkout.
Bottom line: please document how exactly
PkgEval runs the test suite (and a workflow action would probably be a good place to do that, respectively, make it easy to reproduce)