My package is tested and runs with Julia 1.6.x, 1.7.x and 18.8.0-beta3.
And it’s registered on JuliaHub.
Now I get an email: “Hey, what’s up with your package? JuliaHub says it’s broken.”
And indeed, on JuliaHub, under the heading Status, is displayed:
PkgEval redbox: fail.
When searching for an explanation I find a small ‘i’ next to the package name. When I click it I read:
"PkgEval status for Julia@1.9.0.DEV
and package fail(reason: unknown)."
Whatever the intended purpose of this presentation is, I consider it not appropriate. It suggests to users that the package is corrupt and should not be downloaded in its current state. As my experience teaches.
I ask to turn off this disservice to users and package writers.
PkgEval is a CI service that tests each package in General against the master branch of Julia each night. A red badge means your package’s tests currently fail on Julia nightly.
Often this is caused by badly designed tests, which may fail depending on rng streams for example. In that case package authors should fix their tests and make them more robust, otherwise regressions in Julia affecting your package may not get caught. Another common case is packages relying on Julia internals or functions that aren’t considered public API. This is also the package author’s responsibility to fix.
If those two arent the reason for the failure, it may indeed have unearthed a newly introduced bug in Julia. It’s great if people open issues about these, but they will be carefully examined before any release is made, so the bugs don’t make it into the final release. Of course, the release managers’ jobs are made a lot harder by packages failing PkgEval for the first two reasons, so it would be great if more package authors were more aware of this process and got involved by testing their own package against Julia nightly.
I dont have any association with Julia Computing, but I’d guess this is likely the reason this badge exists. I don’t think it is misleading users about the quality of your package. Most package’s READMEs have their own CI badges right at the top. At least on my browser, the PkgEval badge is the bottom-most thing on the page, right above the footer. I fail to see how this would be a disservice to users and package writers.
I believe 'nightly` is moving fast sometimes, so non-compatible packages could be informed, but should not be marked as failing (especially because it could be unclear what is failing).
Agreed. I think having the packages evaluated versus latest release (and latest LTS) is what should be shown much more prominently than nightly. Runtime shouldn’t be an issue either because there are likely far more nightly runs than releases (Julia or Package).
OP makes a good point that new users simply won’t understand what failing nightly means and it may turn them off.
I just loaded Julia Version 1.9.0-DEV.406 (2022-04-18) (0 days old master) and tested everything. There were no errors or other deviations from the results of my release version.
The error message is still on JuliaHub, where it refers to Julia@1.9.0.DEV-1600cb97d2.
May I remind you what Julialang says about the “bleeding-edge binaries”: “The nightly builds are for developer previews and not intended for normal use.” So there shouldn’t be error messages from the use of nightly builds either.
The package page on JuliaHub only makes sense if it is for everyday use and general public. I don’t mind if on JuliaHub the result of tests with the latest official release version of Julia is displayed.
But JuliaHub should stick to the warnings of Julilang and not unsettle users and annoy developers by reporting errors there that are not attributable to the packages advertised there.
JuliaHub is not run by the julia community - have you considered contacting their support, as is suggested when clicking on that little i next to a package name? I’m sure they’d be happy to clarify their UI or add a small info text when hovering over the badge, explaining what PkgEval is. One thing I can imagine would help is including a link to the PkgEval run & its result that show what exactly failed, so people can judge whether the failure is real or due to nightly being broken.