I love JET.jl. It’s really useful to detect typos, type instabilities and other issues with my code. Unfortunately, it is deeply tied to Julia internals, which means that it often breaks on the nightly version of the language. As a result, packages that have JET as a test dependency are likely to always fail PkgEval. This means they will escape consideration when changes to Julia itself are assessed. Arguably a bad state of affairs, because we want people to use JET in their tests (at the very least JET.test_package). So what can we do to fix that?
My first idea would be to only run the JET tests when we detect that the Julia version is a released one. Is there any way to do that with Pkg? Do people here have better suggestions?
4 Likes
I think you can just check if VERSION contains a DEV. Here’s the current output:
$ julia +nightly
julia> VERSION
v"1.12.0-DEV.1726"
Maybe Jet.jl itself could include that check by default instead of erroring? Or is it the installation of Jet.jl that fails already?
Maybe one could also solve this by some compat magic on the side of Jet.jl? I.e. Have strict bounds for the releases and then a generic fallback, where all methods are just dummies for unsupported Julia versions?
1 Like
Thanks, I didn’t think of that!
Another problem is that you can’t even make JET a test dependency if you want to be nightly-friendly, because sometimes just using JET fails to precompile. So you would need Pkg manipulations, à la
if occursin("DEV", string(VERSION))
Pkg.add("JET")
using JET
JET.test_package(MyPackage)
end
which seems rather clumsy.
I agree. I think, at least in concept, compat should be right tool to handle this and would solve the problem quite elegantly. But I don’t know how this would need to be set up in this case specifically.
But then if JET imposes a compat julia = "< 1.12-DEV" for example, that prevents testing (and thus bug fixing) on the latest version?
Well if it just errors during install then it is of no use anyways, right?
Alternatively for developing both Julia and Jet you’d need to use the latest Jet anyways which you’d get via ] add Jet#master or similar which then has a compatible compat.
I’d envision it like this: The API of Jet.jl changes for minor version increases. If that happens then Jet 0.xx.0 actually should be just a “dummy” version of the new API and has no dependencies. Crucially there should be no compat bounds. The following Jet#0.xx.1 and so on then use somewhat restrictive bounds like julia="~1.11" or so to restrict to a specific Julia version (I checked right now the compat is set to julia="1.10").
This means if you want to use Jet, then you should set your compat with Jet to Jet="0.xx" and just use it.
When your package is then tested on a unsupported version of Julia, the only compatible version of Jet (i.e. compatible with your compat and Julia) is Jet 0.xx.0 which is the dummy that works always (and perhaps just @infos when it’s loaded). The test code using Jet also shouldn’t error because Jet’s API should be identical for all 0.xx versions.
I am not quite sure whether this works as I intend to. I am definitively no expert on compat/version management/… The downside if of course that if Jet doesn’t require changes for a new Julia version then a release is needed to adapt the compat - but that seems like a small inconvenience to me.
cc @aviatesk
You can check if it is a PkgEval run using an environment variable, see Hecke.jl/test/runtests.jl at e76d83a97bf3e70437f37701d1e98703d0d68642 · thofma/Hecke.jl · GitHub.
Won’t help if JET cannot be precompiled though I think.
2 Likes
I would question that. I think JET is just much too unstable to be part of any reliable toolchain. It’s a tool best used manually during local development, installed in your base environment.
If you really feel you must have JET as part of the CI, install it on-demand with a Pkg.add statement in your CI script. But don’t declare it as a standard test dependency.
1 Like
I seperate my CI into regular runs on released versions and one workflow specifically for nightly. Since nightly julia is often not that stable either, I don’t mind when it fails due to JET.
Unfortunately, tools that are best used like that end up not being used at all. Being part of the test suite is a much stronger insurance.
I don’t even test my packages on nightly, precisely for this reason. But PkgEval does, whether I like it or not.
4 Likes
If you don’t like it, you can get it on the blacklist, see PkgEval.jl/Packages.toml at 75219bf0db1090f861972e2279ecb90cdb52228a · JuliaCI/PkgEval.jl · GitHub.
To be honest, I don’t really think this is a problem that users of JET.jl should try to solve. JET.jl is obviously useful for the larger ecosystem to check for certain properties on packages. I personally wouldn’t want to give that up just to make PkgEval happy; I mostly look at my own CI & tests (well, and bug reports!) to gauge whether my package works as I expect it to.
From my POV, PkgEval is a tool Core devs use to gauge whether a change they’ve made to Julia is affecting a large part of the ecosystem. If the use of JET.jl is so widespread that PkgEvals on nightly julia are no longer representative of the effects that changes made to julia have on the ecosystem, it’s in the interest of the maintainers of PkgEval to try to fix that in some way.
Maybe it means that the desires of users of JET.jl go counter to the philosophy Base/Core itself is developed by (although I doubt that). Or maybe it means that the features JET.jl provides should be even more tightly integrated with the core of Julia, so that it doesn’t break all the time? Or that the “interface” of the compiler that JET.jl uses should stabilize? That could also enable usage of JET.jl-like functionality inside of Base, which I personally think would be a boon. A more stable interface would also allow alternatives to be created, without having to know everything about the current compiler/becoming a core dev yourself. Either way, I don’t think this is on the users of JET.jl to fix.
2 Likes
I agree, but I am still reporting it as a user of JET.jl to explore the different solution avenues 
1 Like