Well, running tests n (= number of direct deps) times once in a while is definitely feasible. The benefit is that such a run directly indicates which compats to tighten.
Not for every commit, here I agree!
For example, I ran this job on Copulasjl (registered version), and it found definite incompatibility with TaylorSeries = 0.12.0. Some other downgrades also failed tests, but seemingly because tests for that package are non-determenistic and can just randomly fail.
That same CI job (oldnew_compat.yml ¡ GitHub) can also test all versions of each dependency (: Just specify âallâ instead of âtwoâ when running the workflow.
This is supposed to be ran rarely: after you identified that some compat versions donât work, and need to establish what exactly are those. Then, the result directly shows how compat bounds need to be tightened.
OP should probably update to the new GitHub - julia-actions/julia-downgrade-compat: GitHub action to downgrade julia compat entries before testing.
Could this be in a package? Like Aqua? So that I can also run this locally, like Aqua.test_downgrade(MyPackage)
there seems to be some parse error?
Yes, I too would be interested in whether there is a way to do this check locally with no CI in the loopâŚ
Can i see these results somewhere ?
IIUC this is an issue with parsing underscores in the name. Itâs fixed on v1.0.7 but the v1 tag is stuck at v1.0.6
On the new julia-actions/julia-downgrade-compat fork (note the rename) v1 points to v1.0.7. You might just want to try the fork out?
I have a separate private repo with this CI job, that I can run for any registered Julia package â donât know how to share its results specifically. Probably the easiest is to just run it, no setup is needed whatsoever.
Hereâs a screenshot:
Iâm also not too familiar with sharing github actions properly, so simply uploaded and linked it as a yml file above.
@cjdoris I think it would be good to have it find the lowest resolvable minor though, since a lot of packages just choose a major PkgA = "6" and that effectively lower bounds it to the lowest minor that is possible through other dependencies. Testing it on forcing 6.0.0 is a bit stricter than what Pkg can actually do. It is nice to get this all handled but I think the less strict version would still improve the ecosystem but be much easier to get passing.
The trouble with that is that there can be multiple sets of minimal resolvable packages (exponentially many in the worst case) so what do we do then? Just pick one?
The current approach has the advantage of being declarative - you can tell what the minimum versions that will be tested are just by reading Project.toml.
Also the current approach is a very simple modification to Project.toml (you could probably write it in sed).
I appreciate the point about making it easier on users to get this test passing the first time they use it. Probably for those, using aplavinâs recipe as a one-off will get them there - I should add this to the docs for the action.
I tried out the downgrade CI on some simple packages with few deps and it was great, but trying to use it on a big package with many deps just felt like a total waste of time. It needs to do something smarter than just pinning everything to the minimum versions.
This is the exact same problem as trying to install the newest available versions of packages, just flipping the sign of the inequality operator.
Presumably we could just re-use a lot of the resolver code from Pkg, but just flip the comparison operator so it tries to find the oldest installable set of packages instead of the newest installable set.
The same situation can happen with the normal operation as well, right? The newest version of each package could be incompatible just like the lowest compatible versions. How does the tiebreaker work in that case?
No matter how it works, I agree that we should be able to reuse that logic and just swap the comparison operator as the situation should be symmetric.
Since Julia 1.9 (or was it 1.10?), we need to declare the versions of the Base dependencies, like âStatisticsâ, âPrintfâ, etc, to the same version as the Julia one declared. So for instance we can have something like this in the Project.toml:
[compat]
Printf = "1.9"
Julia = "1.9"
This causes some trouble in this downgrade-package test, because I can only test the packages in the minimal Julia version possible, which most likely wonât be the most used one. Am I missing something? I think it would be useful to be able to test with the minimal versions of packages in the current stable Julia version.
Currently the test fails right away if I want to test the package in Julia 1.10, since it wonât be able to install Printf 1.9.
I think thereâs an option to skip stdlibs:
- uses: cjdoris/julia-downgrade-compat-action@v1
with:
skip: Printf