The question is in the title
If you refer in test/Project.toml to the main package like this:
[sources]
MyPackage = {path = ".."}
you inherit the compat bounds of the main Project.toml file.
Only if you use test-only dependencies that might change their api, then include compat bounds for these dependencies in test/Project.toml.
Easy to forget updating these, though.
Thank you. Still: to my understanding, tests are normally running in a separate clean environment anyway. The versions of the downstream dependencies of MyPackage for testing must be resolved according to compat bounds in its Project.toml, whether I refer to it explicitly via MyPackage = {path = ".."} or not. The versions of test-specific packages will be resolved to the max available version. If I don’t want to put an upper limit on any of them, what’s the purpose of defining compat bounds?
You don’t have to, but that makes it more likely that your tests will break in the near future. Putting compat bounds on them makes it clear to the user of the package which version are expected to work.
If you don’t mind the risk that breaking changes in test-only dependencies breaks your tests, feel free to ignore compat bounds. If you know that some higher version breaks your tests and you can’t easily fix it, then compat bounds obviously are very useful.
Actually it’s what the tests are for. I’ll get message on CI failure, and look exactly into the issue.
Is there a way to tell dependabot to not make PRs to add compat to test/Project.toml?
I think there are more uses to compat bounds in test/Project.toml than might first meet the eye. The following are things that I have encountered as a developer on one of Julia’s larger libraries.
Broadly speaking, the logic IMO is the same as in normal Project.toml compat bounds. If your tests need a specific package version to run, or if the tested behaviour only works with a specific package version, then put in a compat entry.
To give a motivating example: let’s say that you are developing a package MyPkg, which pulls in a dependency MyDep. Now, MyDep has two versions, v1 and v2, that are broadly in use in the Julia ecosystem.
You can make your package compatible with both versions of MyDep so as to not force your users to be locked into either one. But when it comes to testing, you only want to test the behaviour of your package with the newer version MyDep@2. (One could argue that you should do downgrade tests with MyDep@1 as well, but pragmatically speaking CI resources are limited, and nobody out there does downgrade tests on all their deps.)
In such a case it is certainly valuable to put a compat bound on MyDep in test/Project.toml. Otherwise, it’s very easy to accidentally pull in a different test dependency which pulls in MyDep@1 and makes your tests fail, and you have to trawl through the CI logs and/or Manifest to figure out what happened.
This is not a hypothetical scenario: generally Julia packages are quite fast to update compat but I can list off the top of my head transition periods with ForwardDiff v0 → v1, JSON v0 → v1, PrettyTables v2 → v3, and various SciML packages. (That’s not meant as a criticism of anyone; it’s just Open Source Life.)
Indeed the two different versions need not even be semver-incompatible. Different semver-compatible versions, for example MyOtherDep@1.1 and MyOtherDep@1.2, can also lead to similar situations.
For example, let’s say MyPkg has certain non-functional requirements that require MyOtherDep@1.2, like it can be guaranteed that MyPkg.f() is allocation-free as long as you load MyOtherDep@1.2. In such a case, you don’t want to necessarily restrict the compat in Project.toml because users can still use the old version of MyOtherDep, it’s just slower. But you do want to test that zero-allocation guarantee in your test suite.
OK, fine, you can just let CI resolve to the latest version of MyOtherDep, right? But one day you discover that MyOtherDep@1.2 has also acquired an extension and hence a compat bound on a different library, SomeoneElsesIndirectDep. If you have a different test dependency which also brings in SomeoneElsesIndirectDep, then it’s quite possible that this will force your test environment to resolve to MyOtherDep@1.1.
If you were diligent enough to include a test/Project.toml compat bound on MyOtherDep, then you’ll get a nice error saying that the test environment couldn’t be resolved. But if you don’t, then the test will just fail and again you’ll have to figure it out by trawling through the CI logs’ Manifest.
I’ve described concrete cases above, but I think collectively they illustrate the point that it’s good to be explicit and encode semantic information about what versions of deps your tests are expected to pass with. Thus, even if the cases above don’t occur in practice with your package, my opinion is that it’s good practice to add compat bounds anyway for any test dep. Sure, it might become stale - but that’s also true of normal package compat bounds, and it’s certainly no less stale than the complete absence of a bound.
Surely, all are valid points. On the other side, packages used in testing fall typically in one of the three categories:
- Packages which also used by the “main” package
MyPkg - Test.jl & Co
- Packages which are not used directly by
MyPkg, but still could be used together in a realistic use case.
For the first group, compat limits will be automatically provided by MyPkg. For the second one, I’d prefer the test to fail and signal me the need to check. For the last one - even more so: If incompatibilities occure, I’d look for workarounds if possible, or set compat limits (in extras?), or at least document the issue.
Yes, that’s true, but the point I was making is they’re not always the same as the test bounds. There are situations where you might not want to restrict the package’s compat bounds, since that has downstream effects on users and their ability to install your package in complex environments. However, you may want to have narrower bounds for testing purposes, since that places no additional burden on your users.
i asked the same question here and am awaiting a reply:
See PSA: GitHub Dependabot now supports Julia - #16 by Eben60
All information is provided without warranty.