What do you want, hard-to-understand runtime error messages or not being able to install mutually incompatible packages in an environment in the first place? There aren’t many other choices. Compat bounds are there to try and do the latter, but if folks here would prefer to deal with the former we can just remove all compat bounds and make everyone happy.
The problem of compat bounds is that one can’t know in advance if and when other non-julia code which doesn’t follow semver will break API/ABI. There are packages, like for example boost, which break the ABI in every single release because they embed the full version number in the soname of libraries. But there’s also the opposite situation, like curl, where the major version bump doesn’t signal any API/ABI breakage (well, they did release a bugged version just because they didn’t run the tests or something like that, but that wasn’t API/ABI related). How do you know? You can’t possibly guess this for an arbitrary package.
But all of this is relatively easy to fix: once you become aware that some compat bounds should be relaxed/tightened you open a pull request to the General registry to fix that and that’s it. This had been explained three months ago, but perhaps complaining about this is easier than actually fixing it once and for all without further discussions. Unless of course one prefers runtime errors and advocates for no compat bounds at all (this hasn’t been suggested yet, but you never know)
Julia packages are expected to follow semver and generally do so. This makes semver rules in Pkg work perfectly fine as expected, for Julia packages.
But libraries from other ecosystem don’t really have to follow semver, and many actually don’t as you correctly point out. So it seems kinda strange to tie Julia package versions with semver rules exactly to upstream library versions. Say, some LibAbc releases weekly updates with new major version each time - does it mean that Julia packages depending on LibAbc_jll would need to update their compats basically every time LibAbc_jll is built again?
Yeah, this is what I had in mind, too. It seems that, ideally, JLLs would have versioning independent of the wrapped library. I guess this would break some expectations, though, and possibly require some work on the relevant infrastructure. Another downside is that maintaining JLLs would become less automated?
It isn’t like this hasn’t been discussed endless times in the BinaryBuilder channel on slack. But any alternative only involves grater complications. And again, API/ABI breakages are unpredictable, in the most generic case you can’t easily know in advance the new version you just released breaks API/ABI or not, so how do choose the version number of your indirection layer before building/registering it?
@giordano , I mostly agree, but now I am curious about best practices in the following case: does the fact that breaking API/ABI does correlates little with jll versions (due to other ecosystems not following semver) imply that compat bounds for jlls should look like "x.y.z - a.b.c" and not like "a.b.c" in order to guard from upstream packages that do not follow semver and might release 1.0.1 and 1.0.2 that are incompatible?
Is it correct to say that compats for jll packages should generally use hyphen specifiers, unless there is some expectation that upstream follows semver?
Edit: And to reiterate your point, given that outside ecosystems might not follow semver, the least bad solution is to be proactive in expanding compat bonds in our projects as our dependencies release new version. Maybe documenting this a bit more will avoid consternation in the future. Would the Pkg docs be a good place to submit a PR explaining all this logic? I (and others here) did not find this immediately obvious.
It can, but this approach really needs more transparency and automation:
Would be great to have all systematic updates to General listed somewhere, so that they easily applied elsewhere. Otherwise, it’s fixing the problem in a registry (the most popular, yes).
Also there should be scripts/jobs/warnings for Project.tomls, of two kinds: “your package’s info has been updated in General, please make sure further updates keep these changes” and “you have a dependency (eg libcurl) whose compat has been updated in a number of packages in General, please consider also updating”.
Otherwise, the process is really opaque. I’m pretty sure many Julia users don’t even know that package info can easily differ between its Project.toml and General registry!