When a patch or minor revision occurs for a v1+ dependency, we expect our project to keep working. If every package only used public API from dependencies, that is easily true. However, some packages intentionally access internals of their dependencies, whether to accomplish things beyond current APIs or for reflection, e.g. MethodAnalysis.jl exposing Julia’s Core.MethodInstance. This requires tighter coupling between the minor revisions of such a package and such a dependency, let’s call them A and B. While a proper minor revision of A poses no risk to a project, a minor revision of B can break A and thus the project, even though it wouldn’t break another project only depending on B. I am wondering if there is a concise term for such a package A and its dependents.
The reason I ask is I think this would be a good idea to indicate upfront in the ecosystem, automatically if possible. Even if the purpose and documentation of A are clear about this, this can be obscured up a chain of dependents, like a package X with a dependency chain X<Y<A<B. It doesn’t seem desirable to allow dependencies on A to spread too far in the ecosystem unknowingly with no conscious preparation to adjust [compat] or reimplement.
This question was inspired by LoopVectorization.jl’s deprecation, but that case was quickly reimplemented by indefinitely falling back to Julia v1.11+'s API, specifically @turbo doing @inbounds @fastmath. Granted, the performance change of the dependents was not satisfactory to users, but their code didn’t technically break. On the other hand, if Core.MethodInstance stops existing in a minor revision of Julia, MethodAnalysis.jl’s current API and any dependents must be isolated to a finite range of Julia versions, and further versions need a new API. Luckily there’s not much reason to use that particular example other than directly for reflection. I don’t know of any examples of the nightmare scenario of something widely used indirectly AND susceptible to difficult breaking.
The proper way to deal with this is to restrict the versions of B in A’s Project.toml with the tilde specifier (which only allows patch upgrades) or even the = specifier, and only extend this after testing new versions explicitly.
Is it really the case that it is impossible to design stable API for something like MethodAnalysis to build onto? I would guess that whatever users of MethodAnalysis (such as ProfileView) and other tooling need to access is at an abstraction layer above whatever is now considered unstable, even if such abstraction layer is not currently explicitly defined.
Usage of internals is very common in the ecosystem. A few prominent examples that use Base internals are StaticArrays.jl, DataFrames.jl, DataStructures.jl, and OrderedCollections.jl. I guess by “breaking permanently” you mean there is no one around to fix them when they break, which is not true for the examples above, but it’s still not great to have so many core packages relying on Base internals.
Yes, the correct thing to do is use the = specifier in the compat section of the Project.toml file, but unfortunately no one ever does this. I’ve suggested this on Discourse before, but multiple core developers dismissed it as too pedantic.
Perhaps the public keyword along with this open issue will eventually improve the situation:
It makes sense to me that reflection only works for the versions where the addressed internals exist. So far the [compat] lists julia = "1" but that could be patched to stop inappropriate Julia versions from adding the package; dependents, if they even exist, would have to do this as well. There probably is a practice to do that defensively in advance that escapes me right now. That wouldn’t spell the end, major revisions can introduce new APIs.