# Term for minor revisions breaking dependents due to non-API usage?

**URL:** <https://discourse.julialang.org/t/term-for-minor-revisions-breaking-dependents-due-to-non-api-usage/112006>\
**Category:** Internals & Design\
**Created:** [March 23, 2024, 8:22am UTC](https://discourse.julialang.org/t/term-for-minor-revisions-breaking-dependents-due-to-non-api-usage/112006 "2024-03-23T08:22:19Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [March 23, 2024, 8:22am UTC](https://discourse.julialang.org/t/term-for-minor-revisions-breaking-dependents-due-to-non-api-usage/112006/1 "2024-03-23T08:22:19Z")

</div>

When a patch or minor revision occurs for a v1+ dependency, we expect our project to keep working. If every package only used public API from dependencies, that is easily true. However, some packages intentionally access internals of their dependencies, whether to accomplish things beyond current APIs or for reflection, e.g. MethodAnalysis.jl exposing Julia’s `Core.MethodInstance`. This requires tighter coupling between the minor revisions of such a package and such a dependency, let’s call them A and B. While a proper minor revision of A poses no risk to a project, a minor revision of B can break A and thus the project, even though it wouldn’t break another project only depending on B. I am wondering if there is a concise term for such a package A and its dependents.

The reason I ask is I think this would be a good idea to indicate upfront in the ecosystem, automatically if possible. Even if the purpose and documentation of A are clear about this, this can be obscured up a chain of dependents, like a package X with a dependency chain X\<Y\<A\<B. It doesn’t seem desirable to allow dependencies on A to spread too far in the ecosystem unknowingly with no conscious preparation to adjust `[compat]` or reimplement.

This question was inspired by LoopVectorization.jl’s deprecation, but that case was quickly reimplemented by indefinitely falling back to Julia v1.11+'s API, specifically `@turbo` doing `@inbounds @fastmath`. Granted, the performance change of the dependents was not satisfactory to users, but their code didn’t technically break. On the other hand, if `Core.MethodInstance` stops existing in a minor revision of Julia, MethodAnalysis.jl’s current API and any dependents must be isolated to a finite range of Julia versions, and further versions need a new API. Luckily there’s not much reason to use that particular example other than directly for reflection. I don’t know of any examples of the nightmare scenario of something widely used indirectly AND susceptible to difficult breaking.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [March 23, 2024, 9:08am UTC](https://discourse.julialang.org/t/term-for-minor-revisions-breaking-dependents-due-to-non-api-usage/112006/2 "2024-03-23T09:08:25Z")

</div>

> [@Benny](#):
>
> a minor revision of B can break A and thus the project

The proper way to deal with this is to restrict the versions of B in A’s Project.toml with the [tilde specifier](https://pkgdocs.julialang.org/v1/compatibility/#Tilde-specifiers) (which only allows patch upgrades) or even the `=` specifier, and only extend this after testing new versions explicitly.

---

<div class="post-metadata">

**Author:** ![simsurace](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simsurace/32/30216_2.png) [@simsurace](https://discourse.julialang.org/u/simsurace)\
**Post date:** [March 23, 2024, 9:28am UTC](https://discourse.julialang.org/t/term-for-minor-revisions-breaking-dependents-due-to-non-api-usage/112006/3 "2024-03-23T09:28:53Z")

</div>

Is it really the case that it is impossible to design stable API for something like MethodAnalysis to build onto? I would guess that whatever users of MethodAnalysis (such as ProfileView) and other tooling need to access is at an abstraction layer above whatever is now considered unstable, even if such abstraction layer is not currently explicitly defined.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [March 23, 2024, 3:16pm UTC](https://discourse.julialang.org/t/term-for-minor-revisions-breaking-dependents-due-to-non-api-usage/112006/4 "2024-03-23T15:16:40Z")

</div>

> [@Benny](#):
>
> I don’t know of any examples of the nightmare scenario of something widely used indirectly AND susceptible to breaking permanently.

Usage of internals is very common in the ecosystem. A few prominent examples that use Base internals are StaticArrays.jl, DataFrames.jl, DataStructures.jl, and OrderedCollections.jl. I guess by “breaking permanently” you mean there is no one around to fix them when they break, which is not true for the examples above, but it’s still not great to have so many core packages relying on Base internals.

> [@Tamas\_Papp](#):
>
> or even the `=` specifier, and only extend this after testing new versions explicitly.

Yes, the correct thing to do is use the `=` specifier in the compat section of the Project.toml file, but unfortunately no one ever does this. I’ve suggested this on Discourse before, but multiple core developers dismissed it as too pedantic.

Perhaps the `public` keyword along with this open issue will eventually improve the situation:

> <https://github.com/JuliaLang/julia/issues/52314>
>
> \_Originally posted by @StefanKarpinski in https://github.com/JuliaLang/julia/iss…ues/49973#issuecomment-1828231790\_
> 
> Not breaking existing code limits what we can do. However, there's a decent argument to be made that since what's breaking is not a public API, blocking access to package internals is not actually technically breaking according to semver. Of course, we still have to be cautious to prevent massive ecosystem breakage. It would not go well to just flip the "no private access" switch for the whole ecosystem at once. Here's a possible transition strategy:
> 
> 1. Allow packages to opt into preventing access to their private internals. This should probably be a flag in the project file, but should maybe also be in the registry. There are three cases for a project that depends on a package that blocks private access:
> 1. Project doesn't access internals so there's no problem;
> 2. Project accesses internals but can easily use a public API instead—minor fix required;
> 3. Project accesses internals and can't easily use a public API instead—can't be easily fixed.
> 2. To handle the last case, we also allow projects to override one of their dependencies blocking private access. A project opting into this acts as an explicit indicator that non-breaking upgrades can break.
> 3. For a while make the flag for whether private access is allowed or not mandatory—you can choose yes or no but you need to have some value in the project file.
> 4. Later we change to preventing private access by default. Projects can drop the flag if the value is to block private access since that's the new default and only packages that need to allow private access need to keep the flag in their project files.
> 
> I'm not sure about the implementation side. If we can control module internals access per depender that would be ideal. That suggests that private/public needs to be metadata associated with the binding itself, which is kind of gnarly. Maybe we can do something with auto-wrapping each package in a public-only wrapper that rebinds only the public bindings of the internal package. That's the simplest to implement, but feels kind of icky.
> 
> Having public/private annotations on fields in structs would also be good, but I'm not quite ready to tackle that.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [March 24, 2024, 12:30am UTC](https://discourse.julialang.org/t/term-for-minor-revisions-breaking-dependents-due-to-non-api-usage/112006/5 "2024-03-24T00:30:33Z")

</div>

> [@CameronBieganek](#):
>
> I guess by “breaking permanently” you mean there is no one around to fix them when they break

Yeah, that is a clearer way to put that.

> [@simsurace](#):
>
> Is it really the case that it is impossible to design stable API for something like MethodAnalysis to build onto?

It makes sense to me that reflection only works for the versions where the addressed internals exist. So far the `[compat]` lists `julia = "1"` but that could be patched to stop inappropriate Julia versions from adding the package; dependents, if they even exist, would have to do this as well. There probably is a practice to do that defensively in advance that escapes me right now. That wouldn’t spell the end, major revisions can introduce new APIs.