# Would slowing Julia's release cadence improve ecosystem quality?

**URL:** <https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114>\
**Category:** Community\
**Created:** [August 22, 2023, 8:24pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114 "2023-08-22T20:24:20Z")\
**Posts on this page:** 20\
**Page:** 3

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [August 23, 2023, 6:18pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/41 "2023-08-23T18:18:24Z")

</div>

> [@mihalybaci](#):
>
> So, if I write 10000 lines of code **with no package dependencies** for Julia 1.9, then it should work in 1.7, 1.8, 1.10, etc.

No, every new version intruduces new features, so 1.9 code is _not_ guaranteed to work on 1.8, but only on 1.10, 1.11 etc.

---

<div class="post-metadata">

**Author:** ![ParadaCarleton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paradacarleton/32/20005_2.png) [@ParadaCarleton](https://discourse.julialang.org/u/ParadaCarleton)\
**Post date:** [August 23, 2023, 6:28pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/42 "2023-08-23T18:28:12Z")

</div>

I feel like the issue here is the use of compiler internals. They’re called internals for a reason, and that’s a big reason Zygote is a broken, buggy mess. Either we need to create a clear, public interface that won’t break from update to update and is enough for Enzyme/Zygote/Diffractor to rely on (is that possible?); or we take the Python approach of “let the compiler leak everywhere and freeze it in place”; or we just admit these packages were a bad idea and cut our losses. We don’t have the big tech money we’d need to keep making substantial fixes to these packages with every minor update. If nothing else, we can’t accept a bus factor that small–I think we’ve seen exactly that situation with Zygote having nobody to work on it after Mike Innes left for Relational AI.

---

<div class="post-metadata">

**Author:** ![mihalybaci](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mihalybaci/32/13528_2.png) [@mihalybaci](https://discourse.julialang.org/u/mihalybaci)\
**Post date:** [August 23, 2023, 6:43pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/43 "2023-08-23T18:43:14Z")

</div>

> [@DNF](#):
>
> No, every new version intruduces new features,

True! I certainly overstated there.

> [@adienes](#):
>
> I think your analogy might have the comparison backwards.

Yeah, that’s probably true for some of the specific cases here. But it still likely caused a compatibility issue for many folks trying to setup environments in python 3.5, and I doubt anyone called out the python developers for that.

I do still feel like a lot these are package issues, not language issues, and the two are being conflated.

---

<div class="post-metadata">

**Author:** ![xiaoxi](https://avatars.discourse-cdn.com/v4/letter/x/a9adbd/32.png) [@xiaoxi](https://discourse.julialang.org/u/xiaoxi)\
**Post date:** [August 23, 2023, 6:53pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/44 "2023-08-23T18:53:41Z")

</div>

> [@ParadaCarleton](#):
>
> public interface that won’t break from update to update and is enough for Enzyme/Zygote/Diffractor to rely on

How long would it take Julia to include all the necessary mechanics for those libraries?

---

<div class="post-metadata">

**Author:** ![bilderbuchi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bilderbuchi/32/13562_2.png) [@bilderbuchi](https://discourse.julialang.org/u/bilderbuchi)\
**Post date:** [August 23, 2023, 6:53pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/45 "2023-08-23T18:53:44Z")

</div>

> [@mihalybaci](#):
>
> As a comparison, over the course of Python 3.x, different ways to to string interpolation were added as features, including f-strings in 3.6. So if I wrote a package for Python 3.11 with the following lines
> 
> ```julia
> x = 1
> print(f"x = {x}")
> 
> ```
> 
> Then my package would break for all users still on Python 3.5 or earlier, yet no one (anyone here?) considers 3.5 → 3.6 a “breaking” Python release. So one question is, given a completely analogous situation, why is does the bar seem so much higher for Julia than other languages?

No, your package would not break, because you would then in your `pyproject.toml` update the [`Requires-Python` entry](https://packaging.python.org/en/latest/specifications/core-metadata/#requires-python) to `Requires-Python: >=3.6`.  
Let’s say you did this in your package version `N`, and in `N-1` it was `>=3.0`.

People in an environment with a python 3.5 would only be offered up to `N-1` for installation. People with newer Python would have up to `N` available.  
If you want, in an env with a Python 3.5 interpreter, want to install version `N` (e.g. from some description file analogous to `Manifest.toml`), the package manager would balk, and offer you more or less useful debugging info which compat bounds cause the conflict.  
It would not install the package and cause you to discover the problem later.

So, I don’t know what your below statement is grounded in? 😕

> But it still likely caused a compatibility issue for many folks trying to setup environments in python 3.5, and I doubt anyone called out the python developers for that.

If you are not using the packaging metadata correctly (e.g. by using 3.6+ features, but not giving the correct `Requires-Python` entry) then 🤷 there’s not much the package manager can do, and I guess this is true for Python as well as Julia?

---

<div class="post-metadata">

**Author:** ![mihalybaci](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mihalybaci/32/13528_2.png) [@mihalybaci](https://discourse.julialang.org/u/mihalybaci)\
**Post date:** [August 23, 2023, 7:04pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/46 "2023-08-23T19:04:20Z")

</div>

> [@bilderbuchi](#):
>
> If you are not using the packaging metadata correctly (e.g. by using 3.6+ features, but not giving the correct `Requires-Python` entry) then 🤷 there’s not much the package manager can do, and I guess this is true for Python as well as Julia?

This is my larger point. The sense I get is that a versioning/metadata issue like this in Python would be blamed on the package, but in Julia often gets blamed on the language. Part of it I think, is that Python’s mega packages (NumPy, SciPy, etc.) take up one dependency, whereas in Julia there are many (dozens? more?) dependencies for the same functionality, so more versioning issues exist simply because more packages are necessary?

edit: the idea of better tooling here for updating lower/upper bounds seems to be a good one.

---

<div class="post-metadata">

**Author:** ![ParadaCarleton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paradacarleton/32/20005_2.png) [@ParadaCarleton](https://discourse.julialang.org/u/ParadaCarleton)\
**Post date:** [August 23, 2023, 7:21pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/47 "2023-08-23T19:21:31Z")

</div>

> [@What lessons could Julia's autodiff ecosystem learn from Stan's TinyGrad?](https://discourse.julialang.org/t/what-lessons-could-julias-autodiff-ecosystem-learn-from-stans-tinygrad/103070/35):
>
> 5M isn’t so much from an organizational perspective

Turing.jl was built off of \<$100k… I can’t even imagine what I could do with $5M. That’s more than Stan got for sure.

---

<div class="post-metadata">

**Author:** ![ParadaCarleton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paradacarleton/32/20005_2.png) [@ParadaCarleton](https://discourse.julialang.org/u/ParadaCarleton)\
**Post date:** [August 23, 2023, 7:24pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/48 "2023-08-23T19:24:07Z")

</div>

I have no idea and I think that’s a major part of the problem. Only a very small number of Julia devs (2-4 maybe) know what kind of deep internals Zygote relies on.

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [August 23, 2023, 8:37pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/49 "2023-08-23T20:37:54Z")

</div>

@mihlybaci what you are getting at is spot on, Julia does take non-breaking releases and semver more seriously.

I wrote my first package for Julia something like 5 years ago and have barely changed it since. Its heavy on macro use and touched some internals to return the right line number nodes.

Some of those internals changed around 2020, and the tiny PR to keep it working on all julia versions was submitted by a core developer when they caught it in testing.

I’m not sure if that happens in other languages, but it seems unlikely. Who else even tests every package in the registry with every release?

---

<div class="post-metadata">

**Author:** ![wsmoses](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wsmoses/32/26497_2.png) [@wsmoses](https://discourse.julialang.org/u/wsmoses)\
**Post date:** [August 24, 2023, 12:17am UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/50 "2023-08-24T00:17:32Z")

</div>

Enzyme really doesn’t have a big need for Julia compiler features. We still support Julia 1.6 (and beyond).

The excecption to that is that I’ve found bugs in Julia’s garbage collector that I’ve upstreamed to Julia which are only available on the later versions. What we do there is we work around them with a slower fallback if we were to generate them and use the faster ones otherwise (see [https://github.com/EnzymeAD/Enzyme.jl/blob/daa30d68bd95fa6bc635decf6205b18225c24d51/src/compiler.jl#L6534](https://github.com/EnzymeAD/Enzyme.jl/blob/daa30d68bd95fa6bc635decf6205b18225c24d51/src/compiler.jl#L6534) here for example).

Of course, we still can take advantage of new features in later julias for better error bactraces/etc, but it’s not a requirement for Enzyme to work.

In contrast, I think this was a much bigger issue for Diffractor.jl, which had a requirement for near bleeding edge julia for the infrastructure required to build it.

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [August 24, 2023, 3:07am UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/51 "2023-08-24T03:07:14Z")

</div>

> [@xiaoxi](#):
>
> Would it make more sense to slow down Julia releases after an LTS that includes most of the machinery needed for those cutting-edge AD libraries?

As someone working in this space.  
Such a version doesn’t exist and cannot exist.  
Because

1. There will be a new ideas for how to do AD better that will require support in the compiler to make possible.
2. The language will add a new feature that will need support from the AD engine.

Now of course that doesn’t mean you can’t freeze both your AD and yourjulia version and have something that keeps on working as well as it ever did. But that’s also true today, like around the time of any stable julia release, give or take a few months.  
You don’t even need the LTS for that

> [@wsmoses](#):
>
> In contrast, I think this was a much bigger issue for Diffractor.jl, which had a requirement for near bleeding edge julia for the infrastructure required to build it.

This, is true.

---

<div class="post-metadata">

**Author:** ![simsurace](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simsurace/32/30216_2.png) [@simsurace](https://discourse.julialang.org/u/simsurace)\
**Post date:** [August 24, 2023, 5:10am UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/52 "2023-08-24T05:10:47Z")

</div>

> [@wsmoses](#):
>
> The excecption to that is that I’ve found bugs in Julia’s garbage collector that I’ve upstreamed to Julia which are only available on the later versions.

If there are bugs in the compiler of the LTS release I‘m surprised the bugfixes are not backported. Do they require new features?

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [August 24, 2023, 9:29am UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/53 "2023-08-24T09:29:36Z")

</div>

On the main topic, I agree with @oxinabox, I don’t think we can envision that things are going to slow down in the near future. I suspect the next LTS won’t have static compilation, won’t have everything the AD engines need, won’t have finished moving the stdlibs out into upgradable packages, will have lots of error messages and documentation that could be improved, etc. In other words, there will continue to be big, exciting and/or important changes made for as long as one can foresee. Why would you deny users the benefits of timely improvements?

@aplavin, I have become more convinced of your point, not through my own experience but from the cases where you’ve given a concrete, reproducible example. I’d strongly encourage you to keep better track of these examples when you encounter them, because documenting these failures and reporting them as issues to the package maintainers is one of the few routes towards fixing this unfortunate situation.

I’m also excited about [Add public keyword by LilithHafner · Pull Request #50105 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/pull/50105) (as noted by @ufechner7); I think that, plus a good Aqua test for packages once that merges, might help a lot.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [August 24, 2023, 9:31am UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/54 "2023-08-24T09:31:19Z")

</div>

Backporting requires manual intervention (sometimes, a lot of it); we basically have one amazing but overstretched backport fairy (Kristoffer Carlson) for the entire language. At this point, it’s pretty selective what gets backported to 1.6.

---

<div class="post-metadata">

**Author:** ![wsmoses](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wsmoses/32/26497_2.png) [@wsmoses](https://discourse.julialang.org/u/wsmoses)\
**Post date:** [August 24, 2023, 10:23am UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/55 "2023-08-24T10:23:59Z")

</div>

For sake of understanding, what are you thinking of by “won’t have everything the AD engines need”?

While definitely, I can attest to benefits of new features in later Julia’s making AD easier, I can only think of Diffractor which requires bleeding edge features?

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [August 24, 2023, 10:33am UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/56 "2023-08-24T10:33:56Z")

</div>

Presumably Tim more meant “want” instead of “need”.i.e. people developing AD engines will continue to run into things with Julia’s runtime that are either inconvenient or blocking certain features for them, and they will want upstream changes to julia’s internals.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [August 24, 2023, 11:05am UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/57 "2023-08-24T11:05:14Z")

</div>

Sorry, indeed I was talking about Diffractor. It is indeed my impression that Enzyme is very standalone and works beautifully with current Julia versions.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [August 24, 2023, 11:09am UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/58 "2023-08-24T11:09:57Z")

</div>

I personally don’t see how making fewer releases will solve anything in isolation. That just means that each release will be bigger and increase the likelihood of issues sneaking through the quality checks.

If anything, more frequent smaller releases would probably be better when it comes to trying to reduce the level of accidental breakage of packages.

So the focus should probably be on what actually ends up getting merged to master, not the release cadence.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [August 24, 2023, 11:27am UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/59 "2023-08-24T11:27:16Z")

</div>

> [@xiaoxi](#):
>
> Could it help improve the quality of the Julia ecosystem if Julia releases slow down their pace after the next LTS release

I don’t think so. And I would want to go the other way, have more frequent releases.

I don’t see much of an ecosystem quality problem, relating to new Julia versions (I think others agree here, except for a minor update problem, and I proposed how to fix that):

> [@Forward compatibility and stability of Julia vs. Packages](https://discourse.julialang.org/t/forward-compatibility-and-stability-of-julia-vs-packages/97968/82):
>
> You brought up at (I didn’t want to answer there/derail the main topic?): It could be argued those packages would depend on 1.x+1 but that wouldn’t solve the problem, because then yes those versions would no longer be installable in that Julia, and would force use of older versions of the packages, that also wouldn’t work. So it got me thinking, should it be the other way around, Julia having [compat] on some packages? I’m just not aware that Julia has a Package.toml file per se (at least not…

I obviously don’t control the frequency of releases, but if each one is trying to be a smaller upgrade (maybe not trying to be very small) then they can be more frequent, and less of a worry to miss the feature freeze. People might argue it’s too much work, but we have up to 6 point releases, and with more 1.x releases, I think we need fewer if any 1.x.y, so less backporting work.

[Julia is the first language I’m involved with contributing to. I’m not sure, does Julia have more regressions than others? I have no way to back that up, could actually be fewer. Do people think too many/more than expected? And I’m thinking of genuine ones, fixes (or needing) in point releases, not talking about stuff an upgrade of packages fixes.]

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [August 24, 2023, 12:39pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/60 "2023-08-24T12:39:22Z")

</div>

> [@tim.holy](#):
>
> I’d strongly encourage you to keep better track of these examples when you encounter them, because documenting these failures and reporting them as issues to the package maintainers is one of the few routes towards fixing this unfortunate situation.

I wonder if there’s anything actually actionable here, for example from the PoV of package developers. How can they fix the breakage after the fact?

[Previous page](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114.md?page=2)

[Next page](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114.md?page=4)