# Would slowing Julia's release cadence improve ecosystem quality?

**URL:** <https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114>\
**Category:** Community\
**Created:** [August 22, 2023, 8:24pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114 "2023-08-22T20:24:20Z")\
**Posts on this page:** 19\
**Page:** 5

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [August 25, 2023, 2:37pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/81 "2023-08-25T14:37:28Z")

</div>

> [@tim.holy](#):
>
> other than just telling people directly that they can’t run old packages on new Julia versions

I think Sebastian’s point is that it doesn’t even really do that without also retrocapping the registry since if JuliaInterpreter v0.9.26 has an upper bound on Julia 1.11 say, then on Julia 1.12 Pkg will just install JuliaInterpreter v0.9.25 which doesn’t have that upper bound. So these upper bounds should also be paired with PRs to General (using [GitHub - JuliaRegistries/RetroCap.jl: Retroactively add "caps" (upper-bounded compat entries) to all packages in one or more registries](https://github.com/JuliaRegistries/RetroCap.jl) or other tooling - which IIRC you have used before, but is probably new to a lot of folks as well).

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [August 25, 2023, 2:46pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/82 "2023-08-25T14:46:25Z")

</div>

Real world case of what happens if you don’t retrocap the registry: [On Julia 1.8.0-DEV, LocalRegistry restricted to v0.3.2 · Issue #50 · GunnarFarneback/LocalRegistry.jl · GitHub](https://github.com/GunnarFarneback/LocalRegistry.jl/issues/50).

---

<div class="post-metadata">

**Author:** ![Eben60](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eben60/32/13475_2.png) [@Eben60](https://discourse.julialang.org/u/Eben60)\
**Post date:** [August 25, 2023, 2:49pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/83 "2023-08-25T14:49:37Z")

</div>

> [@ericphanson](#):
>
> I think Sebastian’s point is that it doesn’t even really do that without also retrocapping the registry

Yes, but it would be sufficient to retrocap just the “known sinners”, without touching directly any package not depending on them.

As soon as there are programmatic means to detect usage of Julia internals, it should become possible to enforce Julia upper bound for such packages during registration.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [August 25, 2023, 6:21pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/84 "2023-08-25T18:21:58Z")

</div>

> [@tim.holy](#):
>
> “make old project/manifest versions work on new Julia versions,”

The following point is a bit tangential, but I _do_ have plans for juliaup to eventually automatically start the exact Julia version that is recorded in the manifest of a project. That obviously won’t solve most of the issues discussed here, but I think overall it might make these issues less frequent as folks will run less often into a situation where they have to update a project because they are using a different Julia version than was used to create the manifest of the project.

---

<div class="post-metadata">

**Author:** ![xiaoxi](https://avatars.discourse-cdn.com/v4/letter/x/a9adbd/32.png) [@xiaoxi](https://discourse.julialang.org/u/xiaoxi)\
**Post date:** [August 25, 2023, 6:30pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/85 "2023-08-25T18:30:17Z")

</div>

Does the python community have the same problem? If so, how do they fix it?

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [August 25, 2023, 9:52pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/86 "2023-08-25T21:52:59Z")

</div>

Yes and they don’t.

Or well, I can’t speak for Conda, but what always happens to me if I have a project with pinned pip dependencies and upgrade the Python version is that at least a few of the packages contain C or C++ code. Being old versions they have no pre-built wheels for the new Python version and they instead try to build locally, and always fail.

---

<div class="post-metadata">

**Author:** ![bilderbuchi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bilderbuchi/32/13562_2.png) [@bilderbuchi](https://discourse.julialang.org/u/bilderbuchi)\
**Post date:** [August 26, 2023, 7:49am UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/87 "2023-08-26T07:49:49Z")

</div>

Try using conda… I’ve never had these kinds of problems (on Windows, even).

The only thing I can recall is in a package I help maintain, we support all non-EOL python versions, and try to run CI with _one_ set of dependency versions (from a manifest file) – it was recently hard to find compatible dependency versions which are valid for all active python versions.

---

<div class="post-metadata">

**Author:** ![simsurace](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simsurace/32/30216_2.png) [@simsurace](https://discourse.julialang.org/u/simsurace)\
**Post date:** [September 2, 2023, 12:47pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/88 "2023-09-02T12:47:12Z")

</div>

> [@ParadaCarleton](#):
>
> Distributions.jl is borderline unusable, despite being arguably the single most important package in the entire statistical ecosystem. There are basic correctness bugs in Distributions.jl that have been stuck there for 3 years now with no incoming fix because there’s nobody to review the PRs.

Do those bugs have open issues or PRs? If yes, please bump them and if not, please open issues.

---

<div class="post-metadata">

**Author:** ![ParadaCarleton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paradacarleton/32/20005_2.png) [@ParadaCarleton](https://discourse.julialang.org/u/ParadaCarleton)\
**Post date:** [September 2, 2023, 5:54pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/89 "2023-09-02T17:54:25Z")

</div>

They’re open and have been bumped many times; the biggest problem is lack of resources/contributors to Distributions.jl 😕

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [September 2, 2023, 7:41pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/90 "2023-09-02T19:41:48Z")

</div>

@ParadaCarleton can you share links to the issues so these conversations aren’t just hearsay? there are over 200 issues, mostly feature requests, so its hard to find what you mean.

---

<div class="post-metadata">

**Author:** ![simsurace](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simsurace/32/30216_2.png) [@simsurace](https://discourse.julialang.org/u/simsurace)\
**Post date:** [September 3, 2023, 11:46am UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/91 "2023-09-03T11:46:39Z")

</div>

Yeah links would be useful. I occasionally contribute to Distributions.jl and would like to get anything that stops others from being able to use it fixed.

---

<div class="post-metadata">

**Author:** ![ParadaCarleton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paradacarleton/32/20005_2.png) [@ParadaCarleton](https://discourse.julialang.org/u/ParadaCarleton)\
**Post date:** [September 3, 2023, 8:04pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/92 "2023-09-03T20:04:16Z")

</div>

> [@Raf](#):
>
> @ParadaCarleton can you share links to the issues so these conversations aren’t just hearsay? there are over 200 issues, mostly feature requests, so its hard to find what you mean.

There’s a bunch [here](https://github.com/JuliaStats/Distributions.jl/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3Abug), but the issues brought up most often were related to:

1. Random samples falling outside of the support (especially Beta+Dirichlet distributions)
2. Bugs caused by pervasive use of `@inbounds` (this was a problem regardless of whether the code used `OffsetArrays`)

Users also tended to be frustrated with a lack of tools to prevent bugs in their own code. Most common complaints were about the lack of support for kwarg constructors and Distributions that take/return tables or labeled arrays, which they tend to take for granted when they come from PyMC.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [September 3, 2023, 9:31pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/93 "2023-09-03T21:31:49Z")

</div>

> [@ParadaCarleton](#):
>
> Distributions that take/return tables or labeled arrays

Distributions that return keyed arrays are available in [GitHub - invenia/KeyedDistributions.jl: Distributions and Sampleables with keys for the variates](https://github.com/invenia/KeyedDistributions.jl). I used them some time ago, worked fine!

---

<div class="post-metadata">

**Author:** ![ParadaCarleton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paradacarleton/32/20005_2.png) [@ParadaCarleton](https://discourse.julialang.org/u/ParadaCarleton)\
**Post date:** [September 4, 2023, 7:44pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/94 "2023-09-04T19:44:27Z")

</div>

> [@aplavin](#):
>
> Distributions that return keyed arrays are available in [GitHub - invenia/KeyedDistributions.jl: Distributions and Sampleables with keys for the variates](https://github.com/invenia/KeyedDistributions.jl). I used them some time ago, worked fine!

Huh, thanks, I’ll check it out.

At the same time, this means I have to take the time out of my day to teach new users how to use two different Julia packages for array labeling. DimensionalData.jl is the standard for Bayesian analysis of posterior draws, because that’s what ArviZ.jl uses. It’s the same kind of pain that comes up when I have to use both Polars and Pandas for a data analysis.

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [September 5, 2023, 11:19am UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/95 "2023-09-05T11:19:29Z")

</div>

I think (at least some of) what KeyedDistributions.jl does just works out of the box in DimensionalData.jl :

```julia
julia> rand(Normal(), X(5), Y('a':'c'))
5×3 DimArray{Float64,2} with dimensions: 
  X,
  Y Categorical{Char} 'a':1:'c' ForwardOrdered
   'a' 'b' 'c'
  0.600073 0.252401 1.60991
  0.558153 -0.603837 -1.23051
 -0.170615 0.0776176 -0.671733
 -1.26675 1.01441 0.0785599
 -0.780844 -1.11786 0.122756

julia> dists = DimArray(Returns(Normal()), X('a':'e'))
5-element DimArray{Normal{Float64},1} Returns(X) with dimensions: 
  X Categorical{Char} 'a':1:'e' ForwardOrdered
 'a' Normal{Float64}(μ=0.0, σ=1.0)
 'b' Normal{Float64}(μ=0.0, σ=1.0)
 'c' Normal{Float64}(μ=0.0, σ=1.0)
 'd' Normal{Float64}(μ=0.0, σ=1.0)
 'e' Normal{Float64}(μ=0.0, σ=1.0)

julia> mean.(dists)
5-element DimArray{Float64,1} with dimensions: 
  X Categorical{Char} 'a':1:'e' ForwardOrdered
 'a' 0.0
 'b' 0.0
 'c' 0.0
 'd' 0.0
 'e' 0.0

julia> rand.(dists)
5-element DimArray{Float64,1} with dimensions: 
  X Categorical{Char} 'a':1:'e' ForwardOrdered
 'a' -0.846532
 'b' 0.0950699
 'c' 0.561192
 'd' -0.390887
 'e' 1.28783

```

Most things like this are generic in DD because we don’t use `Symbol` for dimension names so we own the dispatch methods.

---

<div class="post-metadata">

**Author:** ![ParadaCarleton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paradacarleton/32/20005_2.png) [@ParadaCarleton](https://discourse.julialang.org/u/ParadaCarleton)\
**Post date:** [September 6, 2023, 6:06pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/96 "2023-09-06T18:06:57Z")

</div>

> [@Raf](#):
>
> I think (at least some of) what KeyedDistributions.jl does just works out of the box in DimensionalData.jl :
> 
> ```julia
> 
> ```

Most things in DimensionalData “just work” 80% of the time with no extra effort. Distributions.jl is no exception 😅

```julia
julia> z = randn(3, 3) |> x->x' * x |> x->DimArray(x, (ax1=[:x, :y, :z], ax2=[:x, :y, :z]))
3×3 DimArray{Float64,2} with dimensions: 
  Dim{:ax1} Categorical{Symbol} Symbol[:x, :y, :z] ForwardOrdered,
  Dim{:ax2} Categorical{Symbol} Symbol[:x, :y, :z] ForwardOrdered
        :x :y :z
  :x 0.747829 0.0869189 -0.255987
  :y 0.0869189 4.23079 -0.0218206
  :z -0.255987 -0.0218206 0.0913322

julia> d = MvNormal(z)
ZeroMeanFullNormal(
dim: 3
μ: Zeros(3)
Σ: [0.7478286057002036 0.08691892384943312 -0.25598736147315915; 0.08691892384943312 4.230791186850467 -0.021820611723327138; -0.25598736147315915 -0.021820611723327138 0.0913321678844106]
)

julia> rand(d)
3-element Vector{Float64}:
 -0.3047069352909566
 -0.04075659598335215
  0.0686493909126511

```

(This is why interfaces aren’t a universal replacement for standard packages. Even if we have interfaces that are _supposed_ to work, there are just so many features and possible bugs at the intersections between functions. You end up having to spend all your time tracking them down for _several_ packages, and the Julia community just doesn’t have the manpower for that.)

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [September 6, 2023, 7:30pm UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/97 "2023-09-06T19:30:36Z")

</div>

I feel you on that last 20%

PDMats.jl only just accepted anything other than `Matrix`, and now just converts every other array type to `Matrix` even though it has a type parameter. DD tries pretty hard to propagate dimensions through everything but they cant survive through a `Matrix` constructor.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [September 7, 2023, 7:39am UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/98 "2023-09-07T07:39:53Z")

</div>

Perhaps worth splitting into a Distributions-specific thread?

---

<div class="post-metadata">

**Author:** ![sijo](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@sijo](https://discourse.julialang.org/u/sijo)\
**Post date:** [September 15, 2023, 9:57am UTC](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114/99 "2023-09-15T09:57:27Z")

</div>

> [@Raf](#):
>
> can you share links to the issues so these conversations aren’t just hearsay?

Let me share my recent experience with Distributions.jl. Please nobody take this as criticism of the developers! Rather, it might illustrate how a lack of resources and focus on a central package can make the user experience miserable.

I was writing a textbook implementation of a Kalman filter, where you typically have some states with noise and some without, so the covariance matrix has some zeros on the diagonal. Nothing fancy, a covariance matrix just needs to be positive-semidefinite and I didn’t give it a thought when I wrote the same filter in Matlab, but in Julia it didn’t go well. Here’s a summary using a simplified example:

```julia
julia> using Distributions

julia> x = MvNormal([0, 0], [0 0; 0 1])
ERROR: PosDefException: matrix is not positive definite; Cholesky factorization failed.

```

Weird… let’s look at the documentation!

```julia
help?> MvNormal
search: MvNormal MvNormalCanon MvNormalKnownCov MvLogNormal AbstractMvNormal MultivariateNormal

  MvNormal

  Generally, users don't have to worry about these internal details.

  We provide a common constructor MvNormal, which will construct a distribution of appropriate type depending on the input arguments.

  ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  MvNormal(μ::AbstractVector{<:Real}, Σ::AbstractMatrix{<:Real})

  Construct a multivariate normal distribution with mean μ and covariance matrix Σ.

  ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  MvNormal(Σ::AbstractMatrix{<:Real})

  Construct a multivariate normal distribution with zero mean and covariance matrix Σ.

```

No word about special steps that would be needed to use a positive-semidefinite matrix (also, the first sentence doesn’t make sense but never mind).

I eventually found a GitHub issue with the suggested workaround: install PDMatsExtras.jl and wrap the covariance matrix in `PSDMat`.

```julia
julia> Σ = PSDMat([0 0; 0 1])
ERROR: MethodError: no method matching PSDMat{Int64, Matrix{Int64}}(::Int64, ::Matrix{Int64}, ::CholeskyPivoted{Float64, Matrix{Float64}, Vector{Int64}})

```

This is inconsistent with Distributions.jl which promotes to `Float64` automatically but that’s a minor complaint, let’s move on.

```julia
julia> Σ = PSDMat([0. 0; 0 1])
2×2 PSDMat{Float64, Matrix{Float64}}:
 0.0 0.0
 0.0 1.0

julia> x = MvNormal([0, 0], Σ)
MvNormal{Float64, PSDMat{Float64, Matrix{Float64}}, Vector{Float64}}(
dim: 2
μ: [0.0, 0.0]
Σ: [0.0 0.0; 0.0 1.0]
)

```

It works, I can use `rand(x)` to draw random values! Well no, I found out later that the sampling was giving wrong results ☹

```julia
julia> rand(x)
2-element Vector{Float64}:
 -0.6249974633185146
  0.0

```

Here the two values are swapped (it’s the first one that has zero variance so should always be 0). Hopefully that bug will be fixed soon: [Fix pivoted order in whiten & unwhiten by vandenman · Pull Request #33 · invenia/PDMatsExtras.jl · GitHub](https://github.com/invenia/PDMatsExtras.jl/pull/33) (edit: already merged and tagged, thanks @vandenman and @oxinabox !)

This stuff just works in Python, R, Matlab, etc. I was surprised by the bad experience in Julia. Drawing samples from a multivariate Gaussian is such a basic thing. So I’m not surprised to see user testimonies and blog posts complaining about the immaturity of the language/ecosystem (though I know it’s actually mature in other respects).

[Previous page](https://discourse.julialang.org/t/would-slowing-julias-release-cadence-improve-ecosystem-quality/103114.md?page=4)