On the issues of foundational packages deprecation and lack of replacements

Julia is founded on a premise that if you’re skilled enough, you can do anything, sometimes by calling C or even tinkering with the compiler.
However, this comes with an issue.
If there are some talented programmers adding feature X to the language, now, several packages that depend on X will use this. Normally, these features are for performance.
In the next part comes the reckoning. What if that package support gets discontinued?
And it gets worse, because foundational packages often deal with low level features that can often rely on implementation details, and thus can break a lot, as has happened with the Loopvectorization package. Even when that’s not the case, you still have packages like Datastructures which add basic data structures to Julia. And still, even if the changes are non-breaking, you still have to benchmark to monitor performance regressions. If regression happens, more work needs to be done.
Imagine if static array gets deprecated today. How would you plan to migrate?

What to do?
It’s not like this issue is specific to Julia but Julia seems to suffer from it the most because in languages like Python, an entire team takes care of the user interface and low-level implementation and can fix when needed. In Julia, lots of high-level packages can be built on top of low-level packages with few having the technical capacity to maintain, who can be gone at any year.

I think, that this is overly pessimistic. First of all, in contrast to Python core features like arrays are part of the language or the standard libraries and thus maintained by the core team.
Secondly, libraries that rely heavily on Julia internals are very rare.

Finally, even if an important library gets depreciated you are always free to:

  • restrict your package/ program to the last, stable version
  • or maintain it yourself
  • or pay s.o. to do so

This problem also happened to me with Python. It is not language specific. And yes, it is a good idea to include benchmarks in your regression testing.

You can restrict the version, but then if your package is itself used by other packages then you’ve just shifted the problem down the line.
The other two options assume it’s available. Sometimes, very few people have the technical capacity to maintain such packages, so it might be an expensive option if available at all.

Loopvectorization deprecation is a hit to the Julia community. $200 is being offered for someone to maintain but seriously how many people have the capacity to maintain such a package?

LoopVectorization will have (an even better) replacement. The only issue is that this will probably not be ready when Julia 1.11 is released. Please, be a bit more patient.

Sometimes there are bumps on the road, that happens in every ecosystem that evolves fast.

5 Likes