Julia motivation: why weren't Numpy, Scipy, Numba, good enough?

Now becoming a bit off topic.

Coming back to a pet peeve of mine. It’s still unclear to the community what packages work in 1.0 or what not.

Just an idea - let’s put a page up on julalang.org with a list of known working packages, a list of WIP’s, and a form where people can vote for packages that are not listed.

4 Likes

Hopefully this will help to alleviate some of those issues.

https://github.com/JuliaLang/Pkg.jl/pull/685

@tobias.knopp

No, I did not upgrade 6 months ago, but as I knew things were converging to 0.7/1.0 and as I need future proof code, I paused developing in Julia until things will be smoother in term of package environment.

Yes, I agree with all of that, and my comment was by no way meant to say that people should do more or anything link that, just that the situation was confusing for new users, which will probably not be up to date with the fast-changing Julia environment (0.6.4 in July, 0.7 in August, 1.0 in August…).

A first time user landing in the download section of the Julia website will download 0.7 or 1.0 as recommended stable versions. Now he wants to use NMF, or Compose, or Mamba? He will follow the instruction, add the package, try loading it, this will fail. He will try a few other packages, same result. What do you think he will conclude?

Therefore, maybe there should be a note on the Julia website that 0.6.4 may be preferred to be used instead of 0.7/1.0 for some applications, the time all developers (including me) finish to migrate their packages to 0.7/1.0.

2 Likes

What are the problem packages? I have a couple of projects going with massive dependency trees and everything works fine now in 0.7/1.0

Edit: Oops I missed where you actually listed packages… but doesn’t Compose work now?

But yes It seems mamba and NMF havent been updated. Time to put in a pull request :slight_smile:

2 Likes

I am not sure this is the best strategy (if you otherwise like Julia). It is a perfectly viable choice to keep developing using 0.6 for a while, and then make a quick transition using FemtoCleaner.jl and a bit of manual work.

I can’t say what people I don’t know may conclude, but as others noted above, having a bit of a transition period after major releases is normal for most free software languages.

As some have mentioned, the transition from Python 2 to 3 is still not complete, a decade after release. Some people simply thrown their hands up, and declare that they are two separate programming languages, so forget about the ‘transition’.

Try googling, and you will find up-to-date websites offering guidance on whether to choose version 2 or 3.

It seems that in comparison to this, the transition period for Julia is shaping up to be very short, indeed.

2 Likes

Initially I had the impression that Julia learned a lot from Python:

but then the common recommendation was to drop support for 0.6:

which I think was unfortunate in retrospect (even if, naturally, it was easier on the maintainers).

2 Likes

Julia is still in an early stage of its development with respect to its future potential and general significance (as I expect it). Up to now I would say, that people are still in an experimenting phase of their Julia usage, not so much into production, more or less.

In this stage it is best to drop things which may hinder further development.

In later, more mature stages, it is, in my opinion, better to go a more compatible and smoother path, not the disruptive path.

The disruptiveness in the early stage is important, to make the non-disruptive path possible in later stages.

In my opinion, Julia is doing this currently in the right way, the python 2/3 step wasn’t best as there are now two pythons.

This is on a very general view on things. Reality is more complex and sometimes circumstances are compelling,

1 Like

Hmm. I wonder what is more important, whether the language offers the possibility of supporting both in a relatively convenient way when necessary, or if packages actually support both versions.

If you have the possibility, then in case of gridlock you can add dual support. At the same time, dropping support for for old versions may push users to upgrade more quickly…

As I know you know, we still support 0.6 in the package ecosystem. The choices are:

  • support multiple julia releases on the same git branch of a package. Think of this as the solution using Compat.jl.
  • support multiple julia releases on separate git branches of a package. Think of this as the solution using git.

The advantage of supporting both 0.6 and 1.0 on one branch is that users of 0.6 automatically continue to get new features and bug fixes. The disadvantage of supporting both 0.6 and 1.0 on one branch is that we’ve had quite a few problems where well-intentioned maintainers inadvertently ended up breaking 0.6 (sometimes in subtle ways that depend on interactions between packages that don’t easily show up in tests).

I am personally persuaded by two points:

  • unlike 0.6, 1.0 is a long-term release—it’s appropriate to focus most development activity on it, and the upgrade is hard enough that there’s no need to make extra busy work for contributors when, for many, 0.6 is or soon will be a distant memory
  • git cherry-pick generally makes it easy to “copy” a commit from one branch to another, as long as a maintainer cares enough to do so

Thus I tend to think that leveraging git—rather than Compat.jl—is the best default choice in most circumstances. But of course it can vary on a package-by-package basis.

8 Likes

Technically yes, of course, as package versions that were installable on 0.6 remain so. But given the fast pace some Julia packages are moving at, and the limited resources of maintainers, de facto a lot of packages don’t backport.

Note that I am not saying this is a bad thing, it may be optimal allocation of scarce resources. Just pointing out the inconsistency between the two positions (which, again, is perfectly fine, as the two posts I quoted were written by two different people).

1 Like

Standard Lua is a very nice little language, doesn’t include the kitchen sink, incredibly small with decent performance for a interpreter. It was the first computer language that my kids learned (via the ComputerCraft mod for MineCraft, years ago, and the Codea iPad app [I wish there were a Julia app like that :grinning:])
It’s perfect as an embedded scripting language (it’s used as such in a number of games, as well as in things like the AeroSpike distributed database [for user defined functions, that get run on each member of a cluster])
It’s 30+ years old, and in some places shows it (like many languages defined before Unicode, and not revamped like Python was in version 3.x).

1 Like

What’s the best way of splitting off a separate v0.6.x branch of (a large number) of packages in an ecosystem,
in such a way that v0.6.x users will continue to get subsequent tagged releases in that branch?
(I’d like to do so for the Str* packages in JuliaString.org, and forget about v0.6.4 [except for backporting bugs to that branch if asked to])

Also, some packages seem to be left behind like JLD.jl. Currently, there seems to be no activity from the maintainers to make JLD.jl work on 1.0 but at the same time JLD2.jl says it shouldn’t be used in serious data scenarios because of potential data loss. Featurewise, it also isn’t a simple replacement for JLD.jl.

4 Likes

You can “pro-actively” fork off a julia-0.6 branch in each of these repositories, and do patch-version bumps (presuming you’ve kept a minor version number difference between 0.6 and 0.7). Of course, you can also always add that julia-0.6 branch later when you discover you need it, e.g., when you want to supply a bugfix: you just go back to the commit before requiring julia 0.7, create the branch, and then cherry pick the commits you want to backport.

2 Likes

This topic is very long now and span over a year and half. Can we summarized results of current discussion in some concise way?

Ooo! I have an idea! Let’s put a button just below the first post that says “Summarize This Topic” and does the job automatically! Would that be hard to implement? :slight_smile:

The marked solution is still a great answer to the original question. This thread has meandered in many ways since then — largely discussing the relative strengths/weaknesses of the languages and ecosystems.

1 Like

One hell of an article John :ok_hand:

1 Like

I ran in to NumExpr again today, it was last recommended to me in hmm 2013?

NumExpr is another of Julia’s contemporaries.
An a way to accellerate math in python.
It was created in 2007,

NumExpr is kinda hideous, in that you need to wrap all your math in strings,
so that you can accelerate it.
To be fair we’ve been known to do things not dissimilar using string macros.
But that was a thing people were still kinda excited about doing in 2009.
That was considered a potential way to have less painful fast math in python…