# Julia motivation: why weren't Numpy, Scipy, Numba, good enough?

**URL:** <https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236>\
**Category:** Community\
**Tags:** history\
**Created:** [February 22, 2017, 3:34pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236 "2017-02-22T15:34:33Z")\
**Posts on this page:** 20\
**Page:** 6

<div class="post-metadata">

**Author:** ![tk3369](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tk3369/32/2824_2.png) [@tk3369](https://discourse.julialang.org/u/tk3369)\
**Post date:** [September 19, 2018, 3:36pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/103 "2018-09-19T15:36:46Z")

</div>

Now becoming a bit off topic.

Coming back to a pet peeve of mine. It’s still unclear to the community what packages work in 1.0 or what not.

Just an idea - let’s put a page up on [julalang.org](http://julalang.org) with a list of known working packages, a list of WIP’s, and a form where people can vote for packages that are not listed.

---

<div class="post-metadata">

**Author:** ![adamslc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adamslc/32/3452_2.png) [@adamslc](https://discourse.julialang.org/u/adamslc)\
**Post date:** [September 19, 2018, 4:16pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/104 "2018-09-19T16:16:22Z")

</div>

Hopefully this will help to alleviate some of those issues.

[https://github.com/JuliaLang/Pkg.jl/pull/685](https://github.com/JuliaLang/Pkg.jl/pull/685)

---

<div class="post-metadata">

**Author:** ![charlesll](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/charlesll/32/3255_2.png) [@charlesll](https://discourse.julialang.org/u/charlesll)\
**Post date:** [September 20, 2018, 3:33am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/105 "2018-09-20T03:33:22Z")

</div>

@tobias.knopp

> [@tobias.knopp](#):
>
> That makes no sense. Julia has been released a month ago, why have you upgraded 6 months ago where it was crystal clear that almost any package was not working with 0.7 / 1.0?

No, I did not upgrade 6 months ago, but as I knew things were converging to 0.7/1.0 and as I need future proof code, I paused developing in Julia until things will be smoother in term of package environment.

> [@tobias.knopp](#):
>
> Maybe you could look a little bit into history
> 
> - when has Python 3 been released?
> - when has Numpy/Scipy released a version that is compatible with Python 3?
> 
> I am pretty certain that the delta is larger than one month 😉
> 
> This is just to make clear that the package transition from 0.6 to 1.0 was an huge amount work. I have ported three packages (Gtk, Winston, NFFT) and have helped fixing things in HDF5 and Cairo. If you are seeing packages not working, the best thing is either to stay on 0.6 (which is rock stable) or help porting things to 1.0.

Yes, I agree with all of that, and my comment was by no way meant to say that people should do more or anything link that, just that the situation was confusing for new users, which will probably not be up to date with the fast-changing Julia environment (0.6.4 in July, 0.7 in August, 1.0 in August…).

A first time user landing in the download section of the Julia website will download 0.7 or 1.0 as recommended stable versions. Now he wants to use NMF, or Compose, or Mamba? He will follow the instruction, add the package, try loading it, this will fail. He will try a few other packages, same result. What do you think he will conclude?

Therefore, maybe there should be a note on the Julia website that 0.6.4 may be preferred to be used instead of 0.7/1.0 for some applications, the time all developers (including me) finish to migrate their packages to 0.7/1.0.

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [September 20, 2018, 5:28am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/106 "2018-09-20T05:28:32Z")

</div>

What are the problem packages? I have a couple of projects going with massive dependency trees and everything works fine now in 0.7/1.0

Edit: Oops I missed where you actually listed packages… but doesn’t Compose [work now](https://discourse.julialang.org/t/julia-1-0-support-just-landed-on-gadfly-master/14978)?

But yes It seems mamba and NMF havent been updated. Time to put in a pull request 🙂

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [September 20, 2018, 7:23am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/107 "2018-09-20T07:23:42Z")

</div>

> [@charlesll](#):
>
> as I need future proof code, I paused developing in Julia

I am not sure this is the best strategy (if you otherwise like Julia). It is a perfectly viable choice to keep developing using 0.6 for a while, and then make a quick transition using [FemtoCleaner.jl](https://github.com/JuliaComputing/FemtoCleaner.jl/) and a bit of manual work.

> [@charlesll](#):
>
> add the package, try loading it, this will fail. He will try a few other packages, same result. What do you think he will conclude?

I can’t say what people I don’t know may conclude, but as others noted above, having a bit of a transition period after major releases is normal for most free software languages.

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [September 20, 2018, 7:40am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/108 "2018-09-20T07:40:18Z")

</div>

> [@Tamas\_Papp](#):
>
> I can’t say what people I don’t know may conclude, but as others noted above, having a bit of a transition period after major releases is normal for most free software languages.

As some have mentioned, the transition from Python 2 to 3 is still not complete, a decade after release. Some people simply thrown their hands up, and declare that they are two separate programming languages, so forget about the ‘transition’.

Try googling, and you will find up-to-date websites offering guidance on whether to choose version 2 or 3.

It seems that in comparison to this, the transition period for Julia is shaping up to be very short, indeed.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [September 20, 2018, 7:45am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/109 "2018-09-20T07:45:56Z")

</div>

Initially I had the impression that Julia learned a lot from Python:

> [@1.0 adoption path?](https://discourse.julialang.org/t/1-0-adoption-path/7922/39):
>
> The big issue was upgrade deadlock. Instead of making it possible for packages to support both Python 2 and 3 at the same time, the Python devs provided the [2to3 tool](https://docs.python.org/3/library/2to3.html), hoping that the entire ecosystem would upgrade from Python 2 to 3 all at once. But upgrade effort turned out not to be what stymied the transition. Packages still needed to support Python 2 because most of their users were still using it. And since most packages stayed on Python 2, users did as well – if anything they depended on…

but then the common recommendation was to drop support for 0.6:

> [@\[ANN\] Introducing Upgradathon Fridays](https://discourse.julialang.org/t/ann-introducing-upgradathon-fridays/12047/3):
>
> And here is how you do it! First off, the process will be much simpler if you choose to drop support for Julia 0.6 on the master branch. Unless you have strong reason to create another release compatible with Julia 0.6 I would strongly recommend doing this, several packages, like StaticArrays and DataStructures, have already done so. If you need to release a 0.6 compatible bugfix release of your package you can do this from a branch later. If your package are dependent on other packages, I sug…

which I think was unfortunate in retrospect (even if, naturally, it was easier on the maintainers).

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [September 20, 2018, 9:32am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/110 "2018-09-20T09:32:26Z")

</div>

> [@Tamas\_Papp](#):
>
> which I think was unfortunate in retrospect (even if, naturally, it was easier on the maintainers).

Julia is still in an early stage of its development with respect to its future potential and general significance (as I expect it). Up to now I would say, that people are still in an experimenting phase of their Julia usage, not so much into production, more or less.

In this stage it is best to drop things which may hinder further development.

In later, more mature stages, it is, in my opinion, better to go a more compatible and smoother path, not the disruptive path.

The disruptiveness in the early stage is important, to make the non-disruptive path possible in later stages.

In my opinion, Julia is doing this currently in the right way, the python 2/3 step wasn’t best as there are now two pythons.

This is on a very general view on things. Reality is more complex and sometimes circumstances are compelling,

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [September 20, 2018, 9:50am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/111 "2018-09-20T09:50:38Z")

</div>

> [@Tamas\_Papp](#):
>
> but then the common recommendation was to drop support for 0.6:

Hmm. I wonder what is more important, whether the language offers the _possibility_ of supporting both in a relatively convenient way when necessary, or if packages _actually_ support both versions.

If you have the possibility, then in case of gridlock you can add dual support. At the same time, dropping support for for old versions may push users to upgrade more quickly…

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [September 20, 2018, 10:21am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/112 "2018-09-20T10:21:35Z")

</div>

> [@Tamas\_Papp](#):
>
> which I think was unfortunate in retrospect (even if, naturally, it was easier on the maintainers).

As I know you know, we still support 0.6 in the package ecosystem. The choices are:

- support multiple julia releases _on the same git branch_ of a package. Think of this as the solution using `Compat.jl`.
- support multiple julia releases _on separate git branches_ of a package. Think of this as the solution using `git`.

The advantage of supporting both 0.6 and 1.0 on one branch is that users of 0.6 automatically continue to get new features and bug fixes. The disadvantage of supporting both 0.6 and 1.0 on one branch is that we’ve had quite a few problems where well-intentioned maintainers inadvertently ended up breaking 0.6 (sometimes in subtle ways that depend on interactions between packages that don’t easily show up in tests).

I am personally persuaded by two points:

- unlike 0.6, 1.0 is a long-term release—it’s appropriate to focus most development activity on it, and the upgrade is hard enough that there’s no need to make extra busy work for contributors when, for many, 0.6 is or soon will be a distant memory
- `git cherry-pick` generally makes it easy to “copy” a commit from one branch to another, as long as a maintainer cares enough to do so

Thus I tend to think that leveraging `git`—rather than `Compat.jl`—is the best default choice in most circumstances. But of course it can vary on a package-by-package basis.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [September 20, 2018, 10:52am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/113 "2018-09-20T10:52:28Z")

</div>

> [@tim.holy](#):
>
> we still support 0.6 in the package ecosystem

Technically yes, of course, as package versions that were installable on 0.6 remain so. But given the fast pace some Julia packages are moving at, and the limited resources of maintainers, _de facto_ a lot of packages don’t backport.

Note that I am not saying this is a bad thing, it may be optimal allocation of scarce resources. Just pointing out the inconsistency between the two positions (which, again, is perfectly fine, as the two posts I quoted were written by two different people).

---

<div class="post-metadata">

**Author:** ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)\
**Post date:** [September 20, 2018, 11:46am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/114 "2018-09-20T11:46:22Z")

</div>

Standard Lua is a very nice little language, doesn’t include the kitchen sink, incredibly small with decent performance for a interpreter. It was the first computer language that my kids learned (via the ComputerCraft mod for MineCraft, years ago, and the Codea iPad app [I wish there were a Julia app like that 😀])  
It’s perfect as an embedded scripting language (it’s used as such in a number of games, as well as in things like the AeroSpike distributed database [for user defined functions, that get run on each member of a cluster])  
It’s 30+ years old, and in some places shows it (like many languages defined before Unicode, and not revamped like Python was in version 3.x).

---

<div class="post-metadata">

**Author:** ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)\
**Post date:** [September 20, 2018, 11:50am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/115 "2018-09-20T11:50:28Z")

</div>

What’s the best way of splitting off a separate v0.6.x branch of (a large number) of packages in an ecosystem,  
in such a way that v0.6.x users will continue to get subsequent tagged releases in that branch?  
(I’d like to do so for the Str\* packages in [JuliaString.org](http://JuliaString.org), and forget about v0.6.4 [except for backporting bugs to that branch if asked to])

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [September 20, 2018, 12:03pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/116 "2018-09-20T12:03:50Z")

</div>

Also, some packages seem to be left behind like JLD.jl. Currently, there seems to be no activity from the maintainers to make JLD.jl work on 1.0 but at the same time JLD2.jl says it shouldn’t be used in serious data scenarios because of potential data loss. Featurewise, it also isn’t a simple replacement for JLD.jl.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [September 20, 2018, 2:14pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/117 "2018-09-20T14:14:46Z")

</div>

You can “pro-actively” fork off a `julia-0.6` branch in each of these repositories, and do patch-version bumps (presuming you’ve kept a minor version number difference between 0.6 and 0.7). Of course, you can also always add that `julia-0.6` branch later when you discover you need it, e.g., when you want to supply a bugfix: you just go back to the commit before requiring julia 0.7, create the branch, and then cherry pick the commits you want to backport.

---

<div class="post-metadata">

**Author:** ![KZiemian](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kziemian/32/9020_2.png) [@KZiemian](https://discourse.julialang.org/u/KZiemian)\
**Post date:** [September 20, 2018, 7:20pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/118 "2018-09-20T19:20:24Z")

</div>

This topic is very long now and span over a year and half. Can we summarized results of current discussion in some concise way?

---

<div class="post-metadata">

**Author:** ![NiclasMattsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/niclasmattsson/32/21988_2.png) [@NiclasMattsson](https://discourse.julialang.org/u/NiclasMattsson)\
**Post date:** [September 20, 2018, 7:32pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/119 "2018-09-20T19:32:19Z")

</div>

Ooo! I have an idea! Let’s put a button just below the first post that says “Summarize This Topic” and does the job automatically! Would that be hard to implement? 🙂

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [September 20, 2018, 7:40pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/120 "2018-09-20T19:40:10Z")

</div>

The [marked solution](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/10) is still a great answer to the original question. This thread has meandered in many ways since then — largely discussing the relative strengths/weaknesses of the languages and ecosystems.

---

<div class="post-metadata">

**Author:** ![arnuld](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/arnuld/32/5454_2.png) [@arnuld](https://discourse.julialang.org/u/arnuld)\
**Post date:** [September 21, 2018, 2:28am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/121 "2018-09-21T02:28:43Z")

</div>

One hell of an article John 👌

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [September 21, 2018, 8:29am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/122 "2018-09-21T08:29:16Z")

</div>

> [@StefanKarpinski](#):
>
> I guess I can give some historical motivation here. When we started working on Julia in 2009, the Python ecosystem was nowhere near where it is today:
> 
> - PyPy was pretty new: 1.0 was released in 2007 – it was the first version to be able to run any substantial Python apps and there were no plans at the time for supporting any of NumPy;
> - Cython was even newer: initial release was 2007;
> - Numba didn’t exist: initial release was in 2012.

I ran in to NumExpr again today, it was last recommended to me in hmm 2013?

> **[GitHub - pydata/numexpr: Fast numerical array expression evaluator for...](https://github.com/pydata/numexpr)**
>
> Fast numerical array expression evaluator for Python, NumPy, PyTables, pandas, bcolz and more - GitHub - pydata/numexpr: Fast numerical array expression evaluator for Python, NumPy, PyTables, panda...

NumExpr is another of Julia’s contemporaries.  
An a way to accellerate math in python.  
It was created in 2007,

NumExpr is kinda hideous, in that you need to wrap all your math in strings,  
so that you can accelerate it.  
To be fair we’ve been known to do things not dissimilar using string macros.  
But that was a thing people were still kinda excited about doing in 2009.  
That was considered a potential way to have less painful fast math in python…

[Previous page](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236.md?page=5)

[Next page](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236.md?page=7)