# Julia motivation: why weren't Numpy, Scipy, Numba, good enough?

**URL:** <https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236>\
**Category:** Community\
**Tags:** history\
**Created:** [February 22, 2017, 3:34pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236 "2017-02-22T15:34:33Z")\
**Posts on this page:** 20\
**Page:** 5

<div class="post-metadata">

**Author:** ![RoyiAvital](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/royiavital/32/571_2.png) [@RoyiAvital](https://discourse.julialang.org/u/RoyiAvital)\
**Post date:** [September 13, 2018, 5:03pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/83 "2018-09-13T17:03:24Z")

</div>

Well, I thought it was good as I learned from it thing or two.  
What do you say about the LLVM example?

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [September 13, 2018, 5:32pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/84 "2018-09-13T17:32:25Z")

</div>

> [@RoyiAvital](#):
>
> What do you say about the LLVM example?

It’s wrong. He has an error in his code and is missing a `return` statement, so the generated assembly is gibberish. [Fix that](https://gcc.godbolt.org/z/6LmW7I), and you’ll see that the assembly for `add3` still allocates _two_ vectors (and not just the one he hopes for).

---

<div class="post-metadata">

**Author:** ![RoyiAvital](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/royiavital/32/571_2.png) [@RoyiAvital](https://discourse.julialang.org/u/RoyiAvital)\
**Post date:** [September 13, 2018, 5:39pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/85 "2018-09-13T17:39:37Z")

</div>

Oh…  
I had hopes :-).

You should point it out in StackOverflow.  
As I thought he found gold.

---

<div class="post-metadata">

**Author:** ![josemaria](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/josemaria/32/5365_2.png) [@josemaria](https://discourse.julialang.org/u/josemaria)\
**Post date:** [September 18, 2018, 5:44am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/86 "2018-09-18T05:44:07Z")

</div>

I love python (it get things done) and I am starting to love Julia (I think my head deals better with functional programming than with OOP).

Regarding Julia’ motivation (title of this thread), was Lua not fast enough?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [September 18, 2018, 8:48am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/87 "2018-09-18T08:48:16Z")

</div>

> [@josemaria](#):
>
> Regarding Julia’ motivation (title of this thread), was Lua not fast enough?

Lua’s fast enough, but it doesn’t have the same generic programming constructs that Julia users make use of so it cannot really replace modern Julia.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [September 18, 2018, 3:33pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/88 "2018-09-18T15:33:26Z")

</div>

> [@ChrisRackauckas](#):
>
> Lua’s fast enough

[LuaJIT](https://luajit.org) is also not the standard Lua implementation, [this is](https://www.lua.org/download.html) and they are separate projects. LuaJIT existed in 2009, but was a fairly niche project and only supported 32-bit x86, as far as I’m aware. It now supports a number of other architectures and word sizes, but I think that’s a pretty recent development.

Edit: the [sponsorship page](https://luajit.org/sponsors.html) is a pretty good way to see when ports happened since many of the sponsorships seem to be for porting LuaJIT to various platforms. It seems that the work on the first non-x86-32 port for LuaJIT started in December 2009 and wasn’t completed until 2010 at the earliest (since they got the largest chunk of sponsorship from Google in January that year).

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [September 18, 2018, 3:40pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/89 "2018-09-18T15:40:46Z")

</div>

[LuaJIT](https://luajit.org/) is looking for maintainers. Could be in trouble?

Also, how’s the speed of the [standard Lua](https://www.lua.org)? The benchmarks do not include that, do they?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [September 18, 2018, 3:51pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/90 "2018-09-18T15:51:38Z")

</div>

> [@StefanKarpinski](#):
>
> [LuaJIT](https://luajit.org) is also not the standard Lua implementation, [this is](https://www.lua.org/download.html) and they are separate projects. LuaJIT existed in 2009, but was a fairly niche project and only supported 32-bit x86. It now supports a number of other architectures and word sizes, but I think that’s a pretty recent development.

Yeah I meant LuaJIT (but for reasons mentioned below, Lua and LuaJIT aren’t necessarily the same language 🤷‍♂️)

And there were a lot of GC issues that one can point to that seem to be part of the architecture and design which limit the full use of heap-allocated arrays (which ended up being one of the downfalls of LuaJIT). They weren’t actually fixed until LuaJIT 3.0 in 2017

> **[The LuaJIT Project](https://luajit.org/)**

But LuaJIT 3.0 is very much not backwards compatible with the previous versions and some of the disputes coincided with the main developer leaving.

> [@PetrKryslUCSD](#):
>
> [LuaJIT](https://luajit.org/) is looking for maintainers. Could be in trouble?

Mike Pall left in 2015.

> **[\[ANN\] Looking for new LuaJIT maintainers - luajit - FreeLists](https://www.freelists.org/post/luajit/Looking-for-new-LuaJIT-maintainers)**
>
> \[ANN\] Looking for new LuaJIT maintainers, luajit at FreeLists

There’s enough that LuaJIT 3.0 was able to make it release. Some of the issues were that Lua had its own releases and IIRC LuaJIT is not and somewhat cannot be compatible with the newest Lua, which is some of the 5.3 vs 5.2 vs 5.1 discussions you’ll find on some of the old threads about the departure.

> **[Reddit - Heart of the internet](https://www.reddit.com/r/programming/comments/3gin5e/luajits_main_developer_is_retiring_and_is_looking/?rdt=56228)**

(Honestly, these kinds of issues are big reason to say that the JIT needs to be part of the language’s design)

I don’t think there’s much of a reason to benchmark pure Lua since you can think of it as “dictionary-oriented programming”, with the kind of speed you’d expect from that. But, the fact that a tracing JIT could get that sped up the way it did was truly a remarkable feat which showed the power of JITs long before Julia was a big popular project.

(I think it would be awesome to use Cassette to build a tracing JIT into Julia code in a way that acts like LuaJIT and see what happens…)

---

<div class="post-metadata">

**Author:** ![Tero\_Frondelius](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tero_frondelius/32/7629_2.png) [@Tero\_Frondelius](https://discourse.julialang.org/u/Tero_Frondelius)\
**Post date:** [September 18, 2018, 4:59pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/91 "2018-09-18T16:59:41Z")

</div>

> [@ChrisRackauckas](#):
>
> I think it would be awesome to use Cassette to build a tracing JIT into Julia code in a way that acts like LuaJIT and see what happens…

I had to read more. Here is a link if others are interested: [Tracing just-in-time compilation - Wikipedia](https://en.m.wikipedia.org/wiki/Tracing_just-in-time_compilation)

Would this tracing JIT enable faster than C implementations in Julia? I understand this quickly go some very deep compiler optimization discussion, but in principle can this method make Julia code faster than C?

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [September 18, 2018, 5:04pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/92 "2018-09-18T17:04:49Z")

</div>

It’s already possible to write Julia code that’s faster than C, e.g. by using SIMD and using the fact that we specialize on function arguments. I wouldn’t hold out too much hope for tracing JIT tricks being that much faster than well-written C that already leverages SIMD and doesn’t have higher-order functions. People have been hoping for that for a long time and it hasn’t really materialized outside of some particular cases here and there. Otherwise, you’d see Java beating C speed all the time, which is not a thing that happens much.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [September 18, 2018, 5:07pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/93 "2018-09-18T17:07:50Z")

</div>

> [@Tero\_Frondelius](#):
>
> Would this tracing JIT enable faster than C implementations in Julia? I understand this quickly go some very deep compiler optimization discussion, but in principle can this method make Julia code faster than C?

What @StefanKarpinski said is basically the answer. People tried a lot of tracing JITs before Julia and LuaJIT was the one that did well and basically can match C, with others like those for Javascript getting within 5x. However, I would be more optimistic than Stefan because tracing JITs have done some things really well and other things less well. Slapping one on top of some Julia code might be a cool way to mix them and get the best of both worlds. Of course, I wouldn’t expect well-written type-stable SIMD code to do better, but I think you could get some massive improvements in things like type-unstable code or things with arrays that aren’t strictly typed, and there may be some areas where a tracer kicks in and skips the end of the computation by finding out an analytical solution that may not have been available at compile time.

It would be quite a research project to find out how to make this useful though, and probably contained to a few tricks.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [September 18, 2018, 5:13pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/94 "2018-09-18T17:13:24Z")

</div>

> [@ChrisRackauckas](#):
>
> Of course, I wouldn’t expect well-written type-stable SIMD code to do better, but I think you could get some massive improvements in things like type-unstable code or things with arrays that aren’t strictly typed, and there may be some areas where a tracer kicks in and skips the end of the computation by finding out an analytical solution that may not have been available at compile time.

This sounds a bit like a mix of the proverbial “sufficiently clever compiler” and something that is able to discover that some `Vector{Any}` actually only ever holds `Float64` values and so could be specialized `Vector{Float64}`. Which sounds like more of a use case for a tracing linter than a tracing JIT. I don’t believe in sufficiently clever compilers but a tracing linter seems like it would be quite useful to have.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [September 18, 2018, 5:32pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/95 "2018-09-18T17:32:59Z")

</div>

> [@StefanKarpinski](#):
>
> This sounds a bit like a mix of the proverbial “sufficiently clever compiler” and something that is able to discover that some `Vector{Any}` actually only ever holds `Float64` values and so could be specialized, which sounds like more of a use case for a tracing linter than a tracing JIT.

Indeed. I didn’t think about a tracing linter though: “did you mean `Vector{eltype(eltype(A.x))}` instead of `Vector{Any}`?” would be quite useful for concrete typing in generic programming.

---

<div class="post-metadata">

**Author:** ![ahojukka5](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ahojukka5/32/4376_2.png) [@ahojukka5](https://discourse.julialang.org/u/ahojukka5)\
**Post date:** [September 19, 2018, 7:00am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/96 "2018-09-19T07:00:37Z")

</div>

Tracing linter especially if it is implemented in the same style than femtocleaner, i.e. it checks automatically Julia projects and makes pull requests to improve code performance.

---

<div class="post-metadata">

**Author:** ![charlesll](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/charlesll/32/3255_2.png) [@charlesll](https://discourse.julialang.org/u/charlesll)\
**Post date:** [September 19, 2018, 7:17am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/97 "2018-09-19T07:17:38Z")

</div>

To add my 2 cents to this discussion and some feedback about my recent experience of Julia vs Scipy/Numpy:

I used to use a lot Julia 0.4/0.5/0.6 as Numpy/Scipy were too slow for some of my applications, ~ 2 years ago. I had a great time, really, and firmly believed in Julia’s bright future. In particular, I think Julia is very nice when it comes down to matrix/tensor manipulation. However, I feel Numpy/Scipy gained in speed, such that I usually don’t see much difference nowadays between using Python or Julia.

Now, the dark side. In my opinion, we currently face a problem of numerous packages not working at all in Julia 0.7/1.0, which gives a terrible user experience. Some of my users stopped using Julia because of that, and because of many problem of Julia’s installation. On my side, since the past 6 months, I admit that I automatically go to Python for developing new code, because solving problems between Julia 0.6-0.7/1.0 have become a nightmare…

I’m a bit sorry about that, but I think Julia may loose new users because of the problem of many packages not working in 0.7/1.0 yet… And not some of the least. For instance, I can’t get Mamba, NMF, or even Compose to work from the registered stream, I needed to install them directly from the repo. A new user may not necessarily realize that, driving him away from Julia, and toward Scipy/Numpy and the Python ecosystem which now is very stable and easy to use/install.

---

<div class="post-metadata">

**Author:** ![tobias.knopp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tobias.knopp/32/7551_2.png) [@tobias.knopp](https://discourse.julialang.org/u/tobias.knopp)\
**Post date:** [September 19, 2018, 8:41am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/98 "2018-09-19T08:41:53Z")

</div>

> On my side, since the past 6 months, I admit that I automatically go to Python for developing new code, because solving problems between Julia 0.6-0.7/1.0 have become a nightmare…

That makes no sense. Julia has been released a month ago, why have you upgraded 6 months ago where it was crystal clear that almost any package was not working with 0.7 / 1.0?

Maybe you could look a little bit into history

- when has Python 3 been released?
- when has Numpy/Scipy released a version that is compatible with Python 3?

I am pretty certain that the delta is larger than one month 😉

This is just to make clear that the package transition from 0.6 to 1.0 was an huge amount work. I have ported three packages (Gtk, Winston, NFFT) and have helped fixing things in HDF5 and Cairo. If you are seeing packages not working, the best thing is either to stay on 0.6 (which is rock stable) or help porting things to 1.0.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [September 19, 2018, 9:30am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/99 "2018-09-19T09:30:37Z")

</div>

> [@tobias.knopp](#):
>
> Maybe you could look a little bit into history
> 
> - when has Python 3 been released?
> - when has Numpy/Scipy released a version that is compatible with Python 3?

I could not resist doing this 😉

1. [Numpy 1.5.0](https://docs.scipy.org/doc/numpy/release.html#numpy-1-5-0-release-notes) appears to be the first release supporting Python 3.
2. It was [released on 2010-08-31](https://sourceforge.net/projects/numpy/files/NumPy/1.5.0/).
3. [Python 3 was released on 2008-12-03](https://www.python.org/download/releases/3.0/).

Someone please correct these if I misunderstood something.

---

<div class="post-metadata">

**Author:** ![tobias.knopp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tobias.knopp/32/7551_2.png) [@tobias.knopp](https://discourse.julialang.org/u/tobias.knopp)\
**Post date:** [September 19, 2018, 9:37am UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/100 "2018-09-19T09:37:02Z")

</div>

Thanks. Just to make this clear: I think that the Numpy developers did a great job when upgrading from Python 2 to Python 3 and the transition went smoothly for me. But: I simply used Python 2 until there was the official release of Numpy that worked with Python 3.

---

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [September 19, 2018, 12:05pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/101 "2018-09-19T12:05:20Z")

</div>

Adding to the list: sagemath has no python 3 compatible release yet.

But yeah, 0.6 is imho still the more reliable version, but pointing new users there doesn’t feel right either.

---

<div class="post-metadata">

**Author:** ![tobias.knopp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tobias.knopp/32/7551_2.png) [@tobias.knopp](https://discourse.julialang.org/u/tobias.knopp)\
**Post date:** [September 19, 2018, 12:20pm UTC](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/102 "2018-09-19T12:20:09Z")

</div>

> [@foobar\_lv2](#):
>
> but pointing new users there doesn’t feel right either

At this time I tend to agree. It seems that all “important” packages have been migrated. But this is pretty subjective.

[Previous page](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236.md?page=4)

[Next page](https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236.md?page=6)