Julia motivation: why weren't Numpy, Scipy, Numba, good enough?

[ChrisRackauckas]

@DNF

Numba now has the capability of compiling a whole class with the @jitclass decorator. I was going to try it out, but it requires quite a lot of typing for large classes, so I gave up for the time being.

@giordano

That said, I don’t use Python (let alone Numba), so please do correct me if I’m wrong, but the answer to the “Why not Numba?” question should be “generality”. As far as I understand, Numba works well only with standard data types, is this correct? Instead, Julia has no performance penalty with custom types and the interplay between different custom (and well-designed) types is often very easy, without the need of specific glue code (and as far I can see this is very often false in Python).

http://numba.pydata.org/numba-doc/dev/user/jitclass.html

Yes, generality is the key. Numba can get you going for Float64. And it can now, in very limited cases, get you JIT compiled objects, but that’s not even getting close to Julia’s types. I believe you cannot write your JIT functions to auto-specialize according to input types, so generic programming is out of the question. This only works on a small class of objects with limitations.

But more importantly: Python does not have a good language for talking about types. Numba is trying to bolt on features that get it closer to Julia, but it’s missing the language which makes it easy to actually use and discuss. For example, Julia has parametric types. Parametric types can be “infinitely many” different JIT compiled classes, and many times you want to dispatch differently depending on these type parameters. With what exists in Numba, you technically “can” do it, but it’s all manual and at that point you might as well be writing C++ (or… Julia).

So all of the examples that people give for “but Numba can do it” tend to be “here’s an example looping on Float64”. Yes, bravo, Numba got that. But what I have been finding out in my journey with Julia is that, that’s the simplest case (and you might as well just use C/Fortran if that’s all you want).

What’s interesting about Julia is that same exact code is efficient for arbitrary arithmetic, or AbstractArrays (and gets you auto-compilation of your functions to GPU variants using GPUArrays for example). Numba can keep bolting on what’s needed. It has bolted on stuff for GPUs. If they see that people are using Julia’s generic algorithms with DistributedArrays, they can make a distributed array and change the compiler so that case will work. But Julia is designed correctly so that way these kinds of specializations aren’t “top-down”: no compiler changes are needed. You can do all of this by adding a package.

In the end, I am sure that with enough work Numba can keep trying to keep up with “the standard use cases” of Julia that are beyond Float64, but it’ll still be in a language that has no way to discuss what it’s actually doing, and it’ll still be “given to you” by compiler changes in Numba itself, and making changes to add stuff like this won’t be accessible to standard Python developers without compiler knowledge.