# How hard would it be to implement Numpy.jl, i.e. Numpy in Julia?

**URL:** <https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080>\
**Category:** Numerics\
**Tags:** faq, python\
**Created:** [March 19, 2019, 10:48pm UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080 "2019-03-19T22:48:54Z")\
**Posts on this page:** 1\
**Showing post:** 37

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [March 20, 2019, 6:32pm UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/37 "2019-03-20T18:32:05Z")

</div>

> [@Pierre\_Augier](#):
>
> I can assure you that numerical kernels transpiled in Numpy.jl could be much faster than pure Python / Numpy. It’s pretty easy to beat Numpy code with all the temporary objects (and of course as soon as there are loops). I don’t think it would be faster than Pythran on CPU (but the jit warmup would be faster). I strongly suspect it could be faster than Numba. Moreover, Numpy.jl could also be interesting for GPU support.

As @sdanisch said, I was comparing to what Pythran/Numba etcetera do now. The limiting factor here is Python semantics — you can’t fuse loops as an optimization unless the compiler (or transpiler) can _prove_ that it won’t change the results. In practice, this can only be done for a small set of types and functions recognized by the compiler, which is why things like Pythran and Numba are effectively limited to numpy arrays and a particular set of library functions on those arrays, and get stymied if they see a call to arbitrary Python code. A Julia transpiler or backend would have zero advantage here.

In fact, I wrote a long blog post explaining [Julia’s approach to loop fusion for vectorized operations](https://julialang.org/blog/2017/01/moredots), the challenges of this problem, and how we are able to do it for generic user-defined functions and types. But the advantage we have here is lost if the front-end code is Pythonic.

> [@Pierre\_Augier](#):
>
> Several numerical kernels written in Numpy are “fully type-generic code”, so they can be transpiled in fully type generic Julia code.

You can write type-generic Python code, but you can’t compile it to fast code in general, because the semantics of the language do not allow it. This is why Pythran, Numba, etcetera can only do a good job on a very small subset of the language — only one container type (numpy arrays), only numpy scalar types (or specially annotated struct-like types using language extensions), and very limited polymorphism.

> [@Pierre\_Augier](#):
>
> About the limitation “to Python/Pythran/etc semantics”, I have to say that I don’t understand. Algorithms written in Python / Numpy are fully described. There is the same amount of information as for many other languages, including Julia for many cases.

The question is how much information is easily accessible _at compile time_. Even though every program in every computer language is “fully described” in the sense of prescribing a deterministic sequence of actions (modulo stochastic algorithms and undefined behaviors), there is often lots of information that is [nearly impossible to get _in advance_](https://en.wikipedia.org/wiki/Halting_problem) without actually running the code.

Basically, Julia is designed to provide more information at compile time than traditional dynamic languages. This occurs because of lots of properties of the language and standard library that Python doesn’t have — type-stable libraries, final concrete types, parameterized immutable types, and so on.

Realize that people have tried for years now to develop an effective general-purpose compiler for Python. It’s a hard problem! Why do you think that projects like Numba and Pythran have targeted such a small subset of the language?

The backend of a Python compiler is the easy part, and something that a Julia transpiler would add nothing to — the challenging part is the front-end analysis.

---

_[View the full topic](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080)._