# How hard would it be to implement Numpy.jl, i.e. Numpy in Julia?

**URL:** <https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080>\
**Category:** Numerics\
**Tags:** faq, python\
**Created:** [March 19, 2019, 10:48pm UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080 "2019-03-19T22:48:54Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pierre\_Augier](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pierre_augier/32/7506_2.png) [@Pierre\_Augier](https://discourse.julialang.org/u/Pierre_Augier)\
**Post date:** [March 19, 2019, 10:48pm UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/1 "2019-03-19T22:48:54Z")

</div>

Hi,

I’m working with Pythran ([http://github.com/serge-sans-paille/pythran](http://github.com/serge-sans-paille/pythran)), a Python/Numpy to C++ transpiler, which of course supports only a subset of Python (in particular not all the crazy cool Python stuffs stupid for performance). With Pythran, you can prototype in Python/Numpy and get very efficient C++ which does not use the Python interpreter.

We were thinking about the possibility to implement a Julia backend for Pythran, i.e. to be able to transpile the subset of Python/Numpy supported by Pythran in Julia.

The advantage of having such backend would be:

- people coding in Python could start to use Julia with nearly no code modification (which would help them to then really use Julia).
- most numerical kernels of Python codes could be translated in Julia.

Under the hood, Pythran developers wrote a clone of Numpy in C++. Doing the same in Julia would be the most difficult task to implement this Julia backend for Pythran.

So my question is: do you think it would be possible and doable to implement a clone of Numpy in Julia (at least the core of the API, since Numpy API is large). To be clearer, the Numpy clone has to be written only in Julia and has be able to return what the real Numpy function would return (it’s slightly more complicated than that in terms of types, as we can see for example with Cupy, another Numpy clone, but anyway).

I guess it’s a huge work, but I also guess there are equivalents of basically all Numpy functions in Julia, so I would be interested to have points of view of people knowing Julia well.

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [March 19, 2019, 11:23pm UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/2 "2019-03-19T23:23:22Z")

</div>

Use PyCall? With the new syntax of PyCall, Python code can be copy and pasted in many cases I believe.

---

<div class="post-metadata">

**Author:** ![Zach\_Christensen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zach_christensen/32/7220_2.png) [@Zach\_Christensen](https://discourse.julialang.org/u/Zach_Christensen)\
**Post date:** [March 19, 2019, 11:23pm UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/3 "2019-03-19T23:23:40Z")

</div>

If someone is willing to put that much work into something I’d think it would be more beneficial to do a comprehensive set of benchmarks that show users how to get the same method (svd, transpose, etc) in each language.

---

<div class="post-metadata">

**Author:** ![Diego\_Javier\_Zea](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/diego_javier_zea/32/1858_2.png) [@Diego\_Javier\_Zea](https://discourse.julialang.org/u/Diego_Javier_Zea)\
**Post date:** [March 20, 2019, 12:42am UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/4 "2019-03-20T00:42:43Z")

</div>

There are some mappings here: [https://cheatsheets.quantecon.org/](https://cheatsheets.quantecon.org/)

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [March 20, 2019, 12:46am UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/5 "2019-03-20T00:46:38Z")

</div>

> [@mohamed82008](#):
>
> Use PyCall? With the new syntax of PyCall, Python code can be copy and pasted in many cases I believe.

If the idea is to compile Python into Julia for speed, using PyCall sounds like it defeats the purpose.

> [@Zach\_Christensen](#):
>
> I’d think it would be more beneficial to do a comprehensive set of benchmarks that show users how to get the same method (svd, transpose, etc) in each language.

Many NumPy builds come with MKL, while unless you built Julia from source with MKL, it will have OpenBLAS. MKL is faster on many Intel architectures, especially those with avx-512 (but OpenBLAS support for avx512 is improving).  
Therefore, these benchmarks are not necessarily going to be favorable to Julia, even compared to regular Python + NumPy.

The more actual code relative to BLAS calls involved, the better Julia will do. Loops, computational kernels, etc, optimizing or running MCMC on a function you wrote in Julia or Python, etc.

For performance, the transpiler also ought to be smart about using StaticArrays instead of base arrays, and applying `@simd` and `@inbounds`.  
A pure Julia workflow is probably easier. Test your code. If it gets the correct answer and doesn’t have bounds errors, add `@inbounds` and friends. I guess the Python workflow would be similar: test in Python, and then transpile to Julia with those optimizations.

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [March 20, 2019, 2:48am UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/6 "2019-03-20T02:48:50Z")

</div>

Is it possible and doable? Absolutely. Will it be hard? Yup. Is there enough of a benefit to make the major engineering effort worth the time and money? Probably not.

Folks keep subsetting and/or adapting Python in various ways to improve its performance. There’s a reason for this — the full Python semantics are precisely what keep it from going fast. Any such transpiler into Julia will also need to be a subset of Python if it is to improve on performance. So there’s suddenly a third “language” one must learn in order to use this transpiler. Perhaps it could use the same subset as Pythran, but then why not just use Pythran?

If you want to start using Julia with an existing Python codebase, just use Julia! You can use PyCall (or even ccall into Pythran-compiled libraries) to interface with your existing codebase.

---

<div class="post-metadata">

**Author:** ![Pierre\_Augier](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pierre_augier/32/7506_2.png) [@Pierre\_Augier](https://discourse.julialang.org/u/Pierre_Augier)\
**Post date:** [March 20, 2019, 9:51am UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/7 "2019-03-20T09:51:33Z")

</div>

Hum, it seems that some Julia people are allergic to anything about the Python language 🙂 Do you know that it has also nice features and that it is widely used, also for numerical computing? This discussion could also be a test on how open-minded you are 🙂

I know about the nature of the Python language (too dynamic and with too much introspection for very efficient numerical computing), about some problems of the Python interpreters and about the issue of the border between the native and Python worlds. It is not necessary to tell me about these issues. It is really not the subject here. I won’t tell you about Julia’s issues neither.

The subset of Python / Numpy supported by Pythran is not a third language. It is what people use in practice in numerical kernels written with Numpy / Scipy. There is nothing crazy in term of performance about this subset and it is actually very close to a widely used subset of Julia.

The C++ Numpy clone in Pythran does not use the Python interpreter. It is pure C++ and it is very efficient (sometimes faster than Julia for high level code, even with all the `@simd` and `@inbounds` needed in Julia , and even just linked with OpenBlas).

But it could be interesting to have a Julia backend for Pythran, (1) to increase use of Julia by the Python developers / users (a lot of people), (2) for (maybe easier) GPU support, (3) for translation of Python numerical kernels in efficient Julia. Of course, using PyCall is not an option! It would be less efficient that what we get without Julia even just using Numpy!

Building an efficient Numpy.jl (in pure Julia) would be a great challenge and it would be useful (also for Julia). But of course, you can think that it is a waste of time to increase the interoperability between the two languages / frameworks and to attract Python users 🙂

If Julia could be (as modern C++) a good tool to accelerate numerical kernels written in Python / Numpy, great! (for Julia and for Python) Otherwise, there are other solutions to keep numerical Python competitive in terms of performance.

---

<div class="post-metadata">

**Author:** ![Pierre\_Augier](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pierre_augier/32/7506_2.png) [@Pierre\_Augier](https://discourse.julialang.org/u/Pierre_Augier)\
**Post date:** [March 20, 2019, 10:03am UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/8 "2019-03-20T10:03:55Z")

</div>

> If you want to start using Julia with an existing Python codebase, just use Julia! You can use PyCall (or even ccall into Pythran-compiled libraries) to interface with your existing codebase.

Sorry, it is not an option for me 🙂 If you like Julia, you may understand that I like Python, and that I will rather call Julia from Python than the other way around.

I am really not against coding in Julia when needed, for particular tasks. But for many other tasks, I think Python is really good, efficient and adapted.

I don’t think I am the only person thinking like this…

---

<div class="post-metadata">

**Author:** ![improbable22](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/improbable22/32/5464_2.png) [@improbable22](https://discourse.julialang.org/u/improbable22)\
**Post date:** [March 20, 2019, 10:30am UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/9 "2019-03-20T10:30:39Z")

</div>

> [@Pierre\_Augier](#):
>
> (1) to increase use of Julia by the Python developers / users (a lot of people), (2) for (maybe easier) GPU support,

You seem to be saying in that this kind of use of Julia in place of C++/Fortran/etc in the guts of some Python library would be good advertising, or a good way to convert users, but I don’t really see why. They would be unaware. If Julia’s GPU support offers an easier way to engineer the library you want, (2), then obviously that’s great, and if you contribute stress tests & bug-fixes then there are benefits all round.

> [@Pierre\_Augier](#):
>
> Building an efficient Numpy.jl (in pure Julia) would be a great challenge and it would be useful

Are you picturing a pure-Julia wrapper for base-Julia commands? This could be an _exceptionally_ thin wrapper.

Unless you are picturing it providing Numpy-like syntax, is that the point? A bit like how PyCall overloads dots to look like Python. This could be done, and may have benefits for letting people copy-paste a function body from old Python code you trust into new Julia code, and get a version which runs.

But this seems a separate concern to the use of Julia to write guts of a Python library.

---

<div class="post-metadata">

**Author:** ![davidbp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidbp/32/463_2.png) [@davidbp](https://discourse.julialang.org/u/davidbp)\
**Post date:** [March 20, 2019, 10:47am UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/10 "2019-03-20T10:47:57Z")

</div>

> [@Pierre\_Augier](#):
>
> - people coding in Python could start to use Julia with nearly no code modification (which would help them to then really use Julia).
> - most numerical kernels of Python codes could be translated in Julia.

I think the proper way to put it is “people coding in Pythran” not Python programmers. I have never used Pythran yet I use Python everyday. I have used though Cython, loopy, and other projects to compile python code. The difficulty of working in such ecosystem made me feel frustrated enough to want to start learning Julia. I actually find the multidimensional Arrays already in the language are pretty numpy looking.

Sometimes I miss certain functions (or functionality) from numpy. Maybe a list of functions we miss from numpy could help developers to give us some of the high level methods we love and miss. One of those methods in my case is `np.unique(X,return_counts=True)`.

---

<div class="post-metadata">

**Author:** ![Pierre\_Augier](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pierre_augier/32/7506_2.png) [@Pierre\_Augier](https://discourse.julialang.org/u/Pierre_Augier)\
**Post date:** [March 20, 2019, 10:55am UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/11 "2019-03-20T10:55:02Z")

</div>

> You seem to be saying in that this kind of use of Julia in place of C++/Fortran/etc in the guts of some Python library would be good advertising, or a good way to convert users, but I don’t really see why. They would be unaware.

They would not be completely unaware, because they would have to install Julia, which is already a good first step, isn’t it?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [March 20, 2019, 10:59am UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/12 "2019-03-20T10:59:46Z")

</div>

> [@Pierre\_Augier](#):
>
> Hum, it seems that some Julia people are allergic to anything about the Python language 🙂 Do you know that it has also nice features and that it is widely used, also for numerical computing? This discussion could also be a test on how open-minded you are 🙂

Despite the profusion of smileys, I find this kind of tone quite arrogant.

You asked about the viability of a project, and got reasonable answers from knowledgeable people. The fact that they don’t consider this project viable based on a cost/benefit analysis does not mean that they are not “open-minded”, merely that they would not invest time in it because they can use it for better things.

This is not because they don’t know Python — many Julia programmers used Python formerly, including numerical packages. A lot of experience from Python and similar languages has been incorporated to the design of Julia. Preferring Julia does not mean that one is “allergic to Python”, simply that one prefers Julia. In most cases this is because Julia was designed to address issues that cannot be dealt with in Python without a fundamental redesign of the language.

---

<div class="post-metadata">

**Author:** ![Pierre\_Augier](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pierre_augier/32/7506_2.png) [@Pierre\_Augier](https://discourse.julialang.org/u/Pierre_Augier)\
**Post date:** [March 20, 2019, 11:03am UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/13 "2019-03-20T11:03:00Z")

</div>

You know, methods in numerical computing change. In numerical Python, Cython, Numba and Pythran are improving. Now, it’s very easy to use Pythran inside Cython (and Cython is widely used). You are right that Pythran is not yet very popular, but it is growing.

---

<div class="post-metadata">

**Author:** ![Pierre\_Augier](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pierre_augier/32/7506_2.png) [@Pierre\_Augier](https://discourse.julialang.org/u/Pierre_Augier)\
**Post date:** [March 20, 2019, 11:20am UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/14 "2019-03-20T11:20:13Z")

</div>

> Despite the profusion of smileys, I find this kind of tone quite arrogant.

I’m very sorry for that. It was of course not my will. I’m really sorry that it was interpreted like this. Note that very hard opinions on Python from Julia users/developers could also be felt as arrogant sometimes.

My point was to say that I am not talking about the problematic parts of the Python language here. Again, I don’t think the Numpy API is crazy in terms of performance. You defined arrays and use them. It’s simple. Not too much dynamism or introspection.

Note that the fact that it can be translated to pure efficient C++ proves that it is not so horrible for performance.

> You asked about the viability of a project, and got reasonable answers from knowledgeable people.

It seemed to me that my question wasn’t correctly understood or considered seriously and that I add to reformulate it.

Again, it seems to me that answers based on “Python is bad for perf” are not “reasonable” (for this particular questions).

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [March 20, 2019, 11:28am UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/15 "2019-03-20T11:28:13Z")

</div>

> [@davidbp](#):
>
> `np.unique(X,return_counts=True)`

I think this would this be equivalent to:

```julia
using DataStructures
counter(X)

```

---

<div class="post-metadata">

**Author:** ![Zach\_Christensen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zach_christensen/32/7220_2.png) [@Zach\_Christensen](https://discourse.julialang.org/u/Zach_Christensen)\
**Post date:** [March 20, 2019, 11:31am UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/16 "2019-03-20T11:31:59Z")

</div>

Im a little confused, so that may be part of the problem. If you want to write something that calls existing Python interfaced code but from Julia PyCall shouldn’t be a problem. If you want to call Julia from Python use pyjulia. So I don’t think the problem here is with Python bit why have another step in-between the two?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [March 20, 2019, 11:37am UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/17 "2019-03-20T11:37:13Z")

</div>

> [@Pierre\_Augier](#):
>
> very hard opinions on Python from Julia users/developers could also be felt as arrogant sometimes

I find it best not to form an emotional attachment to a computer language, just treat it as a tool.

Julia can be considered an experiment to address various shortcomings of other languages, including Python, just like Python was created to improve on previous languages, and almost surely one day someone will design a language that supersedes Julia as we know now (it may also be called Julia, but be quite different—that is immaterial).

Discussing shortcomings and advantages of languages should not be considered arrogant, if kept at a technical level. You may find this discussion interesting:

> [@Preaching Julia to biologists](https://discourse.julialang.org/t/preaching-julia-to-biologists/15058):
>
> People, I’m scheduled to give a short (15 min) presentation about Julia to our group on next Tuesday! The crowd is composed of (~15) biologists that work in Animal Behavior (but not in Systems Biology) with a bit of experience in Matlab and R. The goal of the talk is not to teach them Julia, but to convince them that Julia is worth using (and eventually learning). I’m putting something together now, but it would be fantastic if anyone that gave a similar talk could share with me theirs. pray …

---

<div class="post-metadata">

**Author:** ![Pierre\_Augier](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pierre_augier/32/7506_2.png) [@Pierre\_Augier](https://discourse.julialang.org/u/Pierre_Augier)\
**Post date:** [March 20, 2019, 11:40am UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/18 "2019-03-20T11:40:10Z")

</div>

> [@improbable22](#):
>
> Are you picturing a pure-Julia wrapper for base-Julia commands? This could be an _exceptionally_ thin wrapper.
> 
> Unless you are picturing it providing Numpy-like syntax, is that the point? A bit like how PyCall overloads dots to look like Python.

“_exceptionally_ thin wrapper”, I think it would be a little bit more work than that… But yes, it would be a Julia wrapper around Julia functions mimicking the behavior of Numpy (which is most of the time quite reasonable).

The dot notation is really not the issue for a transpiler. It is not at all a problem if we have to transpile `my_arr.sum()` to `np_array_method_sum(my_arr)`.

Something much more important (and maybe harder?) is that the arrays have to behave like Numpy arrays: zero-based indexing and by default continuous memory in the last index.

---

<div class="post-metadata">

**Author:** ![Zach\_Christensen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zach_christensen/32/7220_2.png) [@Zach\_Christensen](https://discourse.julialang.org/u/Zach_Christensen)\
**Post date:** [March 20, 2019, 11:47am UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/19 "2019-03-20T11:47:21Z")

</div>

Actually zero based indexing would be pretty easy. Julia allows custom indexing behavior. Look at `strides` in the docs for memory layout stuff.

---

<div class="post-metadata">

**Author:** ![sdanisch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sdanisch/32/1406_2.png) [@sdanisch](https://discourse.julialang.org/u/sdanisch)\
**Post date:** [March 20, 2019, 11:47am UTC](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080/20 "2019-03-20T11:47:22Z")

</div>

The problem of the thread is: how hard is it to implement Numpy in Julia - which @mbauman answered. Since he wrote large parts of Julia’s array code, I’d trust his judgement 😉

If you further want to convince the Julia community of your project and maybe hope to nerdsnipe someone to start writing Numpy.jl, I’m predicting that you will have a hard time.

Not because we are allergic to Python, but because the Julia users here found their solution with Julia and are rarely looking back to numpy syntax.

So, you’re trying to pitch a project, that won’t improve the life of any Julia user. This is a tough sell! Even if it may improve Julia adoption, there is almost no one here who has the luxury to work on a project that won’t benefit them directly. Well, maybe there is someone here who has to port lots of python code to Julia, but if you want to find those people, you should probably reformulate the question 😉

If you decided this project is worthwhile, you’ll likely be the one to implement it 😉  
So the next step should be to start Numpy.jl, and maybe nerd snipe some Julia users to help with that with some more concrete issues 😉

[Next page](https://discourse.julialang.org/t/how-hard-would-it-be-to-implement-numpy-jl-i-e-numpy-in-julia/22080.md?page=2)