# Benchmark for latest julia?

**URL:** <https://discourse.julialang.org/t/benchmark-for-latest-julia/5991>\
**Category:** Community\
**Tags:** question\
**Created:** [September 20, 2017, 2:47am UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991 "2017-09-20T02:47:24Z")\
**Posts on this page:** 20\
**Page:** 3

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [September 29, 2017, 10:40am UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/41 "2017-09-29T10:40:23Z")

</div>

> [@ChrisRackauckas](#):
>
> It seems like Julia users who’ve been around for a day would know about that

While what you suggest should be done, this highlights a general issue with benchmarks like this: in order to get a fair assessment, we should ask experienced users of _all_ of these languages to optimize their code.

The [Computer Language Benchmarks Game](http://benchmarksgame.alioth.debian.org/) has seen a lot of cycles that go like this:

1. users of language A optimize their toy benchmark,
2. language B is now behind, and someone with time on their hands optimizes _that_ benchmark,
3. the language C community is now miffed, to the point that someone actually tweaks their interpreter to deal better with the benchmark,
4. …

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [September 29, 2017, 11:33am UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/42 "2017-09-29T11:33:23Z")

</div>

> [@Tamas\_Papp](#):
>
> in order to get a fair assessment, we should ask experienced users of all of these languages to optimize their code.

When you benchmark, you have to decide what you are trying to measure.

This particular benchmark is not supposed to test the most optimized possible code for a given problem. e.g. an optimized `fib` would just use a look-up table and would be totally uninteresting as a benchmark. The point of this benchmark is to test the performance of common language constructs — e.g. looping, recursion, and even matrix multiplication, all written in the most “ordinary” way (which is why the Fortran one calls the built-in `matmul` and not `dgemm`, and similarly Julia calls `*` and not `A_mul_B!` or `BLAS.dgemm` to avoid allocations).

It is also useful to benchmark highly optimized code, of course, but that is a very different exercise and calls for very different sorts of problems.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [September 29, 2017, 11:52am UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/43 "2017-09-29T11:52:09Z")

</div>

I did not suggest changing the algorithm. My point was that if we optimize Julia code, an effort should be made to optimize the other code to the same extent (which, I recognize, is hard to quantify). Clearly, experienced Julia users are now working on this code, so to make a fair comparison, one would need to solicit help from experts in other languages (there are possibly many of them in the Julia community).

One can always argue that throwing `@inbounds` in front of something is an obvious optimization. But I am not an expert in the other languages, so I don’t know what the similar low hanging fruits are.

---

<div class="post-metadata">

**Author:** ![John\_Gibson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/john_gibson/32/5321_2.png) [@John\_Gibson](https://discourse.julialang.org/u/John_Gibson)\
**Post date:** [September 29, 2017, 12:00pm UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/44 "2017-09-29T12:00:10Z")

</div>

> [@stevengj](#):
>
> The point of this benchmark is to test the performance of common language constructs — e.g. looping, recursion, and even matrix multiplication, all written in the most “ordinary” way (which is why the Fortran one calls the built-in `matmul` and not `dgemm`, and similarly Julia calls `*` and not `A_mul_B!` or `BLAS.dgemm` to avoid allocations).

In that case we should remove the BLAS calls from the C code and leave Python, R, or Octave alone (pure language constructs, no BLAS).

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [September 29, 2017, 12:36pm UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/45 "2017-09-29T12:36:55Z")

</div>

> [@Tamas\_Papp](#):
>
> But I am not an expert in the other languages, so I don’t know what the similar low hanging fruits are.

Why not just qualify it? For Julia we do the basics as described here: [https://docs.julialang.org/en/stable/manual/performance-tips/](https://docs.julialang.org/en/stable/manual/performance-tips/) . For Python, do the basics as described for example here: [PythonSpeed/PerformanceTips - Python Wiki](https://wiki.python.org/moin/PythonSpeed/PerformanceTips) . Any of the basic “don’t do this in Python” is pretty much the same (otherwise they wouldn’t be basics everyone knows). It’s hard to establish a baseline, but I think this “code in the style like someone who looked at the first page that pops up on Google for ‘xxxx performance tips’” is something that would match “most decently informed people’s code” and is essentially what the Julia benchmarks are trying to hit with only a few changes (but of course the algorithms have to be the same, so `fib` is the recursive algorithm, etc.).

The Computer Language Benchmarks Game is hitting an entirely different audience where bitshift tricks, declaration of non-aliasing scopes, and other bizarre optimizations are fair game. That’s just completely different.

In any case, we should better clarify what our criteria is and stick to it.

---

<div class="post-metadata">

**Author:** ![MikaelSlevinsky](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mikaelslevinsky/32/38545_2.png) [@MikaelSlevinsky](https://discourse.julialang.org/u/MikaelSlevinsky)\
**Post date:** [September 29, 2017, 1:21pm UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/46 "2017-09-29T13:21:13Z")

</div>

Although much of BLAS/LAPACK _has_ been wrapped into nice high-level Julia functions, not all of the packages’ functions have been, and it is easier to call them in Fortran than to create Julia wrappers. To me, benchmarking Fortran without calling BLAS/LAPACK is unfair. (I’m not saying there aren’t any library issues to resolve!)

On the other hand, if we’re benchmarking genericity, such as Fortran calling the generic `matmul`, then why doesn’t the Julia benchmark call the generic one too? Performance aside, this is where Julia and its beautiful types have a linguistic advantage.

---

<div class="post-metadata">

**Author:** ![John\_Gibson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/john_gibson/32/5321_2.png) [@John\_Gibson](https://discourse.julialang.org/u/John_Gibson)\
**Post date:** [September 29, 2017, 2:15pm UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/47 "2017-09-29T14:15:01Z")

</div>

On how much to optimize, there’s a choice between

1. Pure language constructs and dead-simple algorithms, e.g. do matrix multiplication with simple for loops, no BLAS anywhere.
2. Typical usage, what the person with modest familiarity with the language might code with standard tools e,g, BLAS in Fortran & C or anywhere else it’s typical and easy with standard distributions of compilers and libraries, @inbounds etc allowed in Julia, @autojit for Python. But don’t change fib to a look-up table or recompile Python.

Both are meaningful, and I would like to see results for both. The current micro benchmarks are closer to 1 but not entirely consistent. It would be easier to push toward consistency with 1.

I suspect we’re rehashing discussions from the early days of test/perf/micro, and maybe we should either dig those discussions up or let those involved weigh in.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [September 29, 2017, 2:37pm UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/48 "2017-09-29T14:37:29Z")

</div>

> [@MikaelSlevinsky](#):
>
> On the other hand, if we’re benchmarking genericity, such as Fortran calling the generic matmul, then why doesn’t the Julia benchmark call the generic one too?

The Julia benchmark _does_ call the “generic” one: `*` … it’s just that the generic routine dispatches to the fast one. Whether `matmul` or `*` call down to a fast BLAS is a library issue. With a decent Fortran system, `matmul` calls a fast BLAS, and any “official” posted benchmark numbers should reflect such a configuration.

Frankly, I find this particular benchmark pretty uninteresting — as far as I can tell, its only purpose is to highlight the fact that matrix multiplication via the standard library (if one exists) is basically the same speed in all languages, properly configured, because every standard library can be configured to call the same fast BLAS.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [September 29, 2017, 2:52pm UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/49 "2017-09-29T14:52:27Z")

</div>

By the way, for previous discussions on this topic, see e.g.:

[https://github.com/JuliaLang/julia/pull/1084](https://github.com/JuliaLang/julia/pull/1084)  
[https://github.com/JuliaLang/julia/issues/2412](https://github.com/JuliaLang/julia/issues/2412)  
[https://github.com/JuliaLang/julia/issues/5128](https://github.com/JuliaLang/julia/issues/5128)  
[https://github.com/JuliaLang/julia/issues/13042](https://github.com/JuliaLang/julia/issues/13042)

(The usual thing that happens if you ask experts from other languages to work on the benchmarks is that they say “no one would write code this way … you need to vectorize / call optimized library X”, which misses the point of the benchmark.)

---

<div class="post-metadata">

**Author:** ![MikaelSlevinsky](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mikaelslevinsky/32/38545_2.png) [@MikaelSlevinsky](https://discourse.julialang.org/u/MikaelSlevinsky)\
**Post date:** [September 29, 2017, 2:55pm UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/50 "2017-09-29T14:55:47Z")

</div>

Sorry, by generic I meant the appropriate `_generic_matmatmul`, not the high-level `*`.

I agree with your sentiment, which is precisely why the Fortran benchmark should call BLAS.

---

<div class="post-metadata">

**Author:** ![mauro3](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mauro3/32/292_2.png) [@mauro3](https://discourse.julialang.org/u/mauro3)\
**Post date:** [September 29, 2017, 2:57pm UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/51 "2017-09-29T14:57:27Z")

</div>

Maybe some of the benchmarks should be renamed to reflect what they test as opposed to what their test case is. E.g. `fib` → `recursion`.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [September 29, 2017, 5:13pm UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/52 "2017-09-29T17:13:39Z")

</div>

Fortran calling BLAS directly would be fine. We didn’t write the Fortran benchmark code; someone contributed it in 2012: [https://github.com/JuliaLang/julia/pull/917](https://github.com/JuliaLang/julia/pull/917). We took the implementation at face value as a reasonable one – I, for one, am not a Fortran programmer. If you think there are improvements that should be made to the Fortran code for a fairer comparison, please do contribute them.

---

<div class="post-metadata">

**Author:** ![John\_Gibson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/john_gibson/32/5321_2.png) [@John\_Gibson](https://discourse.julialang.org/u/John_Gibson)\
**Post date:** [October 3, 2017, 3:01am UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/53 "2017-10-03T03:01:56Z")

</div>

I posted this PR a few days back. It gets test/perf/micro running again on julia-0.7.0. It’s a few Makefile changes for libraries that have moved in the julia source tree since 0.4.0.  
[https://github.com/JuliaLang/julia/pull/23922](https://github.com/JuliaLang/julia/pull/23922)

A few things/decisions remain before producing new benchmark data for publication:

1. Getting Fortran to call BLAS. I tried @Ralph_Smith’s suggestion above but got utterly horrific rand\_mat\_mul results (~100 times slower). I need to double-check that I linked to the correct -lblas, or that there wasn’t some problem in the interface with wrong integer size.

2. Getting R, Python, and Octave to call BLAS for rand\_mat\_mul. It looks like that’ll require recompiling and relinking these packages (instead of using my non-BLAS default packages on openSUSE). I’m not keen on doing this, and I’m not sure it’s fair.

3. Investigate slowness of mandel on C compared to Java, Javascript, and Lua. I’d welcome help here.

4. Rename the benchmarks. This idea has come up repeatedly as a way to clarify we are testing recursion, not the optimal fib algorithm, for example. My proposed renaming

| old | new |
| --- | --- |
| fib | recursion\_fibonacci |
| quicksort | recursion\_quicksort |
| pi\_sum | iteration\_pi\_sum |
| mandel | iteration\_mandelbrot |
| parse\_int | parse\_integers |
| rand\_mat\_mul | matrix\_multiply |
| rand\_mat\_stat | matrix\_statistics |
| printfd | print\_decimals |

Do those look good?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [October 3, 2017, 3:59am UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/54 "2017-10-03T03:59:03Z")

</div>

I like the renaming.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [October 3, 2017, 10:20am UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/55 "2017-10-03T10:20:29Z")

</div>

> [@ChrisRackauckas](#):
>
> Maybe the last time Lua wasn’t using LuaJiT. What version of Lua is used should be clearly documented.

I believe LuaJIT was always used (or else you wouldn’t have seen that speed with the non-JIT “Lua” implementation).

Replacing “LuaJIT” text with “SciLua” in the PR warranted? Maybe rather say LuaJIT and SciLua on the following line.

I didn’t look to much into SciLua, but it seems to “LuaJIT” language/implementation; with extras similar to Numpy? Does it make any difference, maybe only for BLAS using code, if that?

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [October 3, 2017, 5:28pm UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/56 "2017-10-03T17:28:16Z")

</div>

At some point this plot result was changed to be sorted alphabetically, which I find to be unhelpful for understanding what it’s telling me about overall performance. I believe it was originally sorted by the geometric mean of benchmark times, which would be a good ordering, imo, although I think having Julia in the first column on the [JuliaLang.org](http://JuliaLang.org) benchmarks is also justifiable.

---

<div class="post-metadata">

**Author:** ![stepelu](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@stepelu](https://discourse.julialang.org/u/stepelu)\
**Post date:** [October 4, 2017, 8:39am UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/57 "2017-10-04T08:39:47Z")

</div>

SciLua author here.

First of all, I fully agree on having LuaJIT + SciLua as naming, or LuaJIT using SciLua libraries, instead of just SciLua.  
Without LuaJIT none of my work would have been possible 🙂

Some clarifications:

1. LuaJIT is a JIT implementation of the Lua (5.1 + cherry picked additions from 5.2 and 5.3) language from Mike Pall, version 2.1 brings optimizations and speeds improvement over 2.0
2. The “sci” module is a collection of algorithms for general purpose scientific computing (think GSL in C), relies on OpenBLAS for some matrix operations, but it’s otherwise 100% in LuaJIT
3. the “scilua” executable adds language syntax extensions to LuaJIT in the form of syntactic sugar to facilitate the writing of vector/matrix expressions, requires the “sci” module of point 2.

With SciLua I refer to the framework composed by the module and the executable above, both of which use LuaJIT 2.1.

Please feel free to contact me if you have questions.

---

<div class="post-metadata">

**Author:** ![RoyiAvital](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/royiavital/32/571_2.png) [@RoyiAvital](https://discourse.julialang.org/u/RoyiAvital)\
**Post date:** [October 4, 2017, 11:22am UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/58 "2017-10-04T11:22:38Z")

</div>

What makes it so fast?

Maybe there tricks to learn.

---

<div class="post-metadata">

**Author:** ![John\_Gibson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/john_gibson/32/5321_2.png) [@John\_Gibson](https://discourse.julialang.org/u/John_Gibson)\
**Post date:** [October 5, 2017, 6:28pm UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/59 "2017-10-05T18:28:08Z")

</div>

The latest benchmarks, approaching completion.

 ![benchmarks](https://global.discourse-cdn.com/julialang/original/3X/4/f/4fc3ef8cec6ff0d8a0af84d24e19103f85d46a5d.png)

Revisions since last time

- got BLAS working for Fortran and Octave (thanks, @Ralph_Smith) but not for Python. Not sure if I’ll get Python working with BLAS; this seems fairly involved.

- renamed the benchmarks. Which is better, `print_decimals` or `print_to_file`? The relevant C code is `fprintf(f, "%ld %ld\n", i, i);` where `f` is `/dev/null`.

- reordered putting Julia next to C and other languages reordered by eye. It’s not straightforward to compute the geometric mean because not all languages implement all benchmarks (e.g. print\_to\_file is missing for Fortran).

- investigated the unexpectedly good performance of some iteration\_mandelbrot codes and found no obvious flaws. They all seem to be doing the same combinations of loops and computing the same numbers.

- changed the colormap to a handtooled rainbow since I found some of the default plotting colors hard to distinguish. I has a slightly more vibrant feel than the old colors; let me know if you dislike that.

- reverted “scilua” back to “luaJIT + scilua” in some places

I’m about ready to call it quits and post a PR for the new benchmark table and plot to [julialang.github.com](http://julialang.github.com).

---

<div class="post-metadata">

**Author:** ![NickNack](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nicknack/32/1792_2.png) [@NickNack](https://discourse.julialang.org/u/NickNack)\
**Post date:** [October 5, 2017, 7:19pm UTC](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991/60 "2017-10-05T19:19:16Z")

</div>

Quick suggestion: consider adding faint minor gridlines between the powers of 10. It is not currently visually obvious that the y-axis has a logarithmic scale, but minor gridlines would make this very clear.

EDIT: The colors are great now, but could you fix the spacing issues in the legend before you publish? Also, the x-axis font is kind of ugly (IMHO).

[Previous page](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991.md?page=2)

[Next page](https://discourse.julialang.org/t/benchmark-for-latest-julia/5991.md?page=4)