This is a huge thread with 80 replies, and it makes for some interesting reading. Just to make sure I am getting the bigger lines here, what I currently understand is:
In a big benchmark of matrix arithmetic, MATLAB is faster than julia. Julia 0.6 changes the performance somewhat but does not change that overall conclusion. This does not reflect an overall speed advantage to Matlab, but it may be important for people whose code is heavily based on linear algebra.
Both Matlab and Julia rely on external libraries for linear algebra. Matlab uses MKL which may be faster, but the numbers donāt seem to support it and MKL is proprietary and so not a good platform for an open source language.
The other valid candidate to explain the difference might be multi-threading, but the current benchmarks cannot be used to tell that; it would require a run where threading is turned off in Matlab and another where experimental threading is turned on in Julia. Such tests are the only way to advance the discussion past exchange of opinion.
If multithreading is the culprit, this will be solved in the (near?) future because multithreading is an active area of development.
Is this correct? Thanks!
@RoyiAvital are you on windows by any chance? As far as I know, by default some optimizations (for SIMD etc.) are turned off by default on windows for compatibility with a larger range of machines. To get full performance you need to rebuild the system image. But I think someone more knowledgeable about these details, such as @tkelman can probably say with more certainty.
Julia needs to take care of Loop Fusion (If I understand will happen on 0.6).
Iām not sure about Broadcasting, but it is not up to what you get in MATLAB (MATLAB both apply SIMD and Multi Threading).
Julia should add Multi Threading in general and in the loop fusion and broadcasting case. Iād be happy to see in Julia style, Namely user controlled by Macro. The reason is sometimes the operation is Memory Bounded and then Multi Threading only hurts performance. A macro with @Auto (Maybe even implicit) and @OFF / @ON would be great.
Some Linear Algebra algorithm (Internal) in Julia are slower (See sqrtm(), expm()). Both in MATLAB and Julia they are implemented using the language itself (Mostly).
Some BLAS / LAPACK supported operations are slower in Julia. Iād say it is due to OpenBLAS being slower than Intel MKL.
My personal conclusions so far:
Julia is doing its infant baby steps and those are impressive steps. There are some rough edges with the language (Less predictable than MATLAB regarding the work with Numerical Arrays in my opinion) yet the potential is clearly seen.
I really like the spirit of letting the user control everything (For example the @inbounds Macro). This helps taking things to the extreme when needed.
Julia Pro (The product, not the language) must improve its BLAS / LAPACK engine. Probably by working with OpenBLAS team.
Julia approach with the . (Dot) operator is excellent. I like it. Once this and broadcasting are optimized it will be a killer feature.
Multi Threading - If Julia keeps giving the user the control for Multi Threading (At least giving full control in case the user wants it) and apply it efficiently it will be able to be more efficient than MATLAB.
Consistency - Juliaās performance are more consistent than MATLAB. Like it!
I think this is not shown in benchmarks anywhere. The imgur link posted by @tkelman doesnāt show this and neither do the OpenBLAS benchmarks. I also ran the linpack benchmark on our cluster, MKL and OpenBLAS results are very close as in all other tests. I think much of the MKL myths predate OpenBLAS, when there was a tangible difference between ATLAS and MKL.
Since it got hinted that I messed up my benchmarking, I reran them at work and made sure that they used the correct versions. The results were the same, canāt post them now since I am at home.
If youāre using binaries of Julia on any platform, we set the C compiler when building Juliaās dependencies and the JIT when building the system image to avoid using instructions that arenāt available on older systems.
The only thing that used to be Windows specific was, in 0.4 and earlier, not using the precompiled system image due to debug info issues which led to longer startup time - this was fixed for 0.5.
Thanks. I think I knew that, but my confusion arose because on mac os or linux I always build Julia myself, whereas on Windows I donāt (!), so I obviously have a less optimized setup. I have also never successfully rebuilt the system image on windows.
Back at work now and you are right. BOTH version on 0.5 actually ran with (MKL) which is obvious in hindsight since the timings on 0.6 was different even though the same underlying library should have been used.
I just want to add a point about the argument if one should test with or without multi-threading. For me itās very often the case that my programs are trivially parallelizable and I use only one core for each job to avoid inefficient parallelization. For this case the single core speed is what matters. On the other hand while testing and developing the program I want to use many cores to finish a single run as fast as possible. So I definitely care about both cases.
I also reran the HPL linpack test, here are the results:
Standard HPL OpenBLAS, 32 MPI processes on a single node: 757 Gflops
Standard HPL MKL, 32 MPI processes on a single node: 788 Gflops
Intel HPL MKL, 32 MPI processes on a single node: 814 Gflops
Intel HPL MKL, 2 MPI processes with 16 threads each on a single node: 963 Gflops
From both tests it seems clear to me that MKL wins when threading enters into the equation, but single-core performance is much closer, with the possible exception of the Cholesky and Eigen decompositions.
We did some comparisons of Julia vs MATLAB a while back and gave a talk at JuliaCon, a couple of years back. You can find this talk online and while it is probably out of date, we clearly observed that MATLABās automatic parallelism actually slowed some codes down, as opposed to improving performance. Their heuristics for when to use threads has probably improved. But I donāt believe that automatic parallelism (when how and where threads are used is completely a mystery and not in your control) can match the efficiency of parallel-aware code. For simple stuff though, it looks pretty cool.
If you use threads in Julia (by setting JULIA_NUM_THREADS) and call into a BLAS library, you will likely end up oversubscribing cores and completely destroying performance. This should be obvious when it happens ā the slowdown is pretty massive. The answer is to also set OMP_NUM_THREADS=1, thereby preventing the libraries from starting threads. Of course, this only makes sense if youāre using your Julia threads effectively. True and effective nested parallelism is coming to Julia āsoonā, but it wonāt help with nesting parallelism from your Julia code and one of these BLAS libraries for a good while. It will help, if a BLAS library is written IN Julia though.
@kpamnany, I totally agree with you.
In Julia spirit parallelism should be controlled by user.
Maybe a heuristic drive Auto Default method but certainly option to turn it OFF or ON by user control (And set its parameters).
Yet Iād add that from R2016a and on MATLAB is improving its JIT significantly with each iteration.
Edit:
I wanted to say that this project is detached and move to my organization:
We plan to make Matlab friendly APIs written in native Julia, and then test their performance by comparing it to the Matlab one.
So much more is coming and the benchmark is going to be much broader.
