Benchmark MATLAB & Julia for Matrix Operations

Amin_Yahyaabadi · October 13, 2019, 9:33pm

First, it is all about the overhead. For normal operations using different threads has too much overhead and doesn’t help speed up.

If you go to my repository there are two branches that are for two completely different systems and they both show the same result.

Second, 7 times is very low, susceptible to noise. You should run the function many more times.

In my code for Julia, BenchmmarkTools takes 700 samples and their median is calculated. Additionally, I repeat the process 4 times and average those medians

github.com

juliamatlab/Julia-Matlab-Benchmark/blob/6e6e015efcce883f47e2942d8b60ed69315b265f/JuliaMain.jl#L6


      
          println("started")
          
          # ]add BenchmarkTools
          # ]add MAT
          using BenchmarkTools # for benchmark
          const BenchmarkTools.DEFAULT_PARAMETERS.samples = 700;
          const BenchmarkTools.DEFAULT_PARAMETERS.evals = 1;
          using DelimitedFiles # for readdlm, writedlm
          using Statistics # for median
          using LinearAlgebra # for eigen, pinv, svd, cholesky
          using Random # for randperm
          using MAT
          
          const operationMode = 2; # 0 for test only # 1 for partial benchmark # 2 for full benchmark
          const saveData = true; # true for saving Runtimes to csv and mat
          const JuliaFlag = true; # run Julia benchmark

github.com

juliamatlab/Julia-Matlab-Benchmark/blob/6e6e015efcce883f47e2942d8b60ed69315b265f/JuliaBench.jl#L38


      
          	for ii = 1:length(vMatrixSize)
          		matrixSize = vMatrixSize[ii];
          		mX = randn(matrixSize, matrixSize);
          		mY = randn(matrixSize, matrixSize);
          		println("Matrix Size - $matrixSize");
          
          		jj=1;
          		for fun in allFunctions
          			println("Processing $(allFunctionsString[jj]) - MatrixSize= $matrixSize");
          
          			for kk = 1:numIterations;
          
          				benchIJK =@benchmark $fun($matrixSize, $mX, $mY)
          				# t =@benchmarkable $fun($matrixSize, $mX, $mY);
          				# tune!(t)
          				# run(t)
          
          				mRunTime[ii, jj, kk]=median(benchIJK).time/1e3;
          				# println("$(mRunTime[ii, jj, kk])")
          
          			end

For Matlab, timeit is used which is the function Matlab recommends for benchmarking. Internally, it calls the function many times. Again, I repeat the process 4 times and average those medians.

github.com

juliamatlab/Julia-Matlab-Benchmark/blob/6e6e015efcce883f47e2942d8b60ed69315b265f/MatlabBench.m#L48


      
              mX = randn(matrixSize, matrixSize);
              mY = randn(matrixSize, matrixSize);
              disp(['Matrix Size - ', num2str(matrixSize)]);
          
              for jj = 1:length(cRunTimeFunctions)
                  disp(['Processing ', num2str(cFunctionString{jj}), ' Matrix Size ', num2str(matrixSize)]);
          
                  for kk = 1:numIterations
          
                      fun=@() cRunTimeFunctions{jj}(matrixSize, mX, mY);
                      mRunTime(ii, jj, kk) = timeit(fun)*1e6; % computes median of bench times
                      
          
                  end
                  cRunTime{jj+1,1}=num2str(cFunctionString{jj});
          		cRunTime{1,ii+1}=matrixSize;
          		cRunTime{jj+1,ii+1}=mean(mRunTime(ii, jj,:));
              end
          end
          
          tRunTime=cell2table(cRunTime);

Also, when you measure the time like this, there is a big chance that the operation that you are interested in, is not optimized and compiled! :