Why does my program's performance vary so much from run to run? Can it be fixed?

General Usage Performance
, ,
13

Now I ran an example that uses no memory at all, and the equivalent code in Fortran (time measured with system’s time here). There is no clear difference between the variations in time measures. Also, running both sets side by side, there is a clear correlation of the times of one set (Julia) with the other set (Fortran), indicating that there is a system-wide thing going on, associated with the fluctuations.

times
times399×899 38.1 KB

The no-mem Julia code is:

code 
function f()
  s = 0.
  for i in 1:50_000
    x = sin(i)
    for j in 1:20_000
      y = cos(j)
      if i%j > 2
        s += x*y
      end
    end
  end
  s
end

@time f()

And the Fortran equivalent is:

code 
program main
  integer :: i, j
  double precision :: s, x, y
  s = 0.
  do i = 1,50000
    x = dsin(dble(i))
    do j = 1,20000
      y = dcos(dble(j))
      if(mod(i,j) > 2) then
        s = s + x*y
      end if
    end do
  end do
  write(*,*) s
end program main

Thus, my conclusion is that benchmarking is hard :neutral_face:

edit: Complementing the previous experiment. Now I ran 100 times each of the two calculations above, side by side. The result is:

plotimes

Note that there is a clear correlation between the launch time and the execution time for each language. That is, something happened system-wide that led both to be slower. I have here 4 cores, and these programs were using two of them only (I kept working with a spreadsheet in the meanwhile). The dispersion of the times does not seem to the significantly different from each other. Note also that the first and last executions appear to be faster, so even if there are more cores than programs being executed in the test, they appear to compete with each other (when the Fortran test finished, the Julia executions got quite faster, apparently).

So, this dispersion of execution times does not seem to be any particular of Julia.