Trying PkgBenchmark.jl again

General Usage Performance
,
1

I got to a point where I need to keep track of performance improvements in my packages, specially with new releases of Julia coming up. The PkgBenchmark.jl package proposes a nice interface similar to Base.Test for benchmarking packages against old commits in the history of the project. The idea is really cool, but my attempts to use it are always unsuccessful :disappointed_relieved:

I have updated the benchmark code for the latest release of my package:

using GeoStats
using PkgBenchmark

srand(2017)

datadir = joinpath(@__DIR__,"data")

fname = joinpath(datadir,"permeability.csv")
geodata = readtable(fname, coordnames=[:x,:y])
domain = bounding_grid(geodata, [100,100])
problem = EstimationProblem(geodata, domain, :permeability)

@benchgroup "Kriging" ["solvers"] begin
  solverā‚ = Kriging(:permeability => @NT(variogram=ExponentialVariogram(range=40.)))
  solverā‚‚ = Kriging(:permeability => @NT(variogram=SphericalVariogram(range=40.)))

  @bench "Exponential variogram" solve($problem, solverā‚)
end

I then open the Julia prompt and type:

using PkgBenchmark

results = benchmarkpkg("GeoStats")

The benchmarks apparently did run, but the results object doesnā€™t seem to contain information? I went ahead and tried to benchmark against an old tagged version:

judge("GeoStats", "v0.4.4")

and it again prints an empty object without information? Anyone with experience can explain what is happening and what needs to be done to get the benchmark working?

2

I read ā€œ- ref optional, the commit to judgeā€ as requiring a commit hash rather than a version, but I might be wrong.

3

ref can be a commit, branch or version. With https://github.com/JuliaCI/PkgBenchmark.jl/pull/51 here is what I get.

https://asciinema.org/a/lIkXtQuUawZDJuGvZUBV7z7KF

Seems to work fine.

2 Likes
4

Nice! Worth adding to the docsā€¦ Which version is that? (current release is 0.0.2 of June 1).

1 Like
5

It is master. It needs more testing (by users) before tagging. Or perhaps I should just tag it. It is very breaking though.

6

Nice @kristoffer.carlsson, I was missing the showall command at the end to read the results. I will give it a try again on master. I donā€™t see the progress bar though, I will add more tests and check it.