How to compare commits with PkgBenchmark


I would like to compare benchmarks for two commits. I’m trying to follow the documentation, but I receive an error indicating that the input is not a repo. I tried a few different approach, but with the same result. Can someone please tell me how to setup the benchmark?


using PkgBenchmark
import SequentialSamplingModels

#target_id = "a45db69"
target_id = "a45db6920287bd6442be175dc9bc690eb5ce57d3"
target = benchmarkpkg(SequentialSamplingModels, target_id)


ERROR: /home/dfish/.julia/packages/SequentialSamplingModels/zeGhr/ is not a git repo, cannot benchmark at 0e32c56acea48fd722cd8176cd7e3d1d21d8b93c
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] benchmarkpkg(pkg::String, target::String; script::Nothing, postprocess::Nothing, resultfile::Nothing, retune::Bool, verbose::Bool, logger_factory::Nothing, progressoptions::Nothing, custom_loadpath::String)
   @ PkgBenchmark ~/.julia/packages/PkgBenchmark/nuPay/src/runbenchmark.jl:134
 [3] benchmarkpkg(pkg::String, target::String)
   @ PkgBenchmark ~/.julia/packages/PkgBenchmark/nuPay/src/runbenchmark.jl:44
 [4] top-level scope
   @ ~/.julia/dev/sandbox/benchmarks/ssm_benchmark.jl:16


You should dev the package in question, which will clone a full git repo into .julia/dev. You might then also need to check that the commit you are trying to benchmark is accessible in the repo.

Thank you. The problem was that the package was not in dev mode. I did not see that requirement in the documentation. I will submit an issue so that it is more clear to users.