Benchmarking tests to ensure PRs don't introduce regressions

anon94023334 · January 26, 2018, 11:10pm

Has anyone come up with a clever way to perform benchmarking of specific functions as part of the testing or CI process so that PRs can be checked to make sure no performance regressions are introduced?

omus · January 27, 2018, 3:14am

You may want to checkout https://github.com/JuliaCI/PkgBenchmark.jl

mauro3 · January 27, 2018, 8:38am

But I don’t think you can run the benchmarks on the CI machines, as you have no control over what exact machine one gets and how many other processes are running. At least Julia itself has a dedicated machine for this: nanosoldier. But should be ok to run those BM locally.

anon94023334 · January 28, 2018, 5:45pm

This is a great point. I was hoping to be able to check out a PR locally, and run the benchmarks on it, comparing to previous benchmarks. I’ve been playing around with PkgBenchmark but am not quite sure how to do this properly.

PetrKryslUCSD · January 28, 2018, 6:37pm

Simply time your own tests in your own packages. If they slow down, report it.

ChrisRackauckas · January 28, 2018, 6:44pm

I use this method.

github.com

SciML/OrdinaryDiffEq.jl/blob/master/test/runtests.jl

using Pkg
using SafeTestsets, Test
const LONGER_TESTS = false

const GROUP = get(ENV, "GROUP", "All")
const is_APPVEYOR = Sys.iswindows() && haskey(ENV, "APPVEYOR")

function activate_downstream_env()
    Pkg.activate("downstream")
    Pkg.develop(PackageSpec(path = dirname(@__DIR__)))
    Pkg.instantiate()
end

#Start Test Script

@time begin
    if GROUP == "All" || GROUP == "InterfaceI" || GROUP == "Interface"
        @time @safetestset "Discrete Algorithm Tests" include("interface/discrete_algorithm_test.jl")
        @time @safetestset "Tstops Tests" include("interface/ode_tstops_tests.jl")
        @time @safetestset "Backwards Tests" include("interface/ode_backwards_test.jl")

This file has been truncated. show original

If you know the numbers well you’ll see something fishy and it makes it quite easy to bisect to see what happened. Then we have DiffEqBenchmarks.jl which we try to run after any big change to the integrators to make sure it’s the same. I am planning on setting up a local computer with a CRON job to continually update those as well.