ANN: Automated Property-Based Testing with Checkers.jl v0.0.1

Community
, ,
1

Hi all, pleased to announce https://github.com/pkalikman/Checkers.jl, a QuickCheck-like, property-based, automated randomized testing package for Julia. Please give it a try and let me know what you think.

julia> Pkg.add("Checkers")
INFO: Cloning cache of Checkers from https://github.com/pkalikman/Checkers.jl.git
INFO: Installing Checkers v0.0.1
INFO: Package database updated

julia> using Checkers

julia> f(x) = x^2
f (generic function with 1 method)

julia> @test_forall x in -10:10, f(x) >= 0 #Literally tests each value in the iterator
Test Passed
  Expression: (x in -10:10, f(x) >= 0)

julia> @test_formany -10<x<10, f(x) >= 0 #Randomly selects 100 values to test
Test Passed
  Expression: (:((-10 < x < 10, f(x) >= 0)), :(mode = test_formany))

julia> @test_forall x in -10:10, f(x) > 0 #Should fail b/c f(0) = 0
Test Failed
  Expression: (x in -10:10, f(x) > 0)
ERROR: There was an error during testing


julia> # Control how many tests in @test_formany:
       @test_formany ntests = 10_000  1 < x < 5, x^2 < 30
Test Passed
  Expression: (:(ntests = 10000), :((1 < x < 5, x ^ 2 < 30)), :(mode = test_formany))

julia> # Test a conditional property, passing only if 100 of the tests are not vacuous:
       @test_formany ntests = 100  0 < x < Inf, 0 < y < Inf,  x < y --> log(x) < log(y)
Test Passed
  Expression: (:(ntests = 100), :((0 < x < Inf, 0 < y < Inf, $(Expr(:-->, :(x < y), :(log(x) < log(y)))))), :(mode = test_formany))

julia> # Test a conditional property, passing only if 100 of the tests are not vacuous, but only allow 100 tests. Should Error b/c not enough tests aren't vacuous:
       @test_formany ntests = 100 maxtests = 100  0 < x < 10,  0 < y < 10,  x < y --> x^2 < y^2
Error During Test
  Test threw an exception of type ErrorException
  Expression:
(:(ntests = 100), :(maxtests = 100), :((0 < x < 10, 0 < y < 10, $(Expr(:-->, :(x < y), :(x ^ 2 < y ^ 2))))), :(mode = test_formany))
  Found only 50/100 values satisfying given condition.
  Stacktrace:
   [1] macro expansion at /Users/plk/.julia/v0.6/Checkers/src/./test-cases.jl:242 [inlined]
   [2] anonymous at ./<missing>:?
   [3] eval(::Module, ::Any) at ./boot.jl:235
   [4] eval_user_input(::Any, ::Base.REPL.REPLBackend) at ./REPL.jl:66
   [5] macro expansion at ./REPL.jl:97 [inlined]
   [6] (::Base.REPL.##1#2{Base.REPL.REPLBackend})() at ./event.jl:73
ERROR: There was an error during testing

julia> # Test a conditional property, passing only if 100 of the test are not vacuous, but allow 100,000 tests, so it should Pass:
       @test_formany ntests = 100 maxtests = 100_000 0 < x < 10,  0 < y < 10,  x < y --> x^2 < y^2
Test Passed
  Expression: (:(ntests = 100), :(maxtests = 100000), :((0 < x < 10, 0 < y < 10, $(Expr(:-->, :(x < y), :(x ^ 2 < y ^ 2))))), :(mode = test_formany))

More details and examples at Home - Checkers.jl

Thanks!

4 Likes
2

Looks great!

Is that possible to specify a generator for @test_formany like
@test_formany x = randstring(10), length(x) == 10 ?

3

This looks interesting.

From the perspective of someone who does use such kind of tests, the usability would increase a lot if the error message gave a sample value for which the tests failed

2 Likes
4

This is a good idea. We have some custom generator support but it’s not particularly intuitive at the moment. Thanks!

5

Also a great idea. We did think of this a little, and you can use the logto keyword with @test_formany to write the values tested to a csv and inspect, but indeed offering at least one failure case for each failure of each macro in the output is a great idea. Thanks!