[ANN] RandomizedPropertyTests.jl

RandomizedPropertyTest.jl is a test framework for testing program properties with random (as well as special pre-defined) inputs, inspired by QuickCheck . Sources can be found at https://gitlab.com/quf/randomizedpropertytest.jl and https://git.sr.ht/~quf/RandomizedPropertyTest.jl. Documentation is also found there. It works for any Julia version from 1.0¹. The current version is 0.1.0, so the API is not guaranteed to be stable. However, I think the current API works quite well and I don’t plan on changing it without good reason.

Example use:

julia> using RandomizedPropertyTest

julia> @quickcheck (x^2 ≥ 0) (x :: Float64)
┌ Warning: Property `x ^ 2 ≥ 0` does not hold for x = NaN.
└ @ RandomizedPropertyTest [snip]/RandomizedPropertyTest.jl:[snip]
false
julia> @quickcheck (sin(x + π/2) ≈ cos(x)) (x :: Range{Float64, 0, 2π})
true
julia> @quickcheck n=10^6 (sum(x^k/factorial(k) for k in 20:-1:0) ≈ exp(x)) (x :: Range{Float64, -2, 2})
true

It has builtin support for a few basic Julia datatypes and can be extended for custom datatypes or custom random distributions (see README.md).

I’d be happy to receive any feedback, bug reports, suggestions, &c.

I would especially like some suggestions on how to deal with combinatorial explosion of special cases, as well as parallelism:

Special cases are defined for each basic data type (e.g. NaN for AbstractFloat). For multiple data types (or tuples), all combinations of special cases are checked. This leads to an exponential increase in special cases to be checked. Of course, I could check only a subset of special cases, but it’s not clear to me what that subset should be.

Parallelism would obviously be beneficial for performance, but I would like to keep deterministic test results to aid debuggability. In sequential execution it is achieved by using a PRNG with fixed seed. In a parallel setting multiple PRNGs with different seeds would need to run in parallel depending on the number of threads. Re-running the same check with a different number of threads may give a different result in that case.

¹ Unit tests for 0.1.0 (not master) are broken on Julia 1.0, but this only affects the package’s tests, not the package.

8 Likes

As this is released under GPL3 am I allowed to use it in a CI stage of an MIT-licensed package?

1 Like

If all you do is call @quickcheck, that’s definitely ok. If you extend it for custom datatypes (by adding methods to generate and specialcases), that part counts as a derivative work in my view and would have to be GPL licensed.

Maybe the tests and the additional methods to the aforementioned functions can be released under GPL and the package itself can remain with an MIT license? The source code of the package is of course completely independent of the tests, but the reverse does not hold.