[ANN] UnitTestDesign: All-pairs, and other algorithms to generate unit testing parameters

If you have a function that has a lot of parameters, that takes some time to run, and that you want to test well, then you want to test it against a small set of test cases that will still find all of its faults. A thorough set of tests will run all possible code paths. They will take each path of each if-then decision. The outcome of any if-then is the product of interactions among the parameters we set, so we test efficiently when we maximize the number of combinations of parameter values. At least, that’s one theory.

This library generates combinatorial interaction test sets. The most common is the all-pairs test set, which generates few tests that contain every possible pair of parameter values.

test_cases = all_pairs(
           [1, 2, 3], ["low", "mid" ,"high"], [1.0, 3.7, 4.9], [:greedy, :relax, :optim])
10-element Array{Array{Any,1},1}:
 [1, "low", 1.0, :greedy]
 [1, "mid", 3.7, :relax]
 [1, "high", 4.9, :optim]
 [2, "low", 3.7, :optim]
 [2, "mid", 1.0, :greedy]
 [2, "high", 1.0, :relax]
 [3, "low", 4.9, :relax]
 [3, "mid", 1.0, :optim]
 [3, "high", 3.7, :greedy]
 [2, "mid", 4.9, :greedy]
  • Make unit test cases or test datasets.
  • Higher-order covering arrays, for instance 3-way interactions and 4-way interactions.
  • Combinatorial excursions from a base test case.
  • Full-factorial test sets.
  • With filtering of disallowed tests.
  • With seeded test cases that must be included.

This is a native Julia version of software such as Automated Combinatorial Testing for Software.

I see lots of advanced testing packages in Julia, and testing scientific software is, in general a fascinating problem. If there is some way this piece can work better for people, or fit in better, let’s make that happen.


Nice, this looks pretty useful!

1 Like

Thank you. Please keep at it. Where relevant, this can make a difference.

A lot of the papers on these methods are by the National Institute of Standards and NASA, which is imposing. I found these methods when I was testing a huge Bayesian inference, where I wanted to generate lots of user inputs. And you have a nice sig! bheart