If you have a function that has a lot of parameters, that takes some time to run, and that you want to test well, then you want to test it against a small set of test cases that will still find all of its faults. A thorough set of tests will run all possible code paths. They will take each path of each if-then decision. The outcome of any if-then is the product of interactions among the parameters we set, so we test efficiently when we maximize the number of combinations of parameter values. At least, that’s one theory.
This library generates combinatorial interaction test sets. The most common is the all-pairs test set, which generates few tests that contain every possible pair of parameter values.
test_cases = all_pairs(
[1, 2, 3], ["low", "mid" ,"high"], [1.0, 3.7, 4.9], [:greedy, :relax, :optim])
10-element Array{Array{Any,1},1}:
[1, "low", 1.0, :greedy]
[1, "mid", 3.7, :relax]
[1, "high", 4.9, :optim]
[2, "low", 3.7, :optim]
[2, "mid", 1.0, :greedy]
[2, "high", 1.0, :relax]
[3, "low", 4.9, :relax]
[3, "mid", 1.0, :optim]
[3, "high", 3.7, :greedy]
[2, "mid", 4.9, :greedy]
- Make unit test cases or test datasets.
- Higher-order covering arrays, for instance 3-way interactions and 4-way interactions.
- Combinatorial excursions from a base test case.
- Full-factorial test sets.
- With filtering of disallowed tests.
- With seeded test cases that must be included.
This is a native Julia version of software such as Automated Combinatorial Testing for Software.
I see lots of advanced testing packages in Julia, and testing scientific software is, in general a fascinating problem. If there is some way this piece can work better for people, or fit in better, let’s make that happen.
