Hey all,
I’m happy to announce TestingUtilities.jl, which provides macros to (hopefully) make your package testing experience a lot smoother. This package is set to be registered in the General registry in two days as of writing this post.
If you’ve ever made a number of changes to your code which cause your tests to fail, Julia’s excellent Test module makes it easy to which tests have failed but not specifically why those tests have failed.
The @Test macro (note the capitalization) in this package helps to alleviate this problem by displaying relevant values that caused your test to fail, even if the test expression is somewhat complicated
e.g., evaluating
inner_comparison_func(a,b) = a == 2*b
g = x->2x
a = 2
b = 1
@Test inner_comparison_func(a, g(b))
outputs
Test `inner_comparison_func(a, g(b))` failed with values:
a = 2
`g(b)` = 6
b = 3
Test Failed at REPL[311]:1
Expression: inner_comparison_func(a, g(b))
Evaluated: false
ERROR: There was an error during testing
This macro works by building up a computational graph of the expression until it finds Symbols that are the input values to this test. If the test fails, it displays the input values (as well as the value of the “top-level” args + kwargs from the test expression) that caused said test to fail.
Currently not all Julia syntactical constructions are supported, but if your test expressions are sufficiently simple, this macro should help you reason about why things have gone awry from the output of your logs.
When run from an interactive Julia session, the input variables which cause the test failures are set in the Main module. This can be helpful when, for instance, you’re evaluating an entire @testset at a time with multiple @Test expressions, some of which are failing.
The @test_cases macro allows one to compactly validate a test expression on a number of identically structured test case instances. Similar to @Test, it will output the test case values that cause the each expression to fail, if they do, in fact, fail.
@test_cases begin
a | b | output
1 | 2 | 3
1 | 2 | 4
0 | 0 | 1
0 | 1 | 2
@test a + b == output
end
Outputs
Test Failed at REPL[313]:1
Expression: a + b == output
Evaluated: false
Test `a + b == output` failed with values:
------
`a + b` = 3
output = 4
a = 1
b = 2
ERROR: There was an error during testing
@test_cases halts execution on the first test case that causes a failure when invoked on its own. To capture all failing test instances, run it within a @testset, e.g.,
@testset begin
@test_cases begin
a | b | y
1 | 2 | 3
1 | 2 | 4
0 | 0 | 1
@test a + b == y
@test b^2 + 1 == y
end
end
Outputs:
Test `a + b == y` failed with values:
------
`a + b` = 3
y = 4
a = 1
b = 2
------
`a + b` = 0
y = 1
a = 0
b = 0
Test `b ^ 2 + 1 == y` failed with values:
------
`b ^ 2 + 1` = 5
y = 3
a = 1
b = 2
------
`b ^ 2 + 1` = 5
y = 4
a = 1
b = 2
There is also alternative equivalent syntax for writing test cases, if the |-delimited version results in expressions that are too difficult to read.
@testset begin
@test_cases begin
input1 | input2 | output
(input1 = 5, input2 = "aabcdddfasdfasdfasdfasdf", output = false)
input1 => 0,
input2 => "abcdefgh", output => false
(input1 = 5, input2 = "aaaaa", output = true)
@test (length(input2) == length(input1)) == output
end
end
Outputs
test set: Test Failed at REPL[316]:2
Expression: (length(input2) == length(input1)) == output
Evaluated: false
Test `(length(input2) == length(input1)) == output` failed with values:
------
`length(input2) == length(input1)` = false
output = true
input1 = 5
input2 = "aaaaa"
Test Summary: | Pass Fail Total Time
test set | 2 1 3 0.1s
ERROR: Some tests did not pass: 2 passed, 1 failed, 0 errored, 0 broken.
Not all of Julia’s syntax is currently handled in the expression parsing, but if your test expressions are sufficiently simple, these macros should handle extracting the relevant Symbols and Exprs to output on failure.
If you have any questions or comments, please let me know below or on the Github repo. Happy testing!