I’m happy to announce TestingUtilities.jl, which provides macros to (hopefully) make your package testing experience a lot smoother. This package is set to be registered in the
General registry in two days as of writing this post.
If you’ve ever made a number of changes to your code which cause your tests to fail, Julia’s excellent
Test module makes it easy to which tests have failed but not specifically why those tests have failed.
@Test macro (note the capitalization) in this package helps to alleviate this problem by displaying relevant values that caused your test to fail, even if the test expression is somewhat complicated
inner_comparison_func(a,b) = a == 2*b g = x->2x a = 2 b = 1 @Test inner_comparison_func(a, g(b))
Test `inner_comparison_func(a, g(b))` failed with values: a = 2 `g(b)` = 6 b = 3 Test Failed at REPL:1 Expression: inner_comparison_func(a, g(b)) Evaluated: false ERROR: There was an error during testing
This macro works by building up a computational graph of the expression until it finds
Symbols that are the input values to this test. If the test fails, it displays the input values (as well as the value of the “top-level” args + kwargs from the test expression) that caused said test to fail.
Currently not all Julia syntactical constructions are supported, but if your test expressions are sufficiently simple, this macro should help you reason about why things have gone awry from the output of your logs.
When run from an interactive Julia session, the input variables which cause the test failures are set in the
Main module. This can be helpful when, for instance, you’re evaluating an entire
@testset at a time with multiple
@Test expressions, some of which are failing.
@test_cases macro allows one to compactly validate a test expression on a number of identically structured test case instances. Similar to
@Test, it will output the test case values that cause the each expression to fail, if they do, in fact, fail.
@test_cases begin a | b | output 1 | 2 | 3 1 | 2 | 4 0 | 0 | 1 0 | 1 | 2 @test a + b == output end
Test Failed at REPL:1 Expression: a + b == output Evaluated: false Test `a + b == output` failed with values: ------ `a + b` = 3 output = 4 a = 1 b = 2 ERROR: There was an error during testing
@test_cases halts execution on the first test case that causes a failure when invoked on its own. To capture all failing test instances, run it within a
@testset begin @test_cases begin a | b | y 1 | 2 | 3 1 | 2 | 4 0 | 0 | 1 @test a + b == y @test b^2 + 1 == y end end
Test `a + b == y` failed with values: ------ `a + b` = 3 y = 4 a = 1 b = 2 ------ `a + b` = 0 y = 1 a = 0 b = 0 Test `b ^ 2 + 1 == y` failed with values: ------ `b ^ 2 + 1` = 5 y = 3 a = 1 b = 2 ------ `b ^ 2 + 1` = 5 y = 4 a = 1 b = 2
There is also alternative equivalent syntax for writing test cases, if the
|-delimited version results in expressions that are too difficult to read.
@testset begin @test_cases begin input1 | input2 | output (input1 = 5, input2 = "aabcdddfasdfasdfasdfasdf", output = false) input1 => 0, input2 => "abcdefgh", output => false (input1 = 5, input2 = "aaaaa", output = true) @test (length(input2) == length(input1)) == output end end
test set: Test Failed at REPL:2 Expression: (length(input2) == length(input1)) == output Evaluated: false Test `(length(input2) == length(input1)) == output` failed with values: ------ `length(input2) == length(input1)` = false output = true input1 = 5 input2 = "aaaaa" Test Summary: | Pass Fail Total Time test set | 2 1 3 0.1s ERROR: Some tests did not pass: 2 passed, 1 failed, 0 errored, 0 broken.
Not all of Julia’s syntax is currently handled in the expression parsing, but if your test expressions are sufficiently simple, these macros should handle extracting the relevant
Exprs to output on failure.
If you have any questions or comments, please let me know below or on the Github repo. Happy testing!