[ANN] TestingUtilities.jl

Hey all,

I’m happy to announce TestingUtilities.jl, which provides macros to (hopefully) make your package testing experience a lot smoother. This package is set to be registered in the General registry in two days as of writing this post.

If you’ve ever made a number of changes to your code which cause your tests to fail, Julia’s excellent Test module makes it easy to which tests have failed but not specifically why those tests have failed.

The @Test macro (note the capitalization) in this package helps to alleviate this problem by displaying relevant values that caused your test to fail, even if the test expression is somewhat complicated
e.g., evaluating

inner_comparison_func(a,b) = a == 2*b
g = x->2x
a = 2
b = 1
@Test inner_comparison_func(a, g(b))

outputs

Test `inner_comparison_func(a, g(b))` failed with values:
a = 2
`g(b)` = 6
b = 3

Test Failed at REPL[311]:1
  Expression: inner_comparison_func(a, g(b))
   Evaluated: false

ERROR: There was an error during testing

This macro works by building up a computational graph of the expression until it finds Symbols that are the input values to this test. If the test fails, it displays the input values (as well as the value of the “top-level” args + kwargs from the test expression) that caused said test to fail.

Currently not all Julia syntactical constructions are supported, but if your test expressions are sufficiently simple, this macro should help you reason about why things have gone awry from the output of your logs.

When run from an interactive Julia session, the input variables which cause the test failures are set in the Main module. This can be helpful when, for instance, you’re evaluating an entire @testset at a time with multiple @Test expressions, some of which are failing.

The @test_cases macro allows one to compactly validate a test expression on a number of identically structured test case instances. Similar to @Test, it will output the test case values that cause the each expression to fail, if they do, in fact, fail.

@test_cases begin 
      a | b | output 
      1 | 2 | 3
      1 | 2 | 4
      0 | 0 | 1
      0 | 1 | 2
      @test a + b == output
  end

Outputs

Test Failed at REPL[313]:1
  Expression: a + b == output
   Evaluated: false

Test `a + b == output` failed with values:
------
`a + b` = 3
output = 4
a = 1
b = 2
ERROR: There was an error during testing

@test_cases halts execution on the first test case that causes a failure when invoked on its own. To capture all failing test instances, run it within a @testset, e.g.,

@testset begin 
       @test_cases begin 
           a | b | y
           1 | 2 | 3
           1 | 2 | 4
           0 | 0 | 1
           @test a + b == y 
           @test b^2 + 1 == y
       end
end

Outputs:

Test `a + b == y` failed with values:
------
`a + b` = 3
y = 4
a = 1
b = 2
------
`a + b` = 0
y = 1
a = 0
b = 0
Test `b ^ 2 + 1 == y` failed with values:
------
`b ^ 2 + 1` = 5
y = 3
a = 1
b = 2
------
`b ^ 2 + 1` = 5
y = 4
a = 1
b = 2

There is also alternative equivalent syntax for writing test cases, if the |-delimited version results in expressions that are too difficult to read.

@testset begin 
    @test_cases begin
        input1 | input2 | output
        (input1 = 5, input2 = "aabcdddfasdfasdfasdfasdf", output = false)
        input1 => 0,
             input2 => "abcdefgh", output => false
        (input1 = 5, input2 = "aaaaa", output = true)
        @test (length(input2) == length(input1)) == output
    end
end

Outputs

test set: Test Failed at REPL[316]:2
  Expression: (length(input2) == length(input1)) == output
   Evaluated: false

Test `(length(input2) == length(input1)) == output` failed with values:
------
`length(input2) == length(input1)` = false
output = true
input1 = 5
input2 = "aaaaa"
Test Summary: | Pass  Fail  Total  Time
test set      |    2     1      3  0.1s
ERROR: Some tests did not pass: 2 passed, 1 failed, 0 errored, 0 broken.

Not all of Julia’s syntax is currently handled in the expression parsing, but if your test expressions are sufficiently simple, these macros should handle extracting the relevant Symbols and Exprs to output on failure.

If you have any questions or comments, please let me know below or on the Github repo. Happy testing!

9 Likes

I like to do

for val in (x, y, z)
   @test munge(x)
end

But I don’t do this because I can’t tell whether x, y, or z failed. Maybe this package would solve this problem.

for that I use

@testset "$val" for val in (x, y, z)
    @test munge(val)
end
12 Likes

Look forward to this.
how about a shorter name TestUtils.jl? TestPlus?