Exit on first @test failure in @testset

General Usage
1

I am running package tests, and would like to save time and console spam by just exiting upon the first failure.

The Test documentation hints that this may be possible. There’s an exit_on_error option, although that seems to only apply to the Base tests. There’s also a FallbackTestSet, which “throws immediately on a failure”. According to the @testset docs,

Any custom testset type (subtype of AbstractTestSet ) can be given and it will also be used for any nested @testset invocations.

FallbackTestSet is a subtype of AbstractTestSet, so I assume I can drop it in as follows to this simplified example:

using Test

#ts = Test.DefaultTestSet # This works
ts = Test.FallbackTestSet # Does not work
@testset ts begin
#@testset Test.DefaultTestSet begin # Does not work
    @testset "plus" begin
        @test 1+1 == 3
        @test 2+2 == 4
    end
end

This produces the below error at this line.

MethodError: no method matching Test.FallbackTestSet(::String)

Simple workaround is to setup a constructor that throws away the string. Should this be added to Test.jl?

Test.FallbackTestSet(desc) = Test.FallbackTestSet()

It also required some digging to figure out why the TestSet type cannot be passed directly into the macro:

@testset Test.DefaultTestSet begin # Does not work

And must instead be wrapped in a symbol:

ts = Test.DefaultTestSet
@testset ts begin

Perhaps the documentation can be clarified, especially since it seems like the syntax presented for CustomTestSet at the bottom of this section likely won’t work (need to double-check).

@testset CustomTestSet foo=4 "custom testset inner 2" begin

So now I have exit on failure working, and the logs are less spammy, but still contain a lot of unnecessary info for highly nested testsets. I just want the first Test Failed block.

Test Failed at /home/miles/.julia/dev/DataStructures/test/test_binheap.jl:54
  Expression: isequal(extract_all!(hmin), [1, 2, 3, 4, 7, 8, 9, 10, 14, 16])
   Evaluated: isequal([1, 3, 3, 3, 3, 3, 3, 3, 3, 9], [1, 2, 3, 4, 7, 8, 9, 10, 14, 16])
Error During Test at /home/miles/.julia/dev/DataStructures/test/test_binheap.jl:53
  Got exception outside of a @test
  There was an error during testing
  
Error During Test at /home/miles/.julia/dev/DataStructures/test/test_binheap.jl:29
  Got exception outside of a @test
  There was an error during testing
  caused by [exception 1]
  There was an error during testing
  
Error During Test at /home/miles/.julia/dev/DataStructures/test/test_binheap.jl:28
  Got exception outside of a @test
  There was an error during testing
  caused by [exception 2]
  There was an error during testing
  caused by [exception 1]
  There was an error during testing
  
Error During Test at /home/miles/.julia/dev/DataStructures/test/test_binheap.jl:5
  Got exception outside of a @test
  There was an error during testing
  caused by [exception 3]
  There was an error during testing
  caused by [exception 2]
  There was an error during testing
  caused by [exception 1]
  There was an error during testing
  
Error During Test at /home/miles/.julia/dev/DataStructures/test/test_binheap.jl:3
  Got exception outside of a @test
  There was an error during testing
  caused by [exception 4]
  There was an error during testing
  caused by [exception 3]
  There was an error during testing
  caused by [exception 2]
  There was an error during testing
  caused by [exception 1]
  There was an error during testing
  
Error During Test at /home/miles/.julia/dev/DataStructures/test/shorttest.jl:18
  Got exception outside of a @test
  LoadError: There was an error during testing
  in expression starting at /home/miles/.julia/dev/DataStructures/test/test_binheap.jl:3
  caused by [exception 5]
  There was an error during testing
  caused by [exception 4]
  There was an error during testing
  caused by [exception 3]
  There was an error during testing
  caused by [exception 2]
  There was an error during testing
  caused by [exception 1]
  There was an error during testing
  
ERROR: There was an error during testing

This could be changed by modifying Test.jl, but I was hoping to get a simple hack working by outputting the first few lines of a redirect_stdout, but that requires more troubleshooting effort.

Wondering if I’m missing any simpler ways to output details on the first test failure.

2 Likes
2

Simply omitting the @testset should do the trick.

From the Test documentation:

In the event a test fails, the default behavior is to throw an exception immediately.

3

Omitting the highest wrapping @testset only produces the desired quick exit behavior if all the children are @test (as is the case in my posted simple example). But the existing package tests I’m running have many levels of @testset nesting.

In the following snippet, omitting the highest-level wrapper will still run all four @test statements of @testset "plus", even if the first @test fails.

#@testset "wrapper" # omitted
    @testset "plus" begin # everything in this testset runs
        @testset "plus 1" begin
            @test 1+1 == 3 # this runs and fails
            @test 2+1 == 4 # also runs
        end
        @testset "plus 2" begin
            @test 1+2 == 3 # also runs
            @test 2+2 == 5 # also runs
        end
    end
    @testset "minus" begin
        @testset "minus 1" begin
            @test 1-1 == 3
            @test 2-2 == 4
        end
        @testset "minus 2" begin
            @test 1-2 == 3
            @test 2-2 == 4
        end
    end
#end # wrapper omitted

I’m looking for something to wrap children @testsets that will exit on the first throw of an inner @test. FallbackTestSet accomplishes this, but the exceptions bubble-up through all parents with increasingly-verbose and unwanted error messages.

4

Here’s the workaround for reducing the nested exception printout. Might be nice for behavior like this to be available through Test.jl.

Test.FallbackTestSet(desc) = Test.FallbackTestSet()

function my_tests()
    ts = Test.FallbackTestSet
    @testset ts begin
        # Your tests go here
        include("test_binheap.jl")
    end
end

function run_tests()
    try
        my_tests()
        println("Tests pass")
    catch
    end
end

function func_to_str(f)
    let old_stdout = stdout
        rd, = redirect_stdout()
        try
            f()
        finally
            redirect_stdout(old_stdout) # restore original stdout
        end
        output = String(readavailable(rd))
        return output
    end
end

function first_n_lines(n, s)
    #join(split(s, '\n')[1:n],'\n') # Possible bounds error
    #join(first(split(s, '\n'), n),'\n') # Surprised this isn't built-in
    join(first_n(split(s, '\n'), n),'\n')
end

function first_n(a, n)
    n = min(n, length(a))
    a[1:n]
end

println(first_n_lines(3, func_to_str(run_tests)))

Prints out something like the following:

Test Failed at /home/miles/.julia/dev/DataStructures/test/test_binheap.jl:54
  Expression: isequal(extract_all!(hmin), [1, 2, 3, 4, 7, 8, 9, 10, 14, 16])
   Evaluated: isequal([1, 3, 3, 3, 3, 3, 3, 3, 3, 9], [1, 2, 3, 4, 7, 8, 9, 10, 14, 16])