# Exit on first @test failure in @testset

**URL:** <https://discourse.julialang.org/t/exit-on-first-test-failure-in-testset/28496>\
**Category:** General Usage\
**Created:** [September 7, 2019, 9:06am UTC](https://discourse.julialang.org/t/exit-on-first-test-failure-in-testset/28496 "2019-09-07T09:06:48Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![milesf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/milesf/32/9289_2.png) [@milesf](https://discourse.julialang.org/u/milesf)\
**Post date:** [September 7, 2019, 9:06am UTC](https://discourse.julialang.org/t/exit-on-first-test-failure-in-testset/28496/1 "2019-09-07T09:06:48Z")

</div>

I am running package tests, and would like to save time and console spam by just exiting upon the first failure.

The [Test documentation](https://docs.julialang.org/en/v1/stdlib/Test/) hints that this may be possible. There’s an `exit_on_error` option, although that seems to only apply to the `Base` tests. There’s also a [FallbackTestSet](https://github.com/JuliaLang/julia/blob/cb76f2a0e7f2e13a2d1c500df5fe9d30d73dc82f/stdlib/Test/src/Test.jl#L709), which “throws immediately on a failure”. According to the [@testset](https://docs.julialang.org/en/v1/stdlib/Test/index.html#Test.@testset) docs,

> Any custom testset type (subtype of `AbstractTestSet` ) can be given and it will also be used for any nested `@testset` invocations.

`FallbackTestSet` is a subtype of `AbstractTestSet`, so I assume I can drop it in as follows to this simplified example:

```julia
using Test

#ts = Test.DefaultTestSet # This works
ts = Test.FallbackTestSet # Does not work
@testset ts begin
#@testset Test.DefaultTestSet begin # Does not work
    @testset "plus" begin
        @test 1+1 == 3
        @test 2+2 == 4
    end
end

```

This produces the below error at [this line](https://github.com/JuliaLang/julia/blob/cb76f2a0e7f2e13a2d1c500df5fe9d30d73dc82f/stdlib/Test/src/Test.jl#L1094).

```julia
MethodError: no method matching Test.FallbackTestSet(::String)

```

Simple workaround is to setup a constructor that throws away the string. Should this be added to `Test.jl`?

```julia
Test.FallbackTestSet(desc) = Test.FallbackTestSet()

```

It also required some digging to figure out why the TestSet type cannot be passed directly into the macro:

```julia
@testset Test.DefaultTestSet begin # Does not work

```

And must instead be wrapped in a symbol:

```julia
ts = Test.DefaultTestSet
@testset ts begin

```

Perhaps the documentation can be clarified, especially since it seems like the syntax presented for `CustomTestSet` at the bottom of [this section](https://docs.julialang.org/en/v1/stdlib/Test/index.html#Creating-Custom-AbstractTestSet-Types-1) likely won’t work (need to double-check).

```julia
@testset CustomTestSet foo=4 "custom testset inner 2" begin

```

So now I have exit on failure working, and the logs are less spammy, but still contain a lot of unnecessary info for highly nested testsets. I just want the first `Test Failed` block.

```julia
Test Failed at /home/miles/.julia/dev/DataStructures/test/test_binheap.jl:54
  Expression: isequal(extract_all!(hmin), [1, 2, 3, 4, 7, 8, 9, 10, 14, 16])
   Evaluated: isequal([1, 3, 3, 3, 3, 3, 3, 3, 3, 9], [1, 2, 3, 4, 7, 8, 9, 10, 14, 16])
Error During Test at /home/miles/.julia/dev/DataStructures/test/test_binheap.jl:53
  Got exception outside of a @test
  There was an error during testing
  
Error During Test at /home/miles/.julia/dev/DataStructures/test/test_binheap.jl:29
  Got exception outside of a @test
  There was an error during testing
  caused by [exception 1]
  There was an error during testing
  
Error During Test at /home/miles/.julia/dev/DataStructures/test/test_binheap.jl:28
  Got exception outside of a @test
  There was an error during testing
  caused by [exception 2]
  There was an error during testing
  caused by [exception 1]
  There was an error during testing
  
Error During Test at /home/miles/.julia/dev/DataStructures/test/test_binheap.jl:5
  Got exception outside of a @test
  There was an error during testing
  caused by [exception 3]
  There was an error during testing
  caused by [exception 2]
  There was an error during testing
  caused by [exception 1]
  There was an error during testing
  
Error During Test at /home/miles/.julia/dev/DataStructures/test/test_binheap.jl:3
  Got exception outside of a @test
  There was an error during testing
  caused by [exception 4]
  There was an error during testing
  caused by [exception 3]
  There was an error during testing
  caused by [exception 2]
  There was an error during testing
  caused by [exception 1]
  There was an error during testing
  
Error During Test at /home/miles/.julia/dev/DataStructures/test/shorttest.jl:18
  Got exception outside of a @test
  LoadError: There was an error during testing
  in expression starting at /home/miles/.julia/dev/DataStructures/test/test_binheap.jl:3
  caused by [exception 5]
  There was an error during testing
  caused by [exception 4]
  There was an error during testing
  caused by [exception 3]
  There was an error during testing
  caused by [exception 2]
  There was an error during testing
  caused by [exception 1]
  There was an error during testing
  
ERROR: There was an error during testing

```

This could be changed by modifying `Test.jl`, but I was hoping to get a simple hack working by outputting the first few lines of a [redirect\_stdout](https://docs.julialang.org/en/v1/base/io-network/#Base.redirect_stdout-Tuple%7BFunction,Any%7D), but that requires more troubleshooting effort.

Wondering if I’m missing any simpler ways to output details on the first test failure.

---

<div class="post-metadata">

**Author:** ![hendri54](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hendri54/32/9621_2.png) [@hendri54](https://discourse.julialang.org/u/hendri54)\
**Post date:** [September 7, 2019, 4:58pm UTC](https://discourse.julialang.org/t/exit-on-first-test-failure-in-testset/28496/2 "2019-09-07T16:58:25Z")

</div>

Simply omitting the `@testset` should do the trick.

From the [Test documentation](https://docs.julialang.org/en/v1/stdlib/Test/):

> In the event a test fails, the default behavior is to throw an exception immediately.

---

<div class="post-metadata">

**Author:** ![milesf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/milesf/32/9289_2.png) [@milesf](https://discourse.julialang.org/u/milesf)\
**Post date:** [September 7, 2019, 5:32pm UTC](https://discourse.julialang.org/t/exit-on-first-test-failure-in-testset/28496/3 "2019-09-07T17:32:47Z")

</div>

Omitting the highest wrapping `@testset` only produces the desired quick exit behavior if all the children are `@test` (as is the case in my posted simple example). But the existing package tests I’m running have many levels of `@testset` nesting.

In the following snippet, omitting the highest-level wrapper will still run all four `@test` statements of `@testset "plus"`, even if the first `@test` fails.

```julia
#@testset "wrapper" # omitted
    @testset "plus" begin # everything in this testset runs
        @testset "plus 1" begin
            @test 1+1 == 3 # this runs and fails
            @test 2+1 == 4 # also runs
        end
        @testset "plus 2" begin
            @test 1+2 == 3 # also runs
            @test 2+2 == 5 # also runs
        end
    end
    @testset "minus" begin
        @testset "minus 1" begin
            @test 1-1 == 3
            @test 2-2 == 4
        end
        @testset "minus 2" begin
            @test 1-2 == 3
            @test 2-2 == 4
        end
    end
#end # wrapper omitted

```

I’m looking for something to wrap children `@testset`s that will exit on the first throw of an inner `@test`. `FallbackTestSet` accomplishes this, but the exceptions bubble-up through all parents with increasingly-verbose and unwanted error messages.

---

<div class="post-metadata">

**Author:** ![milesf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/milesf/32/9289_2.png) [@milesf](https://discourse.julialang.org/u/milesf)\
**Post date:** [September 7, 2019, 6:52pm UTC](https://discourse.julialang.org/t/exit-on-first-test-failure-in-testset/28496/4 "2019-09-07T18:52:44Z")

</div>

Here’s the workaround for reducing the nested exception printout. Might be nice for behavior like this to be available through Test.jl.

```julia
Test.FallbackTestSet(desc) = Test.FallbackTestSet()

function my_tests()
    ts = Test.FallbackTestSet
    @testset ts begin
        # Your tests go here
        include("test_binheap.jl")
    end
end

function run_tests()
    try
        my_tests()
        println("Tests pass")
    catch
    end
end

function func_to_str(f)
    let old_stdout = stdout
        rd, = redirect_stdout()
        try
            f()
        finally
            redirect_stdout(old_stdout) # restore original stdout
        end
        output = String(readavailable(rd))
        return output
    end
end

function first_n_lines(n, s)
    #join(split(s, '\n')[1:n],'\n') # Possible bounds error
    #join(first(split(s, '\n'), n),'\n') # Surprised this isn't built-in
    join(first_n(split(s, '\n'), n),'\n')
end

function first_n(a, n)
    n = min(n, length(a))
    a[1:n]
end

println(first_n_lines(3, func_to_str(run_tests)))

```

Prints out something like the following:

```julia
Test Failed at /home/miles/.julia/dev/DataStructures/test/test_binheap.jl:54
  Expression: isequal(extract_all!(hmin), [1, 2, 3, 4, 7, 8, 9, 10, 14, 16])
   Evaluated: isequal([1, 3, 3, 3, 3, 3, 3, 3, 3, 9], [1, 2, 3, 4, 7, 8, 9, 10, 14, 16])

```