Print debug info for failed test

General Usage
1

I am testing some complex functionality with randomly generated values. In case the test fails, I would like to print those values so that I can isolate and debug the problem — but essentially, I am looking for way to execute something conditionally on failing a test.

MWE:

using Test

f(x) = x ≥ 0.01

@testset "example" begin
    for _ in 1:1000
        r = rand()
        @test f(r)
        # QUESTION: when failed, want to print @info r
    end
end
3 Likes
2

When inside a testset, @test seems to return a Tuple on error (instead of a Pass value on success). So that something like the following should work (adjust to your taste):

onfail(body, x) = error("I might have overlooked something: $x")
onfail(body, _::Test.Pass) = nothing
onfail(body, _::Tuple{Test.Fail,T}) where {T} = body()

On your (slightly modified) example:

julia> f(x) = x ≥ 0.01
f (generic function with 1 method)

julia> @testset "example" begin
           for r in 0:0.005:1
               onfail(@test f(r)) do
                   @info r
               end
           end
       end
example: Test Failed at REPL[5]:3
  Expression: f(r)
Stacktrace:
 [1] top-level scope at REPL[5]:3
 [2] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083
 [3] top-level scope at REPL[5]:2
[ Info: 0.0
example: Test Failed at REPL[5]:3
  Expression: f(r)
Stacktrace:
 [1] top-level scope at REPL[5]:3
 [2] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083
 [3] top-level scope at REPL[5]:2
[ Info: 0.005
Test Summary: | Pass  Fail  Total
example       |  199     2    201
ERROR: Some tests did not pass: 199 passed, 2 failed, 0 errored, 0 broken.
5 Likes
3

Thanks. It seems that the actual return types of @test don’t match the docstring, which doesn’t say anything about a Tuple.

Returns a Pass Result if it does, a Fail Result if it is
false, and an Error Result if it could not be evaluated.

eg

julia> VERSION
v"1.2.0-DEV.557"

julia> using Test

julia> @testset "example" begin
           @show typeof(@test 1 == 1)
           @show typeof(@test 1 == 2)
           @show typeof(@test (1 - "a"))
       end
typeof(#= REPL[15]:2 =# @test(1 == 1)) = Test.Pass
example: Test Failed at REPL[15]:3
  Expression: 1 == 2
   Evaluated: 1 == 2
Stacktrace:
 [1] top-level scope at show.jl:573
 [2] top-level scope at REPL[15]:3
 [3] top-level scope at /home/tamas/src/julia-git/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1113
 [4] top-level scope at REPL[15]:2
typeof(#= REPL[15]:3 =# @test(1 == 2)) = Tuple{Test.Fail,Array{Union{Ptr{Nothing}, Base.InterpreterIP},1}}
example: Error During Test at REPL[15]:4
  Test threw exception
  Expression: 1 - "a"
  MethodError: no method matching -(::Int64, ::String)
  Closest candidates are:
    -(::Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}) at int.jl:51
    -(::T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}, !Matched::T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}) where T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8} at int.jl:52
    -(::Union{Int16, Int32, Int64, Int8}, !Matched::BigInt) at gmp.jl:470
    ...
  Stacktrace:
   [1] top-level scope at show.jl:573
   [2] top-level scope at REPL[15]:4
   [3] top-level scope at /home/tamas/src/julia-git/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1113
   [4] top-level scope at REPL[15]:2
  
typeof(#= REPL[15]:4 =# @test(1 - "a")) = Tuple{Test.Error,Bool}
Test Summary: | Pass  Fail  Error  Total
example       |    1     1      1      3
ERROR: Some tests did not pass: 1 passed, 1 failed, 1 errored, 0 broken.

I think I will report an issue for this.

1 Like
4

Yes. And another issue is that @test behaves differently when used within a testset than it does in a “standalone” use. In this latter case, failures and errors seem to cause an exception to be thrown:

julia> @show typeof(@test 1 == 1)
typeof(#= REPL[2]:1 =# @test(1 == 1)) = Test.Pass
Test.Pass

julia> @show typeof(@test 1 == 2)
Test Failed at REPL[3]:1
  Expression: 1 == 2
   Evaluated: 1 == 2
ERROR: There was an error during testing

julia> @show typeof(@test (1 - "a"))
Error During Test at REPL[4]:1
  Test threw exception
  Expression: 1 - "a"
  MethodError: no method matching -(::Int64, ::String)
  Closest candidates are:
    -(::Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}) at int.jl:51
    -(::T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}, !Matched::T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}) where T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8} at int.jl:52                                                 
    -(::Union{Int16, Int32, Int64, Int8}, !Matched::BigInt) at gmp.jl:449
    ...
  Stacktrace:
   [1] top-level scope at show.jl:555
   [2] eval(::Module, ::Any) at ./boot.jl:328
   [3] eval_user_input(::Any, ::REPL.REPLBackend) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/REPL/src/REPL.jl:85
   [4] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/REPL/src/REPL.jl:117 [inlined]
   [5] (::getfield(REPL, Symbol("##26#27")){REPL.REPLBackend})() at ./task.jl:259
ERROR: There was an error during testing
2 Likes
5

Thanks for confirming, I opened

https://github.com/JuliaLang/julia/issues/31495

1 Like
7

Just realized that I never got back here after the issue above was resolved, so in case anyone finds this vintage discussion: the pattern

@testset "my random 😉 tests" begin
    for i in 0:3
        debug = false
        debug |= !(@test(f(i)) isa Test.Pass)
        debug |= !(@test(g(i)) isa Test.Pass)
        if debug
            @info "test did not pass, printing debug info" i
        end
    end
end

works nicely for me for the purposes mentioned above.

(edit: !=== was too strict, corrected to !isa)

2 Likes