# Print debug info for failed test

**URL:** <https://discourse.julialang.org/t/print-debug-info-for-failed-test/22311>\
**Category:** General Usage\
**Tags:** question\
**Created:** [March 25, 2019, 4:05pm UTC](https://discourse.julialang.org/t/print-debug-info-for-failed-test/22311 "2019-03-25T16:05:23Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [March 25, 2019, 4:05pm UTC](https://discourse.julialang.org/t/print-debug-info-for-failed-test/22311/1 "2019-03-25T16:05:24Z")

</div>

I am testing some complex functionality with randomly generated values. In case the test fails, I would like to print those values so that I can isolate and debug the problem — but essentially, I am looking for way to execute something conditionally on failing a test.

MWE:

```julia
using Test

f(x) = x ≥ 0.01

@testset "example" begin
    for _ in 1:1000
        r = rand()
        @test f(r)
        # QUESTION: when failed, want to print @info r
    end
end

```

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [March 25, 2019, 10:03pm UTC](https://discourse.julialang.org/t/print-debug-info-for-failed-test/22311/2 "2019-03-25T22:03:32Z")

</div>

When inside a testset, `@test` seems to return a `Tuple` on error (instead of a `Pass` value on success). So that something like the following should work (adjust to your taste):

```julia
onfail(body, x) = error("I might have overlooked something: $x")
onfail(body, _::Test.Pass) = nothing
onfail(body, _::Tuple{Test.Fail,T}) where {T} = body()

```

On your (slightly modified) example:

```julia
julia> f(x) = x ≥ 0.01
f (generic function with 1 method)

julia> @testset "example" begin
           for r in 0:0.005:1
               onfail(@test f(r)) do
                   @info r
               end
           end
       end
example: Test Failed at REPL[5]:3
  Expression: f(r)
Stacktrace:
 [1] top-level scope at REPL[5]:3
 [2] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083
 [3] top-level scope at REPL[5]:2
[ Info: 0.0
example: Test Failed at REPL[5]:3
  Expression: f(r)
Stacktrace:
 [1] top-level scope at REPL[5]:3
 [2] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083
 [3] top-level scope at REPL[5]:2
[ Info: 0.005
Test Summary: | Pass Fail Total
example | 199 2 201
ERROR: Some tests did not pass: 199 passed, 2 failed, 0 errored, 0 broken.

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [March 27, 2019, 8:29am UTC](https://discourse.julialang.org/t/print-debug-info-for-failed-test/22311/3 "2019-03-27T08:29:49Z")

</div>

Thanks. It seems that the actual return types of `@test` don’t match the docstring, which doesn’t say anything about a `Tuple`.

> Returns a `Pass` `Result` if it does, a `Fail` `Result` if it is  
> `false`, and an `Error` `Result` if it could not be evaluated.

eg

```julia
julia> VERSION
v"1.2.0-DEV.557"

julia> using Test

julia> @testset "example" begin
           @show typeof(@test 1 == 1)
           @show typeof(@test 1 == 2)
           @show typeof(@test (1 - "a"))
       end
typeof(#= REPL[15]:2 =# @test(1 == 1)) = Test.Pass
example: Test Failed at REPL[15]:3
  Expression: 1 == 2
   Evaluated: 1 == 2
Stacktrace:
 [1] top-level scope at show.jl:573
 [2] top-level scope at REPL[15]:3
 [3] top-level scope at /home/tamas/src/julia-git/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1113
 [4] top-level scope at REPL[15]:2
typeof(#= REPL[15]:3 =# @test(1 == 2)) = Tuple{Test.Fail,Array{Union{Ptr{Nothing}, Base.InterpreterIP},1}}
example: Error During Test at REPL[15]:4
  Test threw exception
  Expression: 1 - "a"
  MethodError: no method matching -(::Int64, ::String)
  Closest candidates are:
    -(::Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}) at int.jl:51
    -(::T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}, !Matched::T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}) where T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8} at int.jl:52
    -(::Union{Int16, Int32, Int64, Int8}, !Matched::BigInt) at gmp.jl:470
    ...
  Stacktrace:
   [1] top-level scope at show.jl:573
   [2] top-level scope at REPL[15]:4
   [3] top-level scope at /home/tamas/src/julia-git/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1113
   [4] top-level scope at REPL[15]:2
  
typeof(#= REPL[15]:4 =# @test(1 - "a")) = Tuple{Test.Error,Bool}
Test Summary: | Pass Fail Error Total
example | 1 1 1 3
ERROR: Some tests did not pass: 1 passed, 1 failed, 1 errored, 0 broken.

```

I think I will report an issue for this.

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [March 27, 2019, 8:37am UTC](https://discourse.julialang.org/t/print-debug-info-for-failed-test/22311/4 "2019-03-27T08:37:24Z")

</div>

> [@Tamas\_Papp](#):
>
> Thanks. It seems that the actual return types of `@test` don’t match the docstring, which doesn’t say anything about a `Tuple` .

Yes. And another issue is that `@test` behaves differently when used within a testset than it does in a “standalone” use. In this latter case, failures and errors seem to cause an exception to be thrown:

```julia
julia> @show typeof(@test 1 == 1)
typeof(#= REPL[2]:1 =# @test(1 == 1)) = Test.Pass
Test.Pass

```

```julia
julia> @show typeof(@test 1 == 2)
Test Failed at REPL[3]:1
  Expression: 1 == 2
   Evaluated: 1 == 2
ERROR: There was an error during testing

```

```julia
julia> @show typeof(@test (1 - "a"))
Error During Test at REPL[4]:1
  Test threw exception
  Expression: 1 - "a"
  MethodError: no method matching -(::Int64, ::String)
  Closest candidates are:
    -(::Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}) at int.jl:51
    -(::T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}, !Matched::T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}) where T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8} at int.jl:52                                                 
    -(::Union{Int16, Int32, Int64, Int8}, !Matched::BigInt) at gmp.jl:449
    ...
  Stacktrace:
   [1] top-level scope at show.jl:555
   [2] eval(::Module, ::Any) at ./boot.jl:328
   [3] eval_user_input(::Any, ::REPL.REPLBackend) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/REPL/src/REPL.jl:85
   [4] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/REPL/src/REPL.jl:117 [inlined]
   [5] (::getfield(REPL, Symbol("##26#27")){REPL.REPLBackend})() at ./task.jl:259
ERROR: There was an error during testing

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [March 27, 2019, 8:46am UTC](https://discourse.julialang.org/t/print-debug-info-for-failed-test/22311/5 "2019-03-27T08:46:17Z")

</div>

Thanks for confirming, I opened

[https://github.com/JuliaLang/julia/issues/31495](https://github.com/JuliaLang/julia/issues/31495)

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [June 21, 2024, 10:39am UTC](https://discourse.julialang.org/t/print-debug-info-for-failed-test/22311/7 "2024-06-21T10:39:27Z")

</div>

Just realized that I never got back here after the issue above was resolved, so in case anyone finds this vintage discussion: the pattern

```julia
@testset "my random 😉 tests" begin
    for i in 0:3
        debug = false
        debug |= !(@test(f(i)) isa Test.Pass)
        debug |= !(@test(g(i)) isa Test.Pass)
        if debug
            @info "test did not pass, printing debug info" i
        end
    end
end

```

works nicely for me for the purposes mentioned above.

(_edit_: `!===` was too strict, corrected to `!isa`)