# How to add description to failed unit tests?

**URL:** <https://discourse.julialang.org/t/how-to-add-description-to-failed-unit-tests/40107>\
**Category:** General Usage\
**Tags:** testing\
**Created:** [May 25, 2020, 4:59am UTC](https://discourse.julialang.org/t/how-to-add-description-to-failed-unit-tests/40107 "2020-05-25T04:59:23Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [May 25, 2020, 4:59am UTC](https://discourse.julialang.org/t/how-to-add-description-to-failed-unit-tests/40107/1 "2020-05-25T04:59:23Z")

</div>

In our numerics package we rely on Julia’s `Test` for unit testing. Sometimes we have multiple values returned from a function which we would like to check separately (e.g., L2 errors for different variables). Conceptually they all belong to the same test set, but we would still like to be able to see which of the values failed. Is there a way to add a descriptive text to tests that gets displayed next to the failed test result that can achieve this?

**Example**  
This is the current output for a failed test:

```julia
julia> using Test

julia> l2 = [0.1]; l2_measured = [0.11];

julia> @test isapprox(l2[1], l2_measured[1])
Test Failed at REPL[4]:1
  Expression: isapprox(l2[1], l2_measured[1])
   Evaluated: isapprox(0.1, 0.11)
ERROR: There was an error during testing

```

What we would like to see instead is something like this:

```julia
julia> @test isapprox(l2[1], l2_measured[1]) "L2 error: density"
Test Failed at REPL[4]:1
  Expression: isapprox(l2[1], l2_measured[1])
   Evaluated: isapprox(0.1, 0.11)
 Description: L2 error: density
ERROR: There was an error during testing

```

It seems that something like this is not officially supported yet, but maybe someone knows a good workaround? I already tried highjacking the `@test` macro by chaining expressions (e.g., `!isempty("L2 error: density") && isapprox(...)` but that results in the expression/evaluated info not being shown anymore.

**Edit:** Using a `@testset` for each sub-test is not a suitable approach, as described [below](https://discourse.julialang.org/t/how-to-add-description-to-failed-unit-tests/40107/3).

---

<div class="post-metadata">

**Author:** ![virtualgraham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/virtualgraham/32/15200_2.png) [@virtualgraham](https://discourse.julialang.org/u/virtualgraham)\
**Post date:** [May 25, 2020, 5:05am UTC](https://discourse.julialang.org/t/how-to-add-description-to-failed-unit-tests/40107/2 "2020-05-25T05:05:19Z")

</div>

Perhaps put it into a `@testset`

```julia
@testset "L2 density"
   @test isapprox(l2[1], l2_measured[1])
end

```

This would result in someting like  
`"L2 density" Test Failed at ...`

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [May 25, 2020, 5:42am UTC](https://discourse.julialang.org/t/how-to-add-description-to-failed-unit-tests/40107/3 "2020-05-25T05:42:02Z")

</div>

> [@virtualgraham](#):
>
> Perhaps put it into a `@testset`

I thought about this too (should’ve put it in the question above), but it does not fit the bill:

- it adds a lot of visual noise even in case of success
- it defies the purpose of a separation into tests and test sets (by adding hundreds of `@testset`s with just a single test in it), especially since the test is really “check for L2 error” and the info on which variable fails is only to help users with tracking down errors faster

Thus I am still looking for alternatives (and/or community input whether this is something to write up as a feature request).

---

<div class="post-metadata">

**Author:** ![DrChainsaw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drchainsaw/32/8497_2.png) [@DrChainsaw](https://discourse.julialang.org/u/DrChainsaw)\
**Post date:** [May 25, 2020, 8:37am UTC](https://discourse.julialang.org/t/how-to-add-description-to-failed-unit-tests/40107/4 "2020-05-25T08:37:57Z")

</div>

I was initially a little bit annoyed by this, but I eventually turned and instead stopped using failure strings in other languages and started using more descriptive names.

Anyways, fwiw here is a low effort test macro with a failure string. You can probably whip up something better with a little more effort, maybe by wrapping `@assert` instead.

```julia
julia> macro testmsg(ex, str)
       quote
          try
             @test $ex
          catch e
             println($str)
          end
       end
       end
@testmsg (macro with 1 method)

julia> @testmsg 1==1 "Fail"
Test Passed

julia> @testmsg 1==2 "Fail"
Test Failed at none:4
  Expression: 1 == 2
   Evaluated: 1 == 2
Fail

```

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [May 25, 2020, 9:18am UTC](https://discourse.julialang.org/t/how-to-add-description-to-failed-unit-tests/40107/5 "2020-05-25T09:18:39Z")

</div>

> [@DrChainsaw](#):
>
> I eventually turned and instead stopped using failure strings in other languages and started using more descriptive names

What do you mean exactly by this? I don’t know how I would be able to use a descriptive name if most tests consist of floating point comparisons, as `@test isapprox(...)` is not very descriptive in itself, at least if you iterate over arrays.

One issue I see with your proposed snippet is that it probably disables the rest of the testing infrastructure, as you add additional output & stop throwing an exception on test failure.

---

<div class="post-metadata">

**Author:** ![DrChainsaw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drchainsaw/32/8497_2.png) [@DrChainsaw](https://discourse.julialang.org/u/DrChainsaw)\
**Post date:** [May 25, 2020, 9:59am UTC](https://discourse.julialang.org/t/how-to-add-description-to-failed-unit-tests/40107/6 "2020-05-25T09:59:58Z")

</div>

I didn’t mean to imply that I’m certain you’ll find it satisfactory and I’m sorry if it came off that way. There is obviously no one size fits all for how to apply “started using more descriptive names”.

Just to make it clear there is no magic, here is one example with your MWE:

```julia
julia> l2_error_expected = [0.1];

julia> l2_error_measured = [0.11];

julia> @test l2_error_expected[1] ≈ l2_error_measured[1]
Test Failed at none:1
  Expression: l2_error_expected[1] ≈ l2_error_measured[1]
   Evaluated: 0.1 ≈ 0.11

```

To me this is pretty readble and has the advantage that it will look familiar to anyone who is used to the julia framework

One nifty thing that at least I discovered quite late which in some cases can mitigate the “too many testsets” issue is that it is possible to loop in testsets, generating one set for each iterable.

```julia
julia> @testset "Dummy" begin
       @testset "test for $i" for i in 1:5
       @test min(3,i) == i
       end
       end

Test Summary: | Pass Fail Total
Dummy | 3 2 5
  test for 1 | 1 1
  test for 2 | 1 1
  test for 3 | 1 1
  test for 4 | 1 1
  test for 5 | 1 1

# Outer testset supress visual noise if successful
julia> @testset "Dummy" begin
       @testset "test for $i" for i in 1:3
       @test min(3,i) == i
       end
       end
Test Summary: | Pass Total
Dummy | 3 3

```

Matter of taste ofc, but I find this visually appealing. May or may not apply in your case though.

w.r.t the macro, yeah, you’d want to rethrow the caught error. Not sure if this is enough to not muck up the testframework as I haven’t tested it.

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [May 26, 2020, 3:26pm UTC](https://discourse.julialang.org/t/how-to-add-description-to-failed-unit-tests/40107/7 "2020-05-26T15:26:20Z")

</div>

Thank you for sharing your personal approach to mitigating the “too many testsets” issue. It is very usable in cases where you can re-arrange your test setup as a for loop. While I still do not think that this is worthy of being the “canonical” approach (I think Julia’s unit testing framework could do better), I’ll use this approach for now.