Define test error message programmatically

General Usage
,
1

Is there a way for a test to display a custom error message, without giving up on the nice display of numerical values equality / comparison tests?
The workaround in Custom error message from @test macro is only a partial solution for this:

julia> using Test

julia> x, y = 1, 2;

julia> msg = "The test failed for this reason: ...";

julia> @test x == y  # numerical values, no message displayed
Test Failed at REPL[16]:1
  Expression: x == y
   Evaluated: 1 == 2

ERROR: There was an error during testing

julia> @test x == y || msg  # only symbols, message is displayed
Error During Test at REPL[19]:1
  Expression evaluated to non-Boolean
  Expression: x == y || msg
       Value: "The test failed for this reason: ..."
ERROR: There was an error during testing
2 
x == y || @warn "failure for this reason…"
@test x == y # more error message
1 Like
3 
using Test
x, y = 1, 2
#x, y = 1, 1
msg=" Custom Test ERROR Message"
x==y  || println("$msg")
@test x == y ;
4

Thanks, I don’t know why I was looking for something more complicated.

The second solution works too but here I think using @warn is better than println because it is same logging system used to display the actual error (so there is no risk of desynchronization for example).

5

Both solutions give in my tests:

ERROR: LoadError: There was an error during testing
in expression starting at /home/o/oOo/o/delMe.jl:6

Why?

6

Check the error message, it’s probably the same as mine :arrow_up: That happens because x == y is a Bool but msg is not, so I can’t compute an OR with it

7

if( x !=y ) ; println("$msg"); end gives the same effect and does not solve the issue … So there is something else going wrong and printing an additional final ERROR message making both up to now solutions having an unnecessary side-effect.

8

The solution given by Steven is correct and works on my machine, so my initial problem is solved.
I’m not sure what goes wrong in your case so if you need help please post the complete code through which you arrive at the error.

9

The complete code is given in my answer and results in printing the ERROR if run as script from within Geany text editor. In other words, you don’t have any additional output except this one of @test and the custom error message on your system?

This code runs without any side-effect of an additional ERROR message:

using Test
x, y = 1, 2
#x, y = 1, 1
msg = " Custom ERROR message "
try
    @test x == y
catch e
    println(msg)
    flush(stdout)  # Ensure the message is flushed immediately
    exit()
end
println("END")

on my system if run from within Geany.

10

This is what I get with your code, as expected. The error here is because the test itself failed.

julia> using Test

julia> x, y = 1, 2
       #x, y = 1, 1
(1, 2)

julia> msg=" Custom Test ERROR Message"
" Custom Test ERROR Message"

julia> x==y  || println("$msg")
 Custom Test ERROR Message

julia> @test x == y ;
Test Failed at REPL[5]:1
  Expression: x == y
   Evaluated: 1 == 2

ERROR: There was an error during testing
11

OK … so you are in the REPL … What happens if you run this code as script outside the REPL? Maybe you need then the code I have provided which uses try/catch to avoid an additional ERROR message printed at the end of the output ?

By the way: my question what is needed to arrive while running a script at the behavior in the REPL is still open and without a detailed answer.

In other words what you marked as solution does not work for me if run as script code (i.e. not in the REPL).

12

It’s by design. A failed test is supposed to throw an error.

2 Likes
13

OK … but then in addition to the message printed by @test there will be finally an additional error message while using

x == y || @warn "failure for this reason…"
@test x == y # more error message

in a Julia script run as script (not inside a REPL). The double messaging can be avoided using try/catch without re-throwing the error and exiting the script instead (see my another reply).

16

If that is what you want to achieve, by all means do so. One difference that might not matter for your use case but is somewhat important in CI scripts is the return code of the script.

2 Likes