I understand that
@assert 2 == 3; println("never")
stops execution. But why does
@test 2 == 3; println("never")
abort execution, even with Error? I mean it is just a test which failed, but this should not result in a breaking error.
Does it actually throw an error or does it “just” print that there was an error? What’s the error message you observe? How do you run julia to observe this?
(I’ve moved the topic to “Usage”, since “Meta Discussion” is about discussion of this platform.)
The error message is:
Test Failed at …
Expression: 2 == 3
Evaluated: 2 == 3
ERROR: LoadError: There was an error during testing in expression starting at …
Unfortunately, Julia aborts execution and will never print “never”.
What do you want to do? @test is for unit testing, maybe it isn’t the right tool for what you’re trying to achieve
@test is intended to be used for unit testing inside a @testset, but I’m not sure this erroring is intended behavior either. A cursory look through the callstack indicates that all test failures are treated as having thrown an error, which obviously isn’t true. Ironically, this change seems to have been introduced as part of making tests communicate their failures better…
From my point of view, either behavior is fine, but I can see why you’d think otherwise.
It is okay to see the error message, if the test fails. But execution should just continue with the following instructions, which could be further, more detailed tests.
From a library author POV, encouraging people to use @testset instead of single standalone @test statements seems desirable.
The macro @testset has the same behavior and the following example will break and never print “never”:
@testset begin
@test 2 == 3; println("never")
end
println("never")
There are a few distinct behaviors here that behave quite differently (and I think at least two are buggy):
- Running a file
Contents:
using Test
@testset begin
@test 2 == 3; println("never1")
end
println("never2")
run using julia test_file.jl produces this output for me:
test set: Test Failed at /d/Documents/Projects/juliaProjects/Discourse/test_stuff/test.jl:4
Expression: 2 == 3
Evaluated: 2 == 3
Stacktrace:
[1] macro expansion
@ ~/julia/usr/share/julia/stdlib/v1.7/Test/src/Test.jl:445 [inlined]
[2] macro expansion
@ /d/Documents/Projects/juliaProjects/Discourse/test_stuff/test.jl:4 [inlined]
[3] macro expansion
@ ~/julia/usr/share/julia/stdlib/v1.7/Test/src/Test.jl:1282 [inlined]
[4] top-level scope
@ /d/Documents/Projects/juliaProjects/Discourse/test_stuff/test.jl:4
never1
Test Summary: | Fail Total
test set | 1 1
ERROR: LoadError: Some tests did not pass: 0 passed, 1 failed, 0 errored, 0 broken.
in expression starting at /d/Documents/Projects/juliaProjects/Discourse/test_stuff/test.jl:3
Note that never1 is printed, between the error stacktrace and the test summary, but never2 is not printed (which seems wrong, I agree!)
- Running the same code interactively
This behaves the same as above
- Running just
@testfrom a file
File content
using Test
@test 2 == 3; println("never1")
println("never2")
Output obtained by julia test_file.jl
Test Failed at /d/Documents/Projects/juliaProjects/Discourse/test_stuff/test.jl:3
Expression: 2 == 3
Evaluated: 2 == 3
ERROR: LoadError: There was an error during testing
in expression starting at /d/Documents/Projects/juliaProjects/Discourse/test_stuff/test.jl:3
Nothing is printed (which does seem wrong, agreed).
I do remember this working though, so seems like a regression that slipped through. I think opening an issue about this (and linking this discourse thread as well as PR #37809 since I think that introduced this change) would be awesome. I think this is a problem with handling of the fallback testset (in the case of @test), but I don’t know what the issue is with @testset, that one seems weird to me.