Preventing test error behavior

General Usage
1

Hi

Question about Test and its use; this is my setup.

I have a runtests.jl file containing @test’s and @testset’s. I load it from the shell (don’t ask; that’s what I need and want).

$ julia -e 'include(sometests/runtests.jl)'

How do I prevent the @test macros from generating errors and stopping? This seems possible using Base.runtests, but when do you call it? Also, wrapping everything in a @testset seem to break the subtests data (passed, failed, etc) collection.

Any ideas?

Thanks

Marco

2

You need to wrap @test in a @testset to prevent it from stopping the script, e.g.:

using Test

@testset "Test1" begin
    @test 0 == 1
    @test 1 == 2
end

For me to help you with the breaking subtests issue you will need to post a minimal example of a Julia file that does this.

1 Like
3

That definitely shouldn’t happen, I’m using that nesting everywhere - do you have an example showing this?

1 Like
4

Sometimes wrapping too much in a @testset can cause subtle errors because the @testset introduces an implicit local scope. It’s usually best to make sure any top level definitions like struct definitions or function definitions are not put inside a @testset.

If you’re referring to the @testset not showing the individual test results, that’s normal when all the individual @test results pass. If one or more @test fails, then the @testset will print out a more detailed list of what passed and what failed.

5

Thank you.

I feel I have figured out most of the idiosyncrasies of the Test package.

Let’s just say I was probably expecting something different.

First of all, I expected that the @test macro always printed what happened (having control about it). Secondly, I am writing a specialized test set type in order to gather results and do some computations with them, however, it is unclear (to me) how the nesting behavior works. Finally, why is there the Base.runtests function, if including a file with tests just runs them?

Apologies for the rant, but I am starting to think that a different testing framework would be nice. No, I do not have time to cook up one :smirking_face:

All the best

MA

6

My advice to you would be to start off by reading the Test manual, try to be flexible and open to doing things in the standard way - maybe try it out a bit for yourself. A lot of us are happy with just using the Test standard library, but if you know that this is completely unacceptable to you, there are packages in the General registry that might be able to provide what you want, but doing things in a non-standard way has a price.

The standard place to put tests is in the “test” directory. Do you have a good reason to do something non-standard?

7

Yes. I have a relatively good reason for doing things in a “non standard” way. Even if this means peppering tests and testsets
with println and @show.

Having said that, it has been know for “standards” to improve over time.
I am old enough to have been a witness to many such events.