A not so rare notion in julia is the idea of a test suite function.
That’s some function that you call from your test code which runs a bunch of @test calls to check that an interface has been implemented correctly.
ChainRulesTestUtils.jl is a package built around some of the more sophisticated such test suite functions.
They get pretty complicated. So they themselves need tests.
It is pretty easy to test the happy path: make sure things pass that should pass.
But how do you test the unhappy path, making sure the right things fail for the right reasons?
Enter MetaTest.jl
For several years this code has lived in ChainRulesTestUtils.jl’s /tests as a helper file. (Along side its meta-meta-tests.)
@sethaxen recently extracted it into its own package, for using in another project.
So now you can use it too.
12 Likes
Hmm… but who tests the package that tests the test-suites?
1 Like
What’s the difference of MetaTesting.errors from @test_throws?
1 Like
They are quiet similar.
You can see a use of errors here
The particular thing @test errors(f) does is make sure that f() results in the Error count of an enclosing testsuite being incremented, and a particular message being displayed.
More useful and interesting is MetaTesting.fails which doesn’t have a equivalent. Example. It tells you if the Fail count of the enclosing test suite was incremented.
errors really is provided for symmetry with fails.
In most cases you should be able to use @test_errors instead, since erroring test suites rethrow the errors that end them, IIRC.
1 Like
Thanks for announcing, @oxinabox! The package will soon be registered; we may modify the API a bit first to support more use cases (see issues).
4 Likes
Shouldn’t there be something like @test_throws TestFailException?
That only occurs if the testset is the outermost testset. (According to dynamic scope)
Otherwise tests don’t throw on failure – they push failure onto the parent testset.
Actually this might be a reason you need errors too. To stop the error being cause by the test suites testset and then passed on to the parent testset.
I wrote this several years ago so it’s a bit hard to remember this stuff. The way testsets work is weirder than one might think and I often have to refer back to this package or TestReports.jl to work it out
1 Like