How to test that a testset fails expectedly?

General Usage
1

Let’s say I have a package that defines a test macro. The macro creates some code that runs a @testset and inside a couple @tests. If I want to test that the macro works, I have to test that it succeeds if the conditions are right and that it fails when the conditions are wrong.

To test that it succeeds, I can just use it:

@testset "test that things work" begin
    @my_test some + successful + code
end

Now my question is, how do I test that the inner testset correctly fails when it’s supposed to? This doesn’t work because the function doesn’t actually throw, but it the spirit of what I’m trying to do:

@testset "test that things fail" begin
    @test_throws TestFailure @my_test some + failing + code
end

If I do something with the TestSet returned from @my_test I still think I can’t keep the outer testset from failing due to the way Test works internally, I can’t tell the outer testset to ignore the inner.

2

The global state makes things pretty dicey there.

Perhaps you could create your own testset type that would do the right thing on the failure of an inner testset? Probably by customizing Test.record somehow. That @customtestset could then wrap an invocation of your testing macro.

1 Like
3

More ideas:

  • You wrote a @my_test macro. You could implement the macro by having it do most of its work in a function. Then use the function in the failing test. That would test most of the functionality.
  • And a nice way to test macros is to use @macroexpand. So you test expansion, not execution.
  • And this is silly, but you could use run(pipeline(julia -e $code, stderr=devnull), wait=false) to run a tiny testset that fails and test the exitcode to see that it failed. It would work for this one test.
4

Thanks for the suggestions, what I did in the end was to write a variant of the macro reusing most of the code that explicitly tests the failure case. Seems what I wanted to do was not possible the way Test.jl is designed

5

There is GitHub - JuliaTesting/MetaTesting.jl: Test the testers which I think is exactly for this purpose

3 Likes
6

Nice that looks exactly like what I was thinking about

1 Like