What's the strategy for marking a broken test that you expect to throw when it's not broken?

General Usage
1

I have a function that should be throwing an exception, but currently it’s not. I want to indicate that the not-throwing behavior is broken.

I tried this, but it doesn’t work, because the @test_throws records its failure before the @test_broken gets a chance to swoop in and say “no it’s fine, that was expected”:

julia> @testset "t" begin
           @test_broken @test_throws Exception 2+2  # pretend that this is supposed to throw
       end
t: Test Failed at REPL[10]:2
  Expression: 2 + 2
    Expected: Exception
  No exception thrown
Stacktrace:
 [1] macro expansion
   @ REPL[10]:2 [inlined]
 [2] macro expansion
   @ ~/builds/julia-1.6/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
 [3] top-level scope
   @ REPL[10]:2
Test Summary: | Fail  Broken  Total
t             |    1       1      2
ERROR: Some tests did not pass: 0 passed, 1 failed, 0 errored, 1 broken.

Anyone know the supported way to do this? :pray: thanks!

2 Likes
2

That feels like a bug to me, your code is the natural way to write what you want IMO.

3

Why a bug? Macros are evaluated from the innermost to the outermost and @test_throws doesn’t return a boolean anyways, which is what @test_broken would need to evaluate

4

I guess I can’t speak to the original design intent, but I don’t like the noncomposability of it.

Maybe something like

julia> function throwsp(f, exceptiontype)::Bool
       try
           f()
       catch e
       if e isa exceptiontype
           return true
       end
       end
       return false
       end
throwsp (generic function with 1 method)

julia> throwsp(Exception) do 
       div(0x1,0x0)
       end
true

julia> @test_broken throwsp(Exception) do 
       1/0
       end
Test Broken
  Expression: throwsp(Exception) do 
    1 / 0
end
2 Likes
6

FWIW I ended up writing this, which did what I want, but it’s not very satisfactory:

@test_broken (try 2+2 catch e; e; end) isa Exception

I like your suggestion, @jzr.

@giordana: Yeah, I know that it shouldn’t work, but it felt like maybe it should when I wrote it. Even as I wrote it, i thought “no, this isn’t going to work,” but i don’t know what i’m supposed to write instead.

I imagined/hoped that maybe the @test_broken would modify the context that the @test was operating in, to “magically” invert its result or something, like by setting a task_local_storage value telling @test not to report its result or something.

Anyway, open to other suggestions still :pray:

1 Like