How to use `@test_skip` properly?

testing

#1

So, we are having intermittent failures with a particular test (say @test foo). According to the docs @test_skip is made for this scenario. So:

@test_skip begin
   @test foo
end

… but CI still fails because the test is marked as Broken.

Isn’t the point of @test_skip to allow CI to pass even with broken tests, or am I missing something fundamental?


#2

Currently @test_skip is to replace @test instead of wrapping around it.

FWIW, this seems like a bad API. Wrapping around other test macros is a much better one.


#3

There’s @test_broken, which I understand replaces @test (not @testset).


#4

The docs for @test_broken suggest that if the test passes, it will result in an error. In any case, this isn’t the heart of the issue: the issue is that any broken test will cause CI to fail. I’m not sure this is desired behavior. How do I fix this?

@yuyichao - wrapping in the block seemed to produce the expected result: a test marked broken… unless the inner @test was actually executed?


#5

If you have a broken test, replace @test with @test_broken – this test must fail or CI won’t pass. If you have a flaky test that you want to skip entirely, replace @test with @test_skip – the test body is not evaluated at all (it needs to parse but that’s all).


#6

This is my issue:

using LightGraphs
using LightGraphs.LinAlg
using Base.Test

@testset "foo" begin
@test_skip 1 == 2
end

yields

julia> Pkg.test("LightGraphs")
INFO: Testing LightGraphs
Test Summary: | Broken  Total
  foo         |      1      1
ERROR: LoadError: Some tests did not pass: 0 passed, 0 failed, 0 errored, 1 broken.

which causes CI to fail.


#7

That problem has been fixed on 0.6. On 0.5, I just wrap the outer testset in a try.


#8

thanks, @fengyang.wang. Is it possible to patch 0.5 with this as well?


#9

I don’t know when this was changed or if this change is easily backportable.


#10

It would be good to figure this out and see if we can backport it. At the very least, there should be an issue tracking that this is a problem on 0.5 so that we know something needs to be fixed there.


#11