How to use `@test_skip` properly?

General Usage
1

So, we are having intermittent failures with a particular test (say @test foo). According to the docs @test_skip is made for this scenario. So:

@test_skip begin
   @test foo
end

… but CI still fails because the test is marked as Broken.

Isn’t the point of @test_skip to allow CI to pass even with broken tests, or am I missing something fundamental?

2

Currently @test_skip is to replace @test instead of wrapping around it.

FWIW, this seems like a bad API. Wrapping around other test macros is a much better one.

3

There’s @test_broken, which I understand replaces @test (not @testset).

4

The docs for @test_broken suggest that if the test passes, it will result in an error. In any case, this isn’t the heart of the issue: the issue is that any broken test will cause CI to fail. I’m not sure this is desired behavior. How do I fix this?

@yuyichao - wrapping in the block seemed to produce the expected result: a test marked broken… unless the inner @test was actually executed?

5

If you have a broken test, replace @test with @test_broken – this test must fail or CI won’t pass. If you have a flaky test that you want to skip entirely, replace @test with @test_skip – the test body is not evaluated at all (it needs to parse but that’s all).

6

This is my issue:

using LightGraphs
using LightGraphs.LinAlg
using Base.Test

@testset "foo" begin
@test_skip 1 == 2
end

yields

julia> Pkg.test("LightGraphs")
INFO: Testing LightGraphs
Test Summary: | Broken  Total
  foo         |      1      1
ERROR: LoadError: Some tests did not pass: 0 passed, 0 failed, 0 errored, 1 broken.

which causes CI to fail.

7

That problem has been fixed on 0.6. On 0.5, I just wrap the outer testset in a try.

8

thanks, @fengyang.wang. Is it possible to patch 0.5 with this as well?

9

I don’t know when this was changed or if this change is easily backportable.

10

It would be good to figure this out and see if we can backport it. At the very least, there should be an issue tracking that this is a problem on 0.5 so that we know something needs to be fixed there.

1 Like
11

https://github.com/JuliaLang/julia/issues/21008

1 Like