# How to use \`@test\_skip\` properly?

**URL:** <https://discourse.julialang.org/t/how-to-use-test-skip-properly/2628>\
**Category:** General Usage\
**Tags:** testing\
**Created:** [March 13, 2017, 1:14am UTC](https://discourse.julialang.org/t/how-to-use-test-skip-properly/2628 "2017-03-13T01:14:04Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon94023334](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@anon94023334](https://discourse.julialang.org/u/anon94023334)\
**Post date:** [March 13, 2017, 1:14am UTC](https://discourse.julialang.org/t/how-to-use-test-skip-properly/2628/1 "2017-03-13T01:14:04Z")

</div>

So, we are having intermittent failures with a particular test (say `@test foo`). According to the docs `@test_skip` is made for this scenario. So:

```julia
@test_skip begin
   @test foo
end

```

… but CI still fails because the test is marked as `Broken`.

Isn’t the point of `@test_skip` to allow CI to pass even with broken tests, or am I missing something fundamental?

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [March 13, 2017, 1:27am UTC](https://discourse.julialang.org/t/how-to-use-test-skip-properly/2628/2 "2017-03-13T01:27:28Z")

</div>

Currently `@test_skip` is to replace `@test` instead of wrapping around it.

FWIW, this seems like a bad API. Wrapping around other test macros is a much better one.

---

<div class="post-metadata">

**Author:** ![dpsanders](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dpsanders/32/3573_2.png) [@dpsanders](https://discourse.julialang.org/u/dpsanders)\
**Post date:** [March 13, 2017, 1:28am UTC](https://discourse.julialang.org/t/how-to-use-test-skip-properly/2628/3 "2017-03-13T01:28:04Z")

</div>

There’s `@test_broken`, which I understand replaces `@test` (not `@testset`).

---

<div class="post-metadata">

**Author:** ![anon94023334](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@anon94023334](https://discourse.julialang.org/u/anon94023334)\
**Post date:** [March 13, 2017, 1:30am UTC](https://discourse.julialang.org/t/how-to-use-test-skip-properly/2628/4 "2017-03-13T01:30:48Z")

</div>

The docs for `@test_broken` suggest that if the test passes, it will result in an error. In any case, this isn’t the heart of the issue: the issue is that any broken test will cause CI to fail. I’m not sure this is desired behavior. How do I fix this?

@yuyichao - wrapping in the block seemed to produce the expected result: a test marked broken… unless the inner `@test` was actually executed?

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [March 13, 2017, 3:24am UTC](https://discourse.julialang.org/t/how-to-use-test-skip-properly/2628/5 "2017-03-13T03:24:19Z")

</div>

If you have a broken test, replace `@test` with `@test_broken` – this test must fail or CI won’t pass. If you have a flaky test that you want to skip entirely, replace `@test` with `@test_skip` – the test body is not evaluated at all (it needs to parse but that’s all).

---

<div class="post-metadata">

**Author:** ![anon94023334](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@anon94023334](https://discourse.julialang.org/u/anon94023334)\
**Post date:** [March 13, 2017, 4:02am UTC](https://discourse.julialang.org/t/how-to-use-test-skip-properly/2628/6 "2017-03-13T04:02:35Z")

</div>

This is my issue:

```julia
using LightGraphs
using LightGraphs.LinAlg
using Base.Test

@testset "foo" begin
@test_skip 1 == 2
end

```

yields

```julia
julia> Pkg.test("LightGraphs")
INFO: Testing LightGraphs
Test Summary: | Broken Total
  foo | 1 1
ERROR: LoadError: Some tests did not pass: 0 passed, 0 failed, 0 errored, 1 broken.

```

which causes CI to fail.

---

<div class="post-metadata">

**Author:** ![fengyang.wang](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fengyang.wang/32/104_2.png) [@fengyang.wang](https://discourse.julialang.org/u/fengyang.wang)\
**Post date:** [March 13, 2017, 4:32am UTC](https://discourse.julialang.org/t/how-to-use-test-skip-properly/2628/7 "2017-03-13T04:32:14Z")

</div>

That problem has been fixed on 0.6. On 0.5, I just wrap the outer testset in a `try`.

---

<div class="post-metadata">

**Author:** ![anon94023334](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@anon94023334](https://discourse.julialang.org/u/anon94023334)\
**Post date:** [March 13, 2017, 4:38am UTC](https://discourse.julialang.org/t/how-to-use-test-skip-properly/2628/8 "2017-03-13T04:38:54Z")

</div>

thanks, @fengyang.wang. Is it possible to patch 0.5 with this as well?

---

<div class="post-metadata">

**Author:** ![fengyang.wang](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fengyang.wang/32/104_2.png) [@fengyang.wang](https://discourse.julialang.org/u/fengyang.wang)\
**Post date:** [March 13, 2017, 4:45am UTC](https://discourse.julialang.org/t/how-to-use-test-skip-properly/2628/9 "2017-03-13T04:45:51Z")

</div>

I don’t know when this was changed or if this change is easily backportable.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [March 13, 2017, 2:08pm UTC](https://discourse.julialang.org/t/how-to-use-test-skip-properly/2628/10 "2017-03-13T14:08:54Z")

</div>

It would be good to figure this out and see if we can backport it. At the very least, there should be an issue tracking that this is a problem on 0.5 so that we know something needs to be fixed there.

---

<div class="post-metadata">

**Author:** ![anon94023334](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@anon94023334](https://discourse.julialang.org/u/anon94023334)\
**Post date:** [March 13, 2017, 2:17pm UTC](https://discourse.julialang.org/t/how-to-use-test-skip-properly/2628/11 "2017-03-13T14:17:47Z")

</div>

[https://github.com/JuliaLang/julia/issues/21008](https://github.com/JuliaLang/julia/issues/21008)