# MetaTesting.jl: Who tests the test-suites?

**URL:** <https://discourse.julialang.org/t/metatesting-jl-who-tests-the-test-suites/99273>\
**Category:** Package Announcements\
**Created:** [May 23, 2023, 12:37pm UTC](https://discourse.julialang.org/t/metatesting-jl-who-tests-the-test-suites/99273 "2023-05-23T12:37:08Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [May 23, 2023, 12:37pm UTC](https://discourse.julialang.org/t/metatesting-jl-who-tests-the-test-suites/99273/1 "2023-05-23T12:37:08Z")

</div>

> **[GitHub - JuliaTesting/MetaTesting.jl: Test the testers](https://github.com/JuliaTesting/MetaTesting.jl)**
>
> Test the testers. Contribute to JuliaTesting/MetaTesting.jl development by creating an account on GitHub.

A not so rare notion in julia is the idea of a [test suite function](https://invenia.github.io/blog/2020/11/06/interfacetesting/#the-test-suite).  
That’s some function that you call from your test code which runs a bunch of `@test` calls to check that an interface has been implemented correctly.

[ChainRulesTestUtils.jl](https://github.com/JuliaDiff/ChainRulesTestUtils.jl/) is a package built around some of the more sophisticated such test suite functions.  
They get pretty complicated. So they themselves need tests.  
It is pretty easy to test the happy path: make sure things pass that should pass.  
But how do you test the unhappy path, making sure the right things fail for the right reasons?  
Enter MetaTest.jl

For several years this code has lived in ChainRulesTestUtils.jl’s `/tests` as a helper file. (Along side its meta-meta-tests.)  
@sethaxen recently extracted it into its own package, for using in another project.  
So now you can use it too.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [May 23, 2023, 12:48pm UTC](https://discourse.julialang.org/t/metatesting-jl-who-tests-the-test-suites/99273/2 "2023-05-23T12:48:20Z")

</div>

Hmm… but who tests the package that tests the test-suites?

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [May 23, 2023, 12:48pm UTC](https://discourse.julialang.org/t/metatesting-jl-who-tests-the-test-suites/99273/3 "2023-05-23T12:48:37Z")

</div>

What’s the difference of `MetaTesting.errors` from `@test_throws`?

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [May 23, 2023, 12:59pm UTC](https://discourse.julialang.org/t/metatesting-jl-who-tests-the-test-suites/99273/4 "2023-05-23T12:59:00Z")

</div>

They are quiet similar.  
You can see a [use of `errors` here](https://github.com/JuliaDiff/ChainRulesTestUtils.jl/blob/c16d006de6b95808cf968d6ff1fa0ab2b8c5b676/test/testers.jl#L172-L175)  
The particular thing `@test errors(f)` does is make sure that `f()` results in the Error count of an enclosing testsuite being incremented, and a particular message being displayed.

More useful and interesting is `MetaTesting.fails` which doesn’t have a equivalent. [Example](https://github.com/JuliaDiff/ChainRulesTestUtils.jl/blob/c16d006de6b95808cf968d6ff1fa0ab2b8c5b676/test/testers.jl#L629). It tells you if the Fail count of the enclosing test suite was incremented.

`errors` really is provided for symmetry with `fails`.  
In most cases you should be able to use `@test_errors` instead, since erroring test suites rethrow the errors that end them, IIRC.

---

<div class="post-metadata">

**Author:** ![sethaxen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sethaxen/32/35604_2.png) [@sethaxen](https://discourse.julialang.org/u/sethaxen)\
**Post date:** [May 23, 2023, 1:57pm UTC](https://discourse.julialang.org/t/metatesting-jl-who-tests-the-test-suites/99273/5 "2023-05-23T13:57:35Z")

</div>

Thanks for announcing, @oxinabox! The package will soon be registered; we may modify the API a bit first to support more use cases (see issues).

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [May 23, 2023, 2:13pm UTC](https://discourse.julialang.org/t/metatesting-jl-who-tests-the-test-suites/99273/6 "2023-05-23T14:13:37Z")

</div>

> [@oxinabox](#):
>
> `MetaTesting.fails` which doesn’t have a equivalent

Shouldn’t there be something like `@test_throws TestFailException`?

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [May 23, 2023, 3:30pm UTC](https://discourse.julialang.org/t/metatesting-jl-who-tests-the-test-suites/99273/7 "2023-05-23T15:30:31Z")

</div>

That only occurs if the testset is the outermost testset. (According to **dynamic scope** )  
Otherwise tests don’t throw on failure – they push failure onto the parent testset.  
Actually this might be a reason you need `errors` too. To stop the error being cause by the test suites testset and then passed on to the parent testset.

I wrote this several years ago so it’s a bit hard to remember this stuff. The way testsets work is weirder than one might think and I often have to refer back to this package or TestReports.jl to work it out