# TidyTest.jl -- test execution with auto discovery and a neater test summary

**URL:** <https://discourse.julialang.org/t/tidytest-jl-test-execution-with-auto-discovery-and-a-neater-test-summary/97183>\
**Category:** Package Announcements\
**Tags:** testing\
**Created:** [April 6, 2023, 3:29pm UTC](https://discourse.julialang.org/t/tidytest-jl-test-execution-with-auto-discovery-and-a-neater-test-summary/97183 "2023-04-06T15:29:08Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![HanD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hand/32/213908_2.png) [@HanD](https://discourse.julialang.org/u/HanD)\
**Post date:** [April 6, 2023, 3:29pm UTC](https://discourse.julialang.org/t/tidytest-jl-test-execution-with-auto-discovery-and-a-neater-test-summary/97183/1 "2023-04-06T15:29:08Z")

</div>

I’m happy to announce the first release of [TidyTest.jl](https://github.com/dhanak/TidyTest.jl), a package for test execution with auto discovery and a neater test summary. It is using [ProgressMeter.jl](https://github.com/timholy/ProgressMeter.jl) to report progress as the tests are completed.

## Usage

For the simplest use case, write the following lines in your `runtests.jl` file:

```julia
using TidyTest

@run_tests

```

Add `TidyTest.jl` to the dependencies of your test:

```julia
julia> using Pkg; pkg"activate test; add TidyTest"

```

And then execute your tests:

```bash
$ julia --project -e "using Pkg; Pkg.test()"

```

For example:

![](https://global.discourse-cdn.com/julialang/original/3X/8/d/8d10842d3b3003d33934e263d7b48741a8e2e08c.gif)

The `@run_tests` macro automatically discovers all Julia source files in the directory of `runtests.jl`, and includes all of them. The entire block is wrapped in a single toplevel `@testset` using the custom test set type `SpinnerTestSet`. Test progress is reported using ProgressMeter.jl, continuously updating the status as tests are completed. If some tests fail (or throw an error), the issues are reported as they happen, and a detailed test summary is printed upon completion, using the default test reporting.

### Test filtering

The macro facilitates running tests selectively. Every command line argument is used as a pattern, and only those tests are run which contain any of the arguments as a substring. The search uses smart case matching: if any of the patterns contains at least one capital letter, then matching is case-sensitive,  
otherwise its case-insensitive. To pass command line arguments to `Pkg.test()`, the `test_args` keyword argument must be used:

```bash
$ alias jlt='julia --project -e "using Pkg; Pkg.test(test_args=ARGS)"'
$ jlt some tests
# ...runs test files which have "some" or "test" occurring in their names

```

Alternatively, one can filter tests by passing a `filters` keyword argument to the `@run_tests` macro, with a list of strings:

```julia
@run_tests filters=["some", "tests"]

```

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [April 6, 2023, 8:52pm UTC](https://discourse.julialang.org/t/tidytest-jl-test-execution-with-auto-discovery-and-a-neater-test-summary/97183/2 "2023-04-06T20:52:06Z")

</div>

Neat!  
A couple of questions:

- Can I simply add `using TidyTest; @run_tests` to an existing `runtests.jl` file with `@testsets`? This would lower the entry barrier.
- Showing some progress is nice, to get a feeling of how testing goes! Did you also consider the “print a dot for each testcase” approach utilized my some Python libraries? To have a more immediately obvious visual indicator compared to changing digits.

---

<div class="post-metadata">

**Author:** ![HanD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hand/32/213908_2.png) [@HanD](https://discourse.julialang.org/u/HanD)\
**Post date:** [April 6, 2023, 9:13pm UTC](https://discourse.julialang.org/t/tidytest-jl-test-execution-with-auto-discovery-and-a-neater-test-summary/97183/3 "2023-04-06T21:13:33Z")

</div>

Hi @aplavin!

I’m glad you like it!

The crux of the `@run_tests` macro is the discovery and inclusion of Julia source files in the `test` directory, so in order to convert an existing `runtests.jl` file, you need to:

1. rename your existing `runtests.jl` to something else, preferably break it up along `@testset`s (or some by other organizing logic) into multiple files;
2. add a new `runtests.jl` with `using TidyTest; @run_tests` as its only content.

Alternatively, if you prefer to keep all your tests in the main `runtests.jl` file, just wrap the contents into a toplevel `@testset SpinnerTestSet begin ... end` construct. This will take care of the reporting without using the auto discovery feature.

Perhaps I will add this as a migration guide to the README.

As for the other part of your question: I deliberately didn’t want to print a period for each completed test, because I don’t like the varying line lengths it causes. But there is a package that does just that: [TestSetExtensions.jl](https://github.com/ssfrr/TestSetExtensions.jl). You should check that out, too. (This is in fact mentioned at the end of the README.)

---

<div class="post-metadata">

**Author:** ![HanD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hand/32/213908_2.png) [@HanD](https://discourse.julialang.org/u/HanD)\
**Post date:** [December 5, 2024, 3:35pm UTC](https://discourse.julialang.org/t/tidytest-jl-test-execution-with-auto-discovery-and-a-neater-test-summary/97183/5 "2024-12-05T15:35:17Z")

</div>

@g-gundam, have you managed to figure out how to use `@run_tests` from the REPL? The trick is to run it from the `test` directory.

```julia
julia> using TidyTest
julia> ;<enter>
shell> cd test
shell> <backspace>
julia> @run_tests;

```

It should work without problems. HTH.

---

<div class="post-metadata">

**Author:** ![g-gundam](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/g-gundam/32/47593_2.png) [@g-gundam](https://discourse.julialang.org/u/g-gundam)\
**Post date:** [December 5, 2024, 9:41pm UTC](https://discourse.julialang.org/t/tidytest-jl-test-execution-with-auto-discovery-and-a-neater-test-summary/97183/6 "2024-12-05T21:41:35Z")

</div>

@HanD - Thank you for getting back to me even though I deleted my previous post. Your solution is much better that what I was doing which was calling `Pkg.test()` with various parameters (which wasn’t terrible but not ideal). However, changing into the “test” directory and running `@run_tests` is way faster and exactly what I was looking for.

```julia-repl
julia> cd("test")

julia> @run_tests verbose=true;
Test Summary: | Pass Total Time
ReversedSeries | 4 4 0.0s
  ReversedFrame | 3 3 0.0s
  Reversed | 1 1 0.0s

julia> @run_tests verbose=true filters=["frame"];
Test Summary: | Pass Total Time
ReversedSeries | 3 3 0.0s
  ReversedFrame | 3 3 0.0s

```

That’s what I wanted.

You should definitely mention this tip in your README. TidyTest.jl was already a really nice way to run specific testsets instead of all of them, and this tip makes it even nicer to use.

PS: I’m not proficient enough in Julia to understand why this works. Why does changing into the “test” directory make it print more things?

---

<div class="post-metadata">

**Author:** ![HanD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hand/32/213908_2.png) [@HanD](https://discourse.julialang.org/u/HanD)\
**Post date:** [December 6, 2024, 5:18pm UTC](https://discourse.julialang.org/t/tidytest-jl-test-execution-with-auto-discovery-and-a-neater-test-summary/97183/7 "2024-12-06T17:18:14Z")

</div>

> [@g-gundam](#):
>
> PS: I’m not proficient enough in Julia to understand why this works. Why does changing into the “test” directory make it print more things?

The explanation is simple: the `@run_tests` macro looks for files to include and run as tests in the working directory. When you run tests via the `Pkg.test()` command, the working directory is the `test` directory. When you are in the REPL, the working directory is (at least initially) the main package directory.

---

<div class="post-metadata">

**Author:** ![HanD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hand/32/213908_2.png) [@HanD](https://discourse.julialang.org/u/HanD)\
**Post date:** [December 6, 2024, 5:28pm UTC](https://discourse.julialang.org/t/tidytest-jl-test-execution-with-auto-discovery-and-a-neater-test-summary/97183/8 "2024-12-06T17:28:13Z")

</div>

> [@g-gundam](#):
>
> You should definitely mention this tip in your README. [TidyTest.jl](https://juliahub.com/ui/Packages/General/TidyTest) was already a really nice way to run specific testsets instead of all of them, and this tip makes it even nicer to use.

See [GitHub - dhanak/TidyTest.jl: An `AbstractTestSet` implementation and a helper macro for test execution with auto discovery and a neater test summary.](https://github.com/dhanak/TidyTest.jl?tab=readme-ov-file#running-from-the-repl)

---

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [December 6, 2024, 6:31pm UTC](https://discourse.julialang.org/t/tidytest-jl-test-execution-with-auto-discovery-and-a-neater-test-summary/97183/9 "2024-12-06T18:31:00Z")

</div>

When I first saw the title, I assumed this was related to [Tidier.jl](https://github.com/TidierOrg).

You might want to consider a different name. I think Tidier.jl predates this and it also is related to R’s tidyverse.

On the other hand “tidy” is a common enough word. And I don’t think it’s a big problem if people make the wrong assumption now and then. So leaving it as is, is ok. I just wanted to make you aware.

Also, the package looks great!

---

<div class="post-metadata">

**Author:** ![g-gundam](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/g-gundam/32/47593_2.png) [@g-gundam](https://discourse.julialang.org/u/g-gundam)\
**Post date:** [December 6, 2024, 7:06pm UTC](https://discourse.julialang.org/t/tidytest-jl-test-execution-with-auto-discovery-and-a-neater-test-summary/97183/10 "2024-12-06T19:06:31Z")

</div>

Very nice, and almost perfect. It’s too bad that GitHub doesn’t support `julia-repl` for highlighting like this forum does. GitHub accepts it, but it doesn’t color the prompt differently.

## Using `julia`

```julia
julia> "Notice the prompt is not highlighted."

```

## Using `julia-repl`

```julia-repl
julia> "The prompt is colored now."

```

---

<div class="post-metadata">

**Author:** ![g-gundam](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/g-gundam/32/47593_2.png) [@g-gundam](https://discourse.julialang.org/u/g-gundam)\
**Post date:** [December 6, 2024, 7:26pm UTC](https://discourse.julialang.org/t/tidytest-jl-test-execution-with-auto-discovery-and-a-neater-test-summary/97183/11 "2024-12-06T19:26:24Z")

</div>

> [@jlapeyre](#):
>
> Also, the package looks great!

I’m happy enough with it to migrate my other projects to TidyTest.jl. I honestly wrote fewer tests, because I didn’t like the stock testing infrastructure of Julia. I didn’t want to run every testset every time. I looked around for alternatives and found them a bit confusing, granted it was earlier in my Julia journey.

Fast forward many months, and I’m not sure how I came across TidyTest.jl, but the setup looked simple. I tried it out, and the setup was indeed simple, and it did everything it said it would do.

I’m happy I found it, and if anyone ever asks, I’ll recommend it to others, too.

---

<div class="post-metadata">

**Author:** ![HanD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hand/32/213908_2.png) [@HanD](https://discourse.julialang.org/u/HanD)\
**Post date:** [December 7, 2024, 10:26am UTC](https://discourse.julialang.org/t/tidytest-jl-test-execution-with-auto-discovery-and-a-neater-test-summary/97183/12 "2024-12-07T10:26:46Z")

</div>

Well, I’m really glad to hear that!

---

<div class="post-metadata">

**Author:** ![g-gundam](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/g-gundam/32/47593_2.png) [@g-gundam](https://discourse.julialang.org/u/g-gundam)\
**Post date:** [December 16, 2024, 5:48am UTC](https://discourse.julialang.org/t/tidytest-jl-test-execution-with-auto-discovery-and-a-neater-test-summary/97183/13 "2024-12-16T05:48:06Z")

</div>

@HanD - If I wanted to provide helper functions for my tests, where is a good place to put them?

---

<div class="post-metadata">

**Author:** ![HanD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hand/32/213908_2.png) [@HanD](https://discourse.julialang.org/u/HanD)\
**Post date:** [December 16, 2024, 9:00am UTC](https://discourse.julialang.org/t/tidytest-jl-test-execution-with-auto-discovery-and-a-neater-test-summary/97183/14 "2024-12-16T09:00:56Z")

</div>

You can either put such functions directly in `runtests.jl`, before calling the `@run_tests` macro, or in a subdirectory of `test`, i.e., `test/lib/helpers.jl`. This will work because the auto-discovery of `@run_tests` won’t pick up files from subdirectories.