TidyTest.jl -- test execution with auto discovery and a neater test summary

Package Announcements
1

I’m happy to announce the first release of TidyTest.jl, a package for test execution with auto discovery and a neater test summary. It is using ProgressMeter.jl to report progress as the tests are completed.

Usage

For the simplest use case, write the following lines in your runtests.jl file:

using TidyTest

@run_tests

Add TidyTest.jl to the dependencies of your test:

julia> using Pkg; pkg"activate test; add TidyTest"

And then execute your tests:

$ julia --project -e "using Pkg; Pkg.test()"

For example:

The @run_tests macro automatically discovers all Julia source files in the directory of runtests.jl, and includes all of them. The entire block is wrapped in a single toplevel @testset using the custom test set type SpinnerTestSet. Test progress is reported using ProgressMeter.jl, continuously updating the status as tests are completed. If some tests fail (or throw an error), the issues are reported as they happen, and a detailed test summary is printed upon completion, using the default test reporting.

Test filtering

The macro facilitates running tests selectively. Every command line argument is used as a pattern, and only those tests are run which contain any of the arguments as a substring. The search uses smart case matching: if any of the patterns contains at least one capital letter, then matching is case-sensitive,
otherwise its case-insensitive. To pass command line arguments to Pkg.test(), the test_args keyword argument must be used:

$ alias jlt='julia --project -e "using Pkg; Pkg.test(test_args=ARGS)"'
$ jlt some tests
# ...runs test files which have "some" or "test" occurring in their names

Alternatively, one can filter tests by passing a filters keyword argument to the @run_tests macro, with a list of strings:

@run_tests filters=["some", "tests"]
9 Likes
2

Neat!
A couple of questions:

  • Can I simply add using TidyTest; @run_tests to an existing runtests.jl file with @testsets? This would lower the entry barrier.
  • Showing some progress is nice, to get a feeling of how testing goes! Did you also consider the “print a dot for each testcase” approach utilized my some Python libraries? To have a more immediately obvious visual indicator compared to changing digits.
3

Hi @aplavin!

I’m glad you like it!

The crux of the @run_tests macro is the discovery and inclusion of Julia source files in the test directory, so in order to convert an existing runtests.jl file, you need to:

  1. rename your existing runtests.jl to something else, preferably break it up along @testsets (or some by other organizing logic) into multiple files;
  2. add a new runtests.jl with using TidyTest; @run_tests as its only content.

Alternatively, if you prefer to keep all your tests in the main runtests.jl file, just wrap the contents into a toplevel @testset SpinnerTestSet begin ... end construct. This will take care of the reporting without using the auto discovery feature.

Perhaps I will add this as a migration guide to the README.

As for the other part of your question: I deliberately didn’t want to print a period for each completed test, because I don’t like the varying line lengths it causes. But there is a package that does just that: TestSetExtensions.jl. You should check that out, too. (This is in fact mentioned at the end of the README.)

1 Like