TidyTest.jl -- test execution with auto discovery and a neater test summary

Package Announcements
1

I’m happy to announce the first release of TidyTest.jl, a package for test execution with auto discovery and a neater test summary. It is using ProgressMeter.jl to report progress as the tests are completed.

Usage

For the simplest use case, write the following lines in your runtests.jl file:

using TidyTest

@run_tests

Add TidyTest.jl to the dependencies of your test:

julia> using Pkg; pkg"activate test; add TidyTest"

And then execute your tests:

$ julia --project -e "using Pkg; Pkg.test()"

For example:

The @run_tests macro automatically discovers all Julia source files in the directory of runtests.jl, and includes all of them. The entire block is wrapped in a single toplevel @testset using the custom test set type SpinnerTestSet. Test progress is reported using ProgressMeter.jl, continuously updating the status as tests are completed. If some tests fail (or throw an error), the issues are reported as they happen, and a detailed test summary is printed upon completion, using the default test reporting.

Test filtering

The macro facilitates running tests selectively. Every command line argument is used as a pattern, and only those tests are run which contain any of the arguments as a substring. The search uses smart case matching: if any of the patterns contains at least one capital letter, then matching is case-sensitive,
otherwise its case-insensitive. To pass command line arguments to Pkg.test(), the test_args keyword argument must be used:

$ alias jlt='julia --project -e "using Pkg; Pkg.test(test_args=ARGS)"'
$ jlt some tests
# ...runs test files which have "some" or "test" occurring in their names

Alternatively, one can filter tests by passing a filters keyword argument to the @run_tests macro, with a list of strings:

@run_tests filters=["some", "tests"]
14 Likes
2

Neat!
A couple of questions:

  • Can I simply add using TidyTest; @run_tests to an existing runtests.jl file with @testsets? This would lower the entry barrier.
  • Showing some progress is nice, to get a feeling of how testing goes! Did you also consider the “print a dot for each testcase” approach utilized my some Python libraries? To have a more immediately obvious visual indicator compared to changing digits.
1 Like
3

Hi @aplavin!

I’m glad you like it!

The crux of the @run_tests macro is the discovery and inclusion of Julia source files in the test directory, so in order to convert an existing runtests.jl file, you need to:

  1. rename your existing runtests.jl to something else, preferably break it up along @testsets (or some by other organizing logic) into multiple files;
  2. add a new runtests.jl with using TidyTest; @run_tests as its only content.

Alternatively, if you prefer to keep all your tests in the main runtests.jl file, just wrap the contents into a toplevel @testset SpinnerTestSet begin ... end construct. This will take care of the reporting without using the auto discovery feature.

Perhaps I will add this as a migration guide to the README.

As for the other part of your question: I deliberately didn’t want to print a period for each completed test, because I don’t like the varying line lengths it causes. But there is a package that does just that: TestSetExtensions.jl. You should check that out, too. (This is in fact mentioned at the end of the README.)

2 Likes
5

@g-gundam, have you managed to figure out how to use @run_tests from the REPL? The trick is to run it from the test directory.

julia> using TidyTest
julia> ;<enter>
shell> cd test
shell> <backspace>
julia> @run_tests;

It should work without problems. HTH.

2 Likes
6

@HanD - Thank you for getting back to me even though I deleted my previous post. Your solution is much better that what I was doing which was calling Pkg.test() with various parameters (which wasn’t terrible but not ideal). However, changing into the “test” directory and running @run_tests is way faster and exactly what I was looking for.

julia> cd("test")

julia> @run_tests verbose=true;
Test Summary:   | Pass  Total  Time
ReversedSeries  |    4      4  0.0s
  ReversedFrame |    3      3  0.0s
  Reversed      |    1      1  0.0s

julia> @run_tests verbose=true filters=["frame"];
Test Summary:   | Pass  Total  Time
ReversedSeries  |    3      3  0.0s
  ReversedFrame |    3      3  0.0s

That’s what I wanted.

You should definitely mention this tip in your README. TidyTest.jl was already a really nice way to run specific testsets instead of all of them, and this tip makes it even nicer to use.

PS: I’m not proficient enough in Julia to understand why this works. Why does changing into the “test” directory make it print more things?

1 Like
7

The explanation is simple: the @run_tests macro looks for files to include and run as tests in the working directory. When you run tests via the Pkg.test() command, the working directory is the test directory. When you are in the REPL, the working directory is (at least initially) the main package directory.

1 Like
8

See GitHub - dhanak/TidyTest.jl: An `AbstractTestSet` implementation and a helper macro for test execution with auto discovery and a neater test summary.

1 Like
9

When I first saw the title, I assumed this was related to Tidier.jl.

You might want to consider a different name. I think Tidier.jl predates this and it also is related to R’s tidyverse.

On the other hand “tidy” is a common enough word. And I don’t think it’s a big problem if people make the wrong assumption now and then. So leaving it as is, is ok. I just wanted to make you aware.

Also, the package looks great!

3 Likes
10

Very nice, and almost perfect. It’s too bad that GitHub doesn’t support julia-repl for highlighting like this forum does. GitHub accepts it, but it doesn’t color the prompt differently.

Using julia

julia> "Notice the prompt is not highlighted."

Using julia-repl

julia> "The prompt is colored now."
11

I’m happy enough with it to migrate my other projects to TidyTest.jl. I honestly wrote fewer tests, because I didn’t like the stock testing infrastructure of Julia. I didn’t want to run every testset every time. I looked around for alternatives and found them a bit confusing, granted it was earlier in my Julia journey.

Fast forward many months, and I’m not sure how I came across TidyTest.jl, but the setup looked simple. I tried it out, and the setup was indeed simple, and it did everything it said it would do.

I’m happy I found it, and if anyone ever asks, I’ll recommend it to others, too.

2 Likes
12

Well, I’m really glad to hear that!

1 Like
13

@HanD - If I wanted to provide helper functions for my tests, where is a good place to put them?

14

You can either put such functions directly in runtests.jl, before calling the @run_tests macro, or in a subdirectory of test, i.e., test/lib/helpers.jl. This will work because the auto-discovery of @run_tests won’t pick up files from subdirectories.

2 Likes