ANN: Jute, a Py.Test-inspired testing framework

package
announcement
testing

#1

Link for the impatient

I am a big fan of Py.Test in Python, and Base.Test does not really satisfy me in several respects. In particular, there is no test filtering, one cannot conveniently report non-boolean results from tests (e.g. benchmark values), and there is only basic support for fixtures. I am aware of PyTest.jl, but it has chosen the way of extending Base.Test, which means it shares its limitations (no automatic test collection, limited lifetime control for fixtures, tests are executed as the files are included, so no advanced control of that; more on that later).

So I put together a little proof-of-concept package, and my question is, is it interesting to anybody? If this is the case, any advice/feature requests/bug reports are welcome. It is my first Julia project, so it certainly will have a lot of stuff to fix.

Brief info

Basically, the it operates as follows. All the files satisfying a specific pattern in the test directory are picked up and included. All Testcase objects in the global scope are collected. Grouping is done by putting these objects in submodules.

A testcase is defined like:

tc = testcase() do
    @test 1 == 1
end

(Jute uses Base.Test assertions)

Testcases can be parametrized:

parameterized_testcase = testcase([1, 2], [3, 4]) do x, y
    @test x + y == y + x
end

Parametrization can be done by iterables or by special fixture objects. There are two types currently available, local ones (set up and destroyed right before and after the testcase), and global ones (set up and destroyed only once). For example:

db_connection = fixture(; delayed_teardown=true) do produce
    c = db_connect()

    # this call blocks until all the testcases
    # that use the fixture are executed
    produce([c])

    close(c)
end

db_testcase = testcase(db_connection) do c
    do_something_with(c)
end

More detailed info can be found in the project’s README (proper docs will appear soon).

Future possibilities

This concept, in principle, allows the following features to be implemented:

  • Automatic multi-process test run, with the fixtures initialized only in the processes where they are used
  • “Exclusive” fixtures, which guarantee that two values of that fixture do not exist in one process simultaneously, and the amount of setups and teardowns is minimal (can be used e.g. for some C library that requires setting a global state)
  • Watching test files and rerunning the changed testcases
  • Test tagging and filtering by tags
  • Allowing user to add their own command-line arguments

Current problems

  • Test pick up and execution is a bit slow at the moment, because I load test files dynamically (by evaling includes), and, consequently, have to use invokelatest(). This can be avoided by starting a child process that includes all the required files statically.
  • Code is messy and there are no docs. Also, tests do not cover much. This, of course, will be fixed.

#2

High five :slight_smile: !

This is nice! It’s awesome to see these py.test inspired tools pop up. I need to dig into Jute.jl, and understand the differences to PyTest.jl. I haven’t devoted much time to PyTest.jl recently (none, actually).

Agreed, I tried to mimic py.test first, and Base.Test second, as a principle, but I haven’t really given it much consideration. The choice of that principle was rather pragmatic, so I didn’t really consider the limitations.

Are you using Jute.jl to test something? That’s something I very much missed, since Julia is my side-project-language, not something I use on a daily basis.


#3

Thanks!

The thing is, a while ago I was working on a Python-based GPGPU project, and Py.Test’s features (testcase filtering, custom output, custom preparation/teardown of fixtures) were extremely helpful for testing that. Recently I started to work on a technologically similar project (something like a stochastic simulation toolbox; see here, but it is not even in the alpha stage, I’m still figuring out the architecture), and instantly realized that I will be lacking all this stuff in Base.Test. PyTest.jl was close to what I needed, but since it is based on Base.Test, it eventually wouldn’t be able to support some of the required features too. So I decided to see what I can do myself.


#4

This looks like a nice project. We’d ultimately like the built-in testing framework to be sufficient for most testing purposes, but recognize that it’s far from feature-complete. Writing a package for more featureful testing is a great way to try out new ideas, but as you work on it, keep in mind how some of the features might make their way into Base.Test in the future! Perhaps at some point when you feel like you’ve got things really nicely worked out we can have a discussion about what can/should be moved into Base and what should stay in packages like this one for people who like certain styles of testing.