# ANN: Jute, a Py.Test-inspired testing framework

**URL:** <https://discourse.julialang.org/t/ann-jute-a-py-test-inspired-testing-framework/5028>\
**Category:** Community\
**Tags:** package, announcement, testing\
**Created:** [July 23, 2017, 9:11am UTC](https://discourse.julialang.org/t/ann-jute-a-py-test-inspired-testing-framework/5028 "2017-07-23T09:11:31Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![fjarri](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fjarri/32/630_2.png) [@fjarri](https://discourse.julialang.org/u/fjarri)\
**Post date:** [July 23, 2017, 9:11am UTC](https://discourse.julialang.org/t/ann-jute-a-py-test-inspired-testing-framework/5028/1 "2017-07-23T09:11:31Z")

</div>

[Link for the impatient](https://github.com/fjarri/Jute.jl)

I am a big fan of `Py.Test` in Python, and `Base.Test` does not really satisfy me in several respects. In particular, there is no test filtering, one cannot conveniently report non-boolean results from tests (e.g. benchmark values), and there is only basic support for fixtures. I am aware of [`PyTest.jl`](https://github.com/pdobacz/PyTest.jl), but it has chosen the way of extending `Base.Test`, which means it shares its limitations (no automatic test collection, limited lifetime control for fixtures, tests are executed as the files are included, so no advanced control of that; more on that later).

So I put together a little proof-of-concept package, and my question is, is it interesting to anybody? If this is the case, any advice/feature requests/bug reports are welcome. It is my first Julia project, so it certainly will have a lot of stuff to fix.

### Brief info

Basically, the it operates as follows. All the files satisfying a specific pattern in the `test` directory are picked up and included. All `Testcase` objects in the global scope are collected. Grouping is done by putting these objects in submodules.

A testcase is defined like:

```julia
tc = testcase() do
    @test 1 == 1
end

```

(`Jute` uses `Base.Test` assertions)

Testcases can be parametrized:

```julia
parameterized_testcase = testcase([1, 2], [3, 4]) do x, y
    @test x + y == y + x
end

```

Parametrization can be done by iterables or by special fixture objects. There are two types currently available, local ones (set up and destroyed right before and after the testcase), and global ones (set up and destroyed only once). For example:

```julia
db_connection = fixture(; delayed_teardown=true) do produce
    c = db_connect()

    # this call blocks until all the testcases
    # that use the fixture are executed
    produce([c])

    close(c)
end

db_testcase = testcase(db_connection) do c
    do_something_with(c)
end

```

More detailed info can be found in the project’s `README` (proper docs will appear soon).

### Future possibilities

This concept, in principle, allows the following features to be implemented:

- Automatic multi-process test run, with the fixtures initialized only in the processes where they are used
- “Exclusive” fixtures, which guarantee that two values of that fixture do not exist in one process simultaneously, and the amount of setups and teardowns is minimal (can be used e.g. for some C library that requires setting a global state)
- Watching test files and rerunning the changed testcases
- Test tagging and filtering by tags
- Allowing user to add their own command-line arguments

### Current problems

- Test pick up and execution is a bit slow at the moment, because I load test files dynamically (by `eval`ing `include`s), and, consequently, have to use `invokelatest()`. This can be avoided by starting a child process that includes all the required files statically.
- Code is messy and there are no docs. Also, tests do not cover much. This, of course, will be fixed.

---

<div class="post-metadata">

**Author:** ![pdobacz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdobacz/32/1775_2.png) [@pdobacz](https://discourse.julialang.org/u/pdobacz)\
**Post date:** [November 3, 2017, 10:29pm UTC](https://discourse.julialang.org/t/ann-jute-a-py-test-inspired-testing-framework/5028/2 "2017-11-03T22:29:16Z")

</div>

High five 🙂 !

This is nice! It’s awesome to see these py.test inspired tools pop up. I need to dig into Jute.jl, and understand the differences to PyTest.jl. I haven’t devoted much time to PyTest.jl recently (none, actually).

Agreed, I tried to mimic py.test first, and Base.Test second, as a principle, but I haven’t really given it much consideration. The choice of that principle was rather pragmatic, so I didn’t really consider the limitations.

Are you using Jute.jl to test something? That’s something I very much missed, since Julia is my side-project-language, not something I use on a daily basis.

---

<div class="post-metadata">

**Author:** ![fjarri](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fjarri/32/630_2.png) [@fjarri](https://discourse.julialang.org/u/fjarri)\
**Post date:** [November 3, 2017, 10:52pm UTC](https://discourse.julialang.org/t/ann-jute-a-py-test-inspired-testing-framework/5028/3 "2017-11-03T22:52:10Z")

</div>

Thanks!

The thing is, a while ago I was working on a Python-based GPGPU project, and Py.Test’s features (testcase filtering, custom output, custom preparation/teardown of fixtures) were extremely helpful for testing that. Recently I started to work on a technologically similar project (something like a stochastic simulation toolbox; see [here](https://github.com/fluxion/Fluxion.jl), but it is not even in the alpha stage, I’m still figuring out the architecture), and instantly realized that I will be lacking all this stuff in `Base.Test`. `PyTest.jl` was close to what I needed, but since it is based on `Base.Test`, it eventually wouldn’t be able to support some of the required features too. So I decided to see what I can do myself.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [November 13, 2017, 9:42pm UTC](https://discourse.julialang.org/t/ann-jute-a-py-test-inspired-testing-framework/5028/4 "2017-11-13T21:42:04Z")

</div>

This looks like a nice project. We’d ultimately like the built-in testing framework to be sufficient for most testing purposes, but recognize that it’s far from feature-complete. Writing a package for more featureful testing is a great way to try out new ideas, but as you work on it, keep in mind how some of the features might make their way into Base.Test in the future! Perhaps at some point when you feel like you’ve got things really nicely worked out we can have a discussion about what can/should be moved into Base and what should stay in packages like this one for people who like certain styles of testing.

---

<div class="post-metadata">

**Author:** ![Seanny123](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/seanny123/32/5271_2.png) [@Seanny123](https://discourse.julialang.org/u/Seanny123)\
**Post date:** [November 21, 2018, 5:22pm UTC](https://discourse.julialang.org/t/ann-jute-a-py-test-inspired-testing-framework/5028/5 "2018-11-21T17:22:31Z")

</div>

This is amazing and exactly what I was looking for! I’m also overjoyed to see you’ve kept maintaining it and hope I can contribute meaningfully in the near future.

---

<div class="post-metadata">

**Author:** ![fjarri](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fjarri/32/630_2.png) [@fjarri](https://discourse.julialang.org/u/fjarri)\
**Post date:** [November 21, 2018, 11:17pm UTC](https://discourse.julialang.org/t/ann-jute-a-py-test-inspired-testing-framework/5028/6 "2018-11-21T23:17:31Z")

</div>

Thanks, I am using it in my own projects, so it’ll be alive for some time at least 🙂 It’s especially good to have an outsider perspective on the API, so if you see any rough edges, don’t hesitate to file an issue/write to me directly.

In particular, I’m now thinking about the fate of automatic test file pick-up (as opposed to explicit `include`) — initially I added this feature because that’s how I use `py.test`, but over time I cooled off to it, since with Julia’s small package philosophy the test hierarchy tends to be flat anyway. Plus, it makes stack traces more complicated because I have to use `invokelatest()` to call testcases.