# Best practices for Julia unit testing

**URL:** <https://discourse.julialang.org/t/best-practices-for-julia-unit-testing/30858>\
**Category:** General Usage\
**Created:** [November 8, 2019, 12:44pm UTC](https://discourse.julialang.org/t/best-practices-for-julia-unit-testing/30858 "2019-11-08T12:44:13Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![essenciary](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/essenciary/32/210469_2.png) [@essenciary](https://discourse.julialang.org/u/essenciary)\
**Post date:** [November 8, 2019, 12:44pm UTC](https://discourse.julialang.org/t/best-practices-for-julia-unit-testing/30858/1 "2019-11-08T12:44:13Z")

</div>

I have a few questions about unit testing workflows. Also, maybe we can use this thread to compile a list of best practices for unit testing which we can then contribute to the docs.

1 - where is `runtests.jl` supposed to be run? When I run the tests, I load the environment of the project being tested. Also, sometimes I need to access files and folders relative to the current execution directory. I keep alternating between running from the root of the project and from the `test/` folder. Which is the right way?

P.S. - here it will be ideal to just be able to run `pkg> test` as already discussed elsewhere.

2 - I’ve seen that there are test dependencies in `Project.toml`. Is this feature documented and fully supported? I haven’t found the documentation although I’ve seen this used in the wild. Pkg doesn’t seem to support it out of the box - for instance, there’s no way to add an `extra`. What’s the workflow? Add it as a dependency and move it by hand into `extras` and add the test target? Will that be maintained by Pkg (updates, version bounds, etc).

3 - more towards CI, can anybody advise on testing against real databases (SQLite, MySQL, Postgres). Some sort of CI setup where you test without exposing the credentials – or some sort of DB testing services? Does anybody do this for open source projects?

* * *

In regards to best practices, I would like to mention `TestSetExtensions.jl` (thanks @kristoffer.carlsson).

If you use `TestSetExtensions.jl` for the usability, you can also break your tests into smaller files addressing one area of the tests and use the `@includetests` macro to load them, ie:

```julia
@testset ExtendedTestSet "Genie tests" begin
  @includetests
end

```

Currently broken on v1 but you can use my fork until support for Julia v1 is merged:  
[https://github.com/essenciary/TestSetExtensions.jl/tree/julia\_v1](https://github.com/essenciary/TestSetExtensions.jl/tree/julia_v1)

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 8, 2019, 12:47pm UTC](https://discourse.julialang.org/t/best-practices-for-julia-unit-testing/30858/2 "2019-11-08T12:47:32Z")

</div>

1. you should run from `tests/` if running pieces of test code “manually”, otherwise relative paths will not resolve correctly (but if you don’t have any, eg `include`s, then it is not relevant)

2. see [5. Creating Packages · Pkg.jl](https://julialang.github.io/Pkg.jl/dev/creating-packages/#Adding-tests-to-the-package-1)

---

<div class="post-metadata">

**Author:** ![essenciary](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/essenciary/32/210469_2.png) [@essenciary](https://discourse.julialang.org/u/essenciary)\
**Post date:** [November 8, 2019, 1:02pm UTC](https://discourse.julialang.org/t/best-practices-for-julia-unit-testing/30858/3 "2019-11-08T13:02:35Z")

</div>

Thanks! Oh wow, the `test/Project.toml` is pretty cool, I wasn’t aware of that. It’s nice to see that Pkg is smart about it, like automatically including the package itself.

I set it up quickly and works great:  
 ![image](https://global.discourse-cdn.com/julialang/original/3X/f/9/f96894a94a4f2462b1e6ee127246d9ddd073d11a.png)

I see now that the `extras` and `targets` are for Julia 1.0 and 1.1 – while the `Project.toml` is \>= 1.2. That makes sense.

This is great info, thanks for sharing. It would be nice to have the Pkg docs into Julia docs. Pkg is such an integral part of Julia that I rarely think about reading its docs separately. Plus Julia docs have a section on testing…

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [November 8, 2019, 1:07pm UTC](https://discourse.julialang.org/t/best-practices-for-julia-unit-testing/30858/4 "2019-11-08T13:07:23Z")

</div>

I think one of the best practices is to use SafeTestsets.jl

> **[GitHub - YingboMa/SafeTestsets.jl](https://github.com/YingboMa/SafeTestsets.jl)**
>
> Contribute to YingboMa/SafeTestsets.jl development by creating an account on GitHub.

Since otherwise your testsets can leak, i.e. you can have tests which require being run in order, and if you run them on their own they can fail. Safetestsets puts everything in specific modules, so even if you do things like `include`, the function definitions of one test do not effect another.

Once you have this, you can make every single test script be an independently runnable file, a fully reproducible script for a single idea. Then you can just include the script:

> <https://github.com/SciML/OrdinaryDiffEq.jl/blob/master/test/runtests.jl#L12-L19>

If you do it like this, then every test script can be used independently, which is really nice for development. For example, if this test fails:

> <https://github.com/SciML/OrdinaryDiffEq.jl/blob/master/test/integrators/check_error.jl>

you can just pick up this script and start hacking away at it. There’s no dependency on some function or data defined elsewhere (something I particularly dislike when PRing to some older packages which don’t disentangle their tests…)

There are obviously some downsides to this. For example, maybe you want to use the same function in a few places, and this approach would have you define independent versions, which means it would compile each time. However, I think the advantage of having tests that actually run what you think they run far outweighs this.

I cannot even fully describe the number of times I have found tests in unsafe setups which only pass because of a random function definition that in a different test file that nobody knew about, so on CI the test was actually testing nothing while locally if you run the script in isolation you think it’s a good test.

---

<div class="post-metadata">

**Author:** ![essenciary](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/essenciary/32/210469_2.png) [@essenciary](https://discourse.julialang.org/u/essenciary)\
**Post date:** [November 8, 2019, 1:54pm UTC](https://discourse.julialang.org/t/best-practices-for-julia-unit-testing/30858/5 "2019-11-08T13:54:51Z")

</div>

This is great, thanks for sharing! I fully agree. I recently stumbled into an issue where the first test was setting some state which was then picked by the next test, so the 2nd test wasn’t actually doing its thing. Testing in isolation is extremely important. (Now if only I could make TestSetExtensions and SafeTestsets work together).

---

<div class="post-metadata">

**Author:** ![e3c6](https://avatars.discourse-cdn.com/v4/letter/e/e79b87/32.png) [@e3c6](https://discourse.julialang.org/u/e3c6)\
**Post date:** [November 8, 2019, 8:05pm UTC](https://discourse.julialang.org/t/best-practices-for-julia-unit-testing/30858/6 "2019-11-08T20:05:14Z")

</div>

I would like it if `SafeTestsets` were integrated into the `Tests` stdlib.

---

<div class="post-metadata">

**Author:** ![wookyoung](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wookyoung/32/157_2.png) [@wookyoung](https://discourse.julialang.org/u/wookyoung)\
**Post date:** [November 9, 2019, 10:14am UTC](https://discourse.julialang.org/t/best-practices-for-julia-unit-testing/30858/7 "2019-11-09T10:14:11Z")

</div>

there are many [test files in Octo.jl](https://github.com/wookay/Octo.jl/tree/master/test/adapters) to testing for the databases.  
To run that test files in automatically, I used the Jive.jl package.  
see the [Travis job log](https://travis-ci.org/wookay/Octo.jl/jobs/609590389).

---

<div class="post-metadata">

**Author:** ![SaschaMann](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/saschamann/32/2852_2.png) [@SaschaMann](https://discourse.julialang.org/u/SaschaMann)\
**Post date:** [November 9, 2019, 11:33am UTC](https://discourse.julialang.org/t/best-practices-for-julia-unit-testing/30858/8 "2019-11-09T11:33:32Z")

</div>

> [@essenciary](#):
>
> 3 - more towards CI, can anybody advise on testing against real databases (SQLite, MySQL, Postgres). Some sort of CI setup where you test without exposing the credentials – or some sort of DB testing services? Does anybody do this for open source projects?

Travis and GitHub Actions, and I guess many other CI services, let you set up DBs in the test environment itself, so that you don’t have to host your own. Their docs have examples on how to get it running: [Travis](https://docs.travis-ci.com/user/database-setup), [GitHub](https://github.com/actions/example-services)

---

<div class="post-metadata">

**Author:** ![essenciary](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/essenciary/32/210469_2.png) [@essenciary](https://discourse.julialang.org/u/essenciary)\
**Post date:** [November 9, 2019, 12:49pm UTC](https://discourse.julialang.org/t/best-practices-for-julia-unit-testing/30858/9 "2019-11-09T12:49:06Z")

</div>

The Travis job is very helpful, I’ll try to copy some of that.  
P.S. - would be nice to look into an Octo plugin for Genie. 🐙 + 🧞

---

<div class="post-metadata">

**Author:** ![essenciary](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/essenciary/32/210469_2.png) [@essenciary](https://discourse.julialang.org/u/essenciary)\
**Post date:** [November 9, 2019, 12:55pm UTC](https://discourse.julialang.org/t/best-practices-for-julia-unit-testing/30858/10 "2019-11-09T12:55:38Z")

</div>

That’s great, thank you! I think Travis makes more sense - plus the documentation looks good and it seems they provide a lot more configuration options (also in terms of RDBMS versions).

---

<div class="post-metadata">

**Author:** ![essenciary](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/essenciary/32/210469_2.png) [@essenciary](https://discourse.julialang.org/u/essenciary)\
**Post date:** [November 13, 2019, 6:24pm UTC](https://discourse.julialang.org/t/best-practices-for-julia-unit-testing/30858/11 "2019-11-13T18:24:09Z")

</div>

What would be really great would be a composable test framework, similar to `Logging`, where one can combine and nest various testing packages. For instance, `TestSetExtensions` and `SafeTestsets` don’t seem to work together (at least not based on my early experiments).

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [November 13, 2019, 6:45pm UTC](https://discourse.julialang.org/t/best-practices-for-julia-unit-testing/30858/12 "2019-11-13T18:45:03Z")

</div>

> [@essenciary](#):
>
> For instance, `TestSetExtensions` and `SafeTestsets` don’t seem to work together (at least not based on my early experiments).

You’re right, but they can be made to work alongside one another if `TestSetExtensions` is used in the outer level, and `SafeTestsets` only in the inner testsets:

```julia

@testset ExtendedTestSet "Example" begin
    @safetestset "Feature 1" begin
        a = [1, 2, 3]
        b = [1, 2, 4]

        # Will display nicely thanks to `ExtendedTestSet`
        @test a == b
    end

    @safetestset "Feature 2" begin
        # Will fail thanks to `@safetestset`
        @test a == [1, 2, 3]
    end
end

```

---

<div class="post-metadata">

**Author:** ![essenciary](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/essenciary/32/210469_2.png) [@essenciary](https://discourse.julialang.org/u/essenciary)\
**Post date:** [November 13, 2019, 6:53pm UTC](https://discourse.julialang.org/t/best-practices-for-julia-unit-testing/30858/13 "2019-11-13T18:53:45Z")

</div>

Oh, nice find, thank you! I’ll give it a try! 🤩

---

<div class="post-metadata">

**Author:** ![essenciary](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/essenciary/32/210469_2.png) [@essenciary](https://discourse.julialang.org/u/essenciary)\
**Post date:** [November 13, 2019, 7:01pm UTC](https://discourse.julialang.org/t/best-practices-for-julia-unit-testing/30858/14 "2019-11-13T19:01:54Z")

</div>

Works like a charm, thanks so much!

A bit of a pain now to repeat module imports for each testset, but it’s an acceptable trade-off. 😍

---

<div class="post-metadata">

**Author:** ![vtjeng](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vtjeng/32/3787_2.png) [@vtjeng](https://discourse.julialang.org/u/vtjeng)\
**Post date:** [September 29, 2020, 4:38am UTC](https://discourse.julialang.org/t/best-practices-for-julia-unit-testing/30858/15 "2020-09-29T04:38:16Z")

</div>

@ChrisRackauckas I’ve organized the tests in my package so they can be run as independently runnable files, but am trying to figure out where to place code that is shared across different files.

In my case, I’ve just placed everything into a [module](https://github.com/vtjeng/MIPVerify.jl/blob/master/test/TestHelpers.jl) which I import via [relative paths](https://github.com/vtjeng/MIPVerify.jl/blob/49cd9c7379f5f21c538e266248aa9c070f0e2ed4/test/batch_processing_helpers/integration.jl#L4). Are there better alternatives?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [September 29, 2020, 1:08pm UTC](https://discourse.julialang.org/t/best-practices-for-julia-unit-testing/30858/16 "2020-09-29T13:08:32Z")

</div>

That’s a way to do it.