I have a few questions about unit testing workflows. Also, maybe we can use this thread to compile a list of best practices for unit testing which we can then contribute to the docs.
1 - where is runtests.jl supposed to be run? When I run the tests, I load the environment of the project being tested. Also, sometimes I need to access files and folders relative to the current execution directory. I keep alternating between running from the root of the project and from the test/ folder. Which is the right way?
P.S. - here it will be ideal to just be able to run pkg> test as already discussed elsewhere.
2 - I’ve seen that there are test dependencies in Project.toml. Is this feature documented and fully supported? I haven’t found the documentation although I’ve seen this used in the wild. Pkg doesn’t seem to support it out of the box - for instance, there’s no way to add an extra. What’s the workflow? Add it as a dependency and move it by hand into extras and add the test target? Will that be maintained by Pkg (updates, version bounds, etc).
3 - more towards CI, can anybody advise on testing against real databases (SQLite, MySQL, Postgres). Some sort of CI setup where you test without exposing the credentials – or some sort of DB testing services? Does anybody do this for open source projects?
In regards to best practices, I would like to mention TestSetExtensions.jl (thanks @kristoffer.carlsson).
If you use TestSetExtensions.jl for the usability, you can also break your tests into smaller files addressing one area of the tests and use the @includetests macro to load them, ie:
@testset ExtendedTestSet "Genie tests" begin
@includetests
end
you should run from tests/ if running pieces of test code “manually”, otherwise relative paths will not resolve correctly (but if you don’t have any, eg includes, then it is not relevant)
Thanks! Oh wow, the test/Project.toml is pretty cool, I wasn’t aware of that. It’s nice to see that Pkg is smart about it, like automatically including the package itself.
I set it up quickly and works great:
I see now that the extras and targets are for Julia 1.0 and 1.1 – while the Project.toml is >= 1.2. That makes sense.
This is great info, thanks for sharing. It would be nice to have the Pkg docs into Julia docs. Pkg is such an integral part of Julia that I rarely think about reading its docs separately. Plus Julia docs have a section on testing…
I think one of the best practices is to use SafeTestsets.jl
Since otherwise your testsets can leak, i.e. you can have tests which require being run in order, and if you run them on their own they can fail. Safetestsets puts everything in specific modules, so even if you do things like include, the function definitions of one test do not effect another.
Once you have this, you can make every single test script be an independently runnable file, a fully reproducible script for a single idea. Then you can just include the script:
If you do it like this, then every test script can be used independently, which is really nice for development. For example, if this test fails:
you can just pick up this script and start hacking away at it. There’s no dependency on some function or data defined elsewhere (something I particularly dislike when PRing to some older packages which don’t disentangle their tests…)
There are obviously some downsides to this. For example, maybe you want to use the same function in a few places, and this approach would have you define independent versions, which means it would compile each time. However, I think the advantage of having tests that actually run what you think they run far outweighs this.
I cannot even fully describe the number of times I have found tests in unsafe setups which only pass because of a random function definition that in a different test file that nobody knew about, so on CI the test was actually testing nothing while locally if you run the script in isolation you think it’s a good test.
This is great, thanks for sharing! I fully agree. I recently stumbled into an issue where the first test was setting some state which was then picked by the next test, so the 2nd test wasn’t actually doing its thing. Testing in isolation is extremely important. (Now if only I could make TestSetExtensions and SafeTestsets work together).
there are many test files in Octo.jl to testing for the databases.
To run that test files in automatically, I used the Jive.jl package.
see the Travis job log.
Travis and GitHub Actions, and I guess many other CI services, let you set up DBs in the test environment itself, so that you don’t have to host your own. Their docs have examples on how to get it running: Travis, GitHub
That’s great, thank you! I think Travis makes more sense - plus the documentation looks good and it seems they provide a lot more configuration options (also in terms of RDBMS versions).
What would be really great would be a composable test framework, similar to Logging, where one can combine and nest various testing packages. For instance, TestSetExtensions and SafeTestsets don’t seem to work together (at least not based on my early experiments).
You’re right, but they can be made to work alongside one another if TestSetExtensions is used in the outer level, and SafeTestsets only in the inner testsets:
@testset ExtendedTestSet "Example" begin
@safetestset "Feature 1" begin
a = [1, 2, 3]
b = [1, 2, 4]
# Will display nicely thanks to `ExtendedTestSet`
@test a == b
end
@safetestset "Feature 2" begin
# Will fail thanks to `@safetestset`
@test a == [1, 2, 3]
end
end
@ChrisRackauckas I’ve organized the tests in my package so they can be run as independently runnable files, but am trying to figure out where to place code that is shared across different files.
In my case, I’ve just placed everything into a module which I import via relative paths. Are there better alternatives?
+ 
