Nesting @testset; good idea, or not?

General Usage
1

I see two common patterns when it comes to testing larger packages.

Coarse Style

runtests.jl (Coarse Style)

(this is often done with a loop over filenames but I’ll write it here in full)

using Base.Test
@testset "compA" begin
    include(compA.jl)
end

@testset "compB" begin
    include(compB.jl)
end

compA.jl and similarly compB.jl (Course Style)

using Base.Test #reuse this in the included file so can run standalone

# Tests for foo
@test foo(1)==7
@test foo(2)==9
@test foo(3)==16

# Tests for bar
@test bar(1)==8
@test bar(2)==-8

##Fine Style

runtests.jl (FineStyle)

(this is also often done with a loop over filenames but I’ll write it here in full)

using Base.Test
include(compA.jl)
include(compB.jl)

compA.jl and similarly compB.jl (Fine Style)

using Base.Test #reuse this in the included file so can run standalone
@testset "foo" begin
    @test foo(1)==7 
    @test foo(2)==9
    @test foo(3)==16
end

@testset "bar" begin
    @test bar(1)==8
    @test bar(2)==-8
end

Are there any pros or cons to combining them?
That is to say taking runtest.jl from the coarse style, and compA.jl and compB.jl from the fine style.
I know it is legal.
As I recall what happens when you do julia runtest.jl, is that you only see the records for the outer most layer – that is the coarse runtests.jl.
But it fully includes the total counts from the lower levels.

Seems like it is purely, since now if running a subset of the tests eg julia compA.jl, we would also get a nice breakdown table; at the fine grained level.

But I’ve never seen this done in packages.
So perhaps I am wrong and it doesn’t work so good

2 Likes
2

I use the course style because I just want to know which test failed, and the line number.

This setup gives me all of the information that I need, is easy to extend, and separates tests of different features well.

What I like about it too is that I can keep all of my test files as examples, and just develop through tests. For example

https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/master/test/ode/ode_event_tests.jl

This looks like any other script, but with a few @tests around. But if you run it, it’ll just work. And you can see I commented out common plot commands I use. I develop it interactively in that file, and then add the test when it’s complete.

If I wanted to do the “Fine style”, then I would have to gather the tests, or put a big testset around it. Juno will by default be funny with the big block, and inline execution will try to run the whole block every time, so that’s a no-go. Secondly, I’d have to group a lot of my terms if I wanted to use the same names. So just making the whole file “normal” is just easier for interactivity I think.

2 Likes
3

I definitely nest my testsets, Generally a top-level @testset "MyPkg Tests" begin ... end that just includes all the test files, then each test file includes a top-level @testset, then each test case within that has its own @testset. Example here.

I generally think of my inner-most @testsets as test cases that generally only have a couple @test assertions within them. That way I can name what each test is doing pretty granularly. If everything passes it only displays the top-level statistics, but if there are any failures within then it should show the whole testset tree down to the one that failed. The indentation was messed up for a while so it was a little harder to see the nesting, but that’s been fixed recently.

4 Likes
4

I’ve lately tried the

@testset "foo" begin
    include("test.jl")
end

style, and found unfortunately that line number information seems to be destroyed by this style. Has anyone else found similar behaviour? What workarounds exist?

2 Likes