My unit tests are interferring with each other

New to Julia
1

I got a large module and to make sure everything works, I have created more than forty unit tests. But now they started to interfere with each other.

I have produce a smallest working example below to explain what my problem is.

File: runtests.jl

using Test

function tests()
    @testset "All tests" begin
        include("test_bar.jl")
        include("test_foo.jl")
    end
end

tests()

File: test_bar.jl

using Test

func1(x::Int64) = x^2

function bar(x,f)
    f(x)
end

@testset "bar function" begin

code = :( bar(2,func1) ); result = eval(code)
println("$code = ", result, " ", @test result == 4 )

end

File: test_foo.jl

using Test

func1(x) = x^3

function foo(x,f)
    f(x)
end

@testset "foo function" begin

code = :( foo(2,func1) ); result = eval(code)
println("$code = ", result, " ", @test result == 8 )

end

Individually each test works perfectly but when I execute “runtests.jl” this is the output

Output of runtests.jl

bar(2,func1) = 4 Test Passed
foo function: Test Failed at /Users/ssiew/juliascript/test_prob_dir/test_foo.jl:12
  Expression: result == 8
   Evaluated: 4 == 8
Stacktrace:
 [1] macro expansion at /Users/ssiew/juliascript/test_prob_dir/test_foo.jl:12 [inlined]
 [2] macro expansion at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
 [3] top-level scope at /Users/ssiew/juliascript/test_prob_dir/test_foo.jl:11
foo(2,func1) = 4 (Test Failed at /Users/ssiew/juliascript/test_prob_dir/test_foo.jl:12
  Expression: result == 8
   Evaluated: 4 == 8, Union{Ptr{Nothing}, InterpreterIP}[Ptr{Nothing} @0x000000011d86cac8, Ptr{Nothing} @0x000000010b148983, Ptr{Nothing} @0x000000011d869ad4, Ptr{Nothing} @0x000000011d86bf95, Ptr{Nothing} @0x000000010b148983, Ptr{Nothing} @0x000000010b179be2, Ptr{Nothing} @0x000000010b154d7f, Ptr{Nothing} @0x000000010b17a7d3, Ptr{Nothing} @0x000000010e275670, Ptr{Nothing} @0x000000010e2800bc, Ptr{Nothing} @0x000000011d85fd67, Ptr{Nothing} @0x000000011d85ed44, Ptr{Nothing} @0x000000010b148983, Ptr{Nothing} @0x000000010b30daa8, Ptr{Nothing} @0x000000010b30c59c, Ptr{Nothing} @0x000000010b30cae8, InterpreterIP(CodeInfo(
 1 ─ %1 = tests()                                                                                                                                                                                                        │
 └──      return %1                                                                                                                                                                                                      │
), 0x0000000000000000), Ptr{Nothing} @0x000000010b30cb40, Ptr{Nothing} @0x000000010b1796cd, Ptr{Nothing} @0x000000010b154d7f, Ptr{Nothing} @0x000000010e2aeabd, Ptr{Nothing} @0x000000011d85694f, Ptr{Nothing} @0x000000011d85d252, Ptr{Nothing} @0x000000010b148983, Ptr{Nothing} @0x000000011d85720f, Ptr{Nothing} @0x000000011d855c5c, Ptr{Nothing} @0x000000010b148983, Ptr{Nothing} @0x000000011d844aa7, Ptr{Nothing} @0x000000010b148983, Ptr{Nothing} @0x000000011d831d1f, Ptr{Nothing} @0x000000011d83244d, Ptr{Nothing} @0x000000010b159372, Ptr{Nothing} @0x000000011d82eff0, Ptr{Nothing} @0x000000011d82f075, Ptr{Nothing} @0x000000010b163b70])
Test Summary:  | Pass  Fail  Total
All tests      |    1     1      2
  bar function |    1            1
  foo function |          1      1
ERROR: LoadError: Some tests did not pass: 1 passed, 1 failed, 0 errored, 0 broken.
in expression starting at /Users/ssiew/juliascript/test_prob_dir/runtests.jl:10
julia>

What should I do to fix this?

2

You should reorganize your unit tests so that you don’t overwrite definitions. After some iterations writing unit tests, I came up with a very standard approach in all my packages. You can check this runtests.jl file for an example: GeoStatsDevTools.jl/runtests.jl at master · juliohm/GeoStatsDevTools.jl · GitHub

Basically you load all the packages once, and define common functions at a top level at runtests.jl and then include small test files for specific concepts, all wrapped in @testset.

2 Likes
3

Is there not a way that I could make “func1” local only to the unit test and not appear in another unit test? I tried the keyword local but it does not work.

I would prefer not to define func1 on the runtests.jl because I would like to see the definition of func1 in the unittest that it is used in.

4

include is the issue here. This library fixes it with @safetestset

3 Likes
5

Chris, How does running it inside a module using SafeTestsets.jl fixes the problem? func1 defined in unittest A would still be seen by unittest B

6

No, the included file is in a module, so it’s not added to Main, so the two func1s are different functions.

7

SafeTests seems interesting. I was not aware of it. As an alternative, I modeled my tests after the tests in Distributions.jl, which loops through each file and calls @eval module $(Symbol("Test_", t)) to ensure that each test is conducted in an isolated scope.

8

Christ Fisher

Yes that does looks interesting. But why is there a return keyword in it?

File: runtests.jl

using Test

alltests = ["bar","foo"]

function tests()
    @testset "All tests" begin   
        res = map(alltests) do testname
            @eval module $(Symbol("testmodule_", testname))
                include("test_" * $testname * ".jl")
            end
            return    # WHY IS THERE A RETURN KEYWORD HERE
        end
    end
end

tests()
9

I think it is equivalent to return nothing, which prevents the function from returning the most recent result.

1 Like