Scope and structure for unit test files

New to Julia
1

I have a query about how to structure the contents of a unit test file.

In many cases, ideally, for each test we will have an independent scope. This is to prevent information leakage between different test cases.

The documentation suggests that the @testset macro is used to define a scope.

However, it also suggests the purpose of this macro serves a different purpose - to group tests into logical groups.

The @testset macro can be used to group tests into sets . All the tests in a test set will be run, and at the end of the test set a summary will be printed. If any of the tests failed, or could not be evaluated due to an error, the test set will then throw a TestSetException .

Here’s an example of how a typical runTests.jl file might be structured.

# runTests.jl

include("testComponentA.jl")
include("testComponentB.jl")

To run it, I use a short script, which lives in the same directory.

#!/usr/bin/bash
julia --project -e 'using Pkg; Pkg.test()'

Here’s how each of the individual test files might be structured. I didn’t make these files into modules by including module ... end. Possibly I should have.

Each component of the source code, for example a module which performs some self-contained function, has a corresponding file in the test directory containing its tests.

# testComponentA.jl
# this is not a module, maybe it should be

include("../src/ComponentA.jl")

using Main.ComponentAModule

using Test

@testset "ComponentAModuleTests" begin

    # this is a single unit test
    component = ComponentA()
    @test component.someFunction() == someValue

    # this is an independent unit test from the above
    # should it be part of the same test set?
    component2 = ComponentA()
    @test component2.someOtherFunction() == someOtherValue

    # information leakage: component != component2
    # easy to use `component` instead of `component2`
    # and thus make a mistake in test code

end

My question here being, is this the right use of @testset or should @testset be used to create and destroy scope for individual unit tests. (To prevent information leakage.)

Related to the above, is it possible to create a new scope using a let statement? Something like

let
    # put a single test here
end

I’m not sure if that is valid syntax, because the let is not associated with any value. When I tried it, it seemed to be valid however.

Perhaps a combination of both should be used?

2

Have you tried to look at GitHub - YingboMa/SafeTestsets.jl
This package might solve your concern about leaking information. DifferentialEquations.jl uses the package to separate the tests into several files that can be run independent. DifferentialEquations.jl/test/runtests.jl at master · SciML/DifferentialEquations.jl · GitHub

1 Like
3

First, if I am not mistaken, @testset evaluates its inner expression in a separate scope. At least, quick testing suggests so:

julia> @testset "Tests" begin
       x = 10
       @test 2.0 == 2
       end
Test Summary: | Pass  Total  Time
Tests         |    1      1  0.0s
Test.DefaultTestSet("Tests", Any[], 1, false, false, true, 1.730812778488919e9, 1.73081277851395e9, false, "REPL[3]")

julia> x
ERROR: UndefVarError: `x` not defined

Second, any expression in Julia evaluates to a value. The value of let ... end expression is equal to the value of the last evaluated subexpression. You can check that following works perfectly fine:

let
    x = 1
    x + 1
end == 2

So,

@test let
    x = 2
    x^2 == 4
end

is perfectly valid.

4

There are myriads of examples of usage of Test.jl in the source code of julia here: julia/test at master · JuliaLang/julia · GitHub

5

You can also nest testsets:

julia> @testset "outer" begin
           @testset "inner1" begin
               x = 2
               @test 1+1 == 2
           end
           @testset "inner2" begin
               y = 3
               @test 1+x == y
           end
       end
inner2: Error During Test at REPL[6]:8
  Test threw exception
  Expression: 1 + x == y
  UndefVarError: `x` not defined

Using let is fine, too — even if there are no “let bindings” you explicitly introduce, it’s still creating a scope block.