Nested async testsets

General Usage
,
1

Documenting this mostly for myself, but may be useful for others as well.

Julia currently does not handle nested asynchronous test sets correctly.

julia> @testset verbose=true "foo" begin
           @sync @async @testset "bar" begin
               @test true
           end
       end;
Test Summary: | Pass  Total  Time
bar           |    1      1  0.0s
Test Summary: |Time
foo           | None  0.0s

The following macro fixes this.

"""
   @async_testset args...

Like `@async @testset args...`, but makes sure that the given test set is
executed within the scope of the enclosing test set (if any).
"""
macro async_testset(args...)
    testset_stack = gensym("testset_stack")
    esc(quote
        let $testset_stack = Base.task_local_storage(:__BASETESTNEXT__)
            @async begin
                Base.task_local_storage(
                    :__BASETESTNEXT__,
                    $testset_stack
                )
                @testset($(args...))
            end
        end
    end)
end

Example:

julia> @testset verbose=true "foo" begin
           @sync @async_testset "bar" begin
               @test true
           end
       end;
Test Summary: | Pass  Total  Time
foo           |    1      1  0.0s
  bar         |    1      1  0.0s
3 Likes
2

Looks good. Why not open a PR for the Test module?

3

Actually, it’s broken.

julia> @testset verbose=true "parent" begin
           @sync begin
               @async_testset "child 1" begin
                   sleep(1.0)
               end
               @async_testset "child 2" begin
               end
           end
       end;
Test Summary: |Time
parent        | None  1.0s
  child 1     | None  1.0s

I believe the problem is that the testing library gets confused if pushes and pops don’t align. The fact that the authors of Test explicitly used task-local storage should have been a hint, really…

4

Yes, I think some form of synchronization between tasks is needed in order to ensure that sub-testsets are recorded sequentially in their parent.

Something like this should mostly work (although the output of concurrent tasks gets mangled):

using Test


struct SilentTestSet <: Test.AbstractTestSet
    ts
    SilentTestSet(args...; kwargs...) = new(Test.DefaultTestSet(args...; kwargs...))
end

Test.record(ts::SilentTestSet, arg) = Test.record(ts.ts, arg)
Test.finish(ts::SilentTestSet) = ts.ts


function async_testsets(body)
    channel = Channel()
    task = let parent = Test.get_testset()
        @async for ts in channel
            Test.record(parent, ts)
        end
    end

    function run_testset(body, name)
        try
            ts = @testset SilentTestSet "$name" begin
                body()
            end
            push!(channel, ts)
        catch e
            e isa Test.TestSetException || rethrow()
        end
    end
    body(run_testset)

    close(channel)
    wait(task)
end

julia> @testset "Root" begin
           async_testsets() do run_testset
               @sync begin
                   @async run_testset("Child 1") do
                       @test 1==1
                       @test 1==2
                   end
                   @async run_testset("Child 2") do
                       @test 1==1
                       @test 3==4
                   end
               end
           end
       end
Child 1: Test Failed at REPL[6]:6
  Expression: 1 == 2
   Evaluated: 1 == 2
Stacktrace:
 [1] Child 2macro expansion: 
Test Failed   @  at ~/.asdf/installs/julia/1.8.5/share/julia/stdlib/v1.8/Test/src/REPL[6]:10
Test.jl:464  Expression:  [inlined]3 == 4

   Evaluated: 3 == 4
Stacktrace:
 [1]  macro expansion[2]
    @ (::var"#8#13")~/.asdf/installs/julia/1.8.5/share/julia/stdlib/v1.8/Test/src/(Test.jl:464) [inlined]

   @  Main[2]  ./(::var"#10#15")REPL[6]:6()

   @ Main ./REPL[6]:10
Test Summary: | Pass  Fail  Total  Time
Root          |    2     2      4  0.4s
  Child 1     |    1     1      2      
  Child 2     |    1     1      2      
ERROR: Some tests did not pass: 2 passed, 2 failed, 0 errored, 0 broken.
5

I think you can fix the output mangling by recording test results within @async contexts and then playing them back serially after the @async sections completed. But for my part, I came to the conclusion that for my purposes, a simple @assert works just as well as an @test, and doing so resolves all concurrency issues without straying too far from standard Julia.