# Nested async testsets

**URL:** <https://discourse.julialang.org/t/nested-async-testsets/95367>\
**Category:** General Usage\
**Tags:** testing, async\
**Created:** [March 1, 2023, 7:27am UTC](https://discourse.julialang.org/t/nested-async-testsets/95367 "2023-03-01T07:27:09Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ettersi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ettersi/32/6829_2.png) [@ettersi](https://discourse.julialang.org/u/ettersi)\
**Post date:** [March 1, 2023, 7:27am UTC](https://discourse.julialang.org/t/nested-async-testsets/95367/1 "2023-03-01T07:27:09Z")

</div>

Documenting this mostly for myself, but may be useful for others as well.

* * *

Julia currently does not handle nested asynchronous test sets correctly.

```julia
julia> @testset verbose=true "foo" begin
           @sync @async @testset "bar" begin
               @test true
           end
       end;
Test Summary: | Pass Total Time
bar | 1 1 0.0s
Test Summary: |Time
foo | None 0.0s

```

The following macro fixes this.

```julia
"""
   @async_testset args...

Like `@async @testset args...`, but makes sure that the given test set is
executed within the scope of the enclosing test set (if any).
"""
macro async_testset(args...)
    testset_stack = gensym("testset_stack")
    esc(quote
        let $testset_stack = Base.task_local_storage(: __BASETESTNEXT__ )
            @async begin
                Base.task_local_storage(
                    : __BASETESTNEXT__ ,
                    $testset_stack
                )
                @testset($(args...))
            end
        end
    end)
end

```

Example:

```julia
julia> @testset verbose=true "foo" begin
           @sync @async_testset "bar" begin
               @test true
           end
       end;
Test Summary: | Pass Total Time
foo | 1 1 0.0s
  bar | 1 1 0.0s

```

---

<div class="post-metadata">

**Author:** ![zgornel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zgornel/32/217487_2.png) [@zgornel](https://discourse.julialang.org/u/zgornel)\
**Post date:** [March 2, 2023, 11:03am UTC](https://discourse.julialang.org/t/nested-async-testsets/95367/2 "2023-03-02T11:03:04Z")

</div>

Looks good. Why not open a PR for the `Test` module?

---

<div class="post-metadata">

**Author:** ![ettersi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ettersi/32/6829_2.png) [@ettersi](https://discourse.julialang.org/u/ettersi)\
**Post date:** [March 2, 2023, 12:58pm UTC](https://discourse.julialang.org/t/nested-async-testsets/95367/3 "2023-03-02T12:58:41Z")

</div>

Actually, it’s broken.

```julia
julia> @testset verbose=true "parent" begin
           @sync begin
               @async_testset "child 1" begin
                   sleep(1.0)
               end
               @async_testset "child 2" begin
               end
           end
       end;
Test Summary: |Time
parent | None 1.0s
  child 1 | None 1.0s

```

I believe the problem is that the testing library gets confused if pushes and pops don’t align. The fact that the authors of `Test` explicitly used task-local storage should have been a hint, really…

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [March 2, 2023, 1:58pm UTC](https://discourse.julialang.org/t/nested-async-testsets/95367/4 "2023-03-02T13:58:48Z")

</div>

> [@ettersi](#):
>
> I believe the problem is that the testing library gets confused if pushes and pops don’t align. The fact that the authors of `Test` explicitly used task-local storage should have been a hint, really…

Yes, I think some form of synchronization between tasks is needed in order to ensure that sub-testsets are recorded sequentially in their parent.

Something like this should mostly work (although the output of concurrent tasks gets mangled):

```julia
using Test

struct SilentTestSet <: Test.AbstractTestSet
    ts
    SilentTestSet(args...; kwargs...) = new(Test.DefaultTestSet(args...; kwargs...))
end

Test.record(ts::SilentTestSet, arg) = Test.record(ts.ts, arg)
Test.finish(ts::SilentTestSet) = ts.ts

function async_testsets(body)
    channel = Channel()
    task = let parent = Test.get_testset()
        @async for ts in channel
            Test.record(parent, ts)
        end
    end

    function run_testset(body, name)
        try
            ts = @testset SilentTestSet "$name" begin
                body()
            end
            push!(channel, ts)
        catch e
            e isa Test.TestSetException || rethrow()
        end
    end
    body(run_testset)

    close(channel)
    wait(task)
end

```

```julia
julia> @testset "Root" begin
           async_testsets() do run_testset
               @sync begin
                   @async run_testset("Child 1") do
                       @test 1==1
                       @test 1==2
                   end
                   @async run_testset("Child 2") do
                       @test 1==1
                       @test 3==4
                   end
               end
           end
       end
Child 1: Test Failed at REPL[6]:6
  Expression: 1 == 2
   Evaluated: 1 == 2
Stacktrace:
 [1] Child 2macro expansion: 
Test Failed @ at ~/.asdf/installs/julia/1.8.5/share/julia/stdlib/v1.8/Test/src/REPL[6]:10
Test.jl:464 Expression: [inlined]3 == 4

   Evaluated: 3 == 4
Stacktrace:
 [1] macro expansion[2]
    @ (::var"#8#13")~/.asdf/installs/julia/1.8.5/share/julia/stdlib/v1.8/Test/src/(Test.jl:464) [inlined]

   @ Main[2] ./(::var"#10#15")REPL[6]:6()

   @ Main ./REPL[6]:10
Test Summary: | Pass Fail Total Time
Root | 2 2 4 0.4s
  Child 1 | 1 1 2      
  Child 2 | 1 1 2      
ERROR: Some tests did not pass: 2 passed, 2 failed, 0 errored, 0 broken.

```

---

<div class="post-metadata">

**Author:** ![ettersi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ettersi/32/6829_2.png) [@ettersi](https://discourse.julialang.org/u/ettersi)\
**Post date:** [March 4, 2023, 1:38pm UTC](https://discourse.julialang.org/t/nested-async-testsets/95367/5 "2023-03-04T13:38:04Z")

</div>

I think you can fix the output mangling by recording test results within `@async` contexts and then playing them back serially after the `@async` sections completed. But for my part, I came to the conclusion that for my purposes, a simple `@assert` works just as well as an `@test`, and doing so resolves all concurrency issues without straying too far from standard Julia.