Skipping a whole `@testset`?

General Usage
1

I have a bunch of similar @testsets, and I know that some of them currently fail, and will do so until some upgrades are made to my package. For skipping single tests, there’s @test_skip or @test_broken, but is there a nice way to skip a whole @testset? I don’t want to alter the individual @test calls because they’re inside a function that’s called in @testsets that pass (and I don’t want to waste time running stuff that I know will fail). Putting @test_skip @testset ... sort of works: the @testsets are not run, and some ‘Broken’ entries appear in the output, but it would be nice if, for example, the @testsets’ names would still appear in verbose output, but highlighted and marked as broken. I guess a skip=true optional argument to @testset could do this?

Edit: even better than a skip=true argument would be skip="foobar" where the string can give a reason the test is skipped, which could be printed beside the @testset’s name in the summary.

2

see also Standard ways to run common commands - #12 by djsegal

PS sorry I missed that thread before opening this one - I made the mistake of searching for @testset, which apparently looks for posts by a user named @testset rather than posts mentioning the macro…

3

I ran into this problem, so I wrote a small macro:

s = ArgParseSettings()
@add_arg_table! s begin
    "--include_tests"
    arg_type = String
    required = false
    action = "store_arg"
    nargs = '*'
end


parsed_args = parse_args(s)
include_tests = Set(parsed_args["include_tests"])

macro test_if_included(expr::Expr)
  # For now, assume the expression is a macro followed by a string, e.g.
  # @timeit "name1" or @testset "name2"
  name = expr.args[3]
  if isempty(include_tests)
    esc(expr)
  elseif name in include_tests
    esc(expr)
  else
    nothing
  end
end

# Single test with @timeit
@test_if_included @timeit "name" begin ...

# Multiple tests with @testset
@test_if_included @testset "name" begin...

Then you can run it like so:

using Pkg; Pkg.test(test_args=["--include_tests", "my_test_set"])

2 Likes
4

Thanks @Satvik, but I was aiming in particular at the case when some tests are ‘broken’, rather than wanting to switch between them interactively.

I spent a few hours learning that wrapping @testset in another macro isn’t possible (at least for someone with my limited knowledge of macros), see Calling a macro from within a macro, revisited, https://github.com/JuliaLang/julia/issues/37691, and Nested macros and esc. After giving up on that, I ended up coming up with this macro that skips a testset and adds a DefaultTestSet with a dummy broken test so that it can show up in the summary:

import Test: Test, finish
using Test: DefaultTestSet, Broken
using Test: parse_testset_args

"""
Skip a testset

Use `@testset_skip` to replace `@testset` for some tests which should be skipped.

Usage
-----
Replace `@testset` with `@testset "reason"` where `"reason"` is a string saying why the
test should be skipped (which should come before the description string, if that is
present).
"""
macro testset_skip(args...)
    isempty(args) && error("No arguments to @testset_skip")
    length(args) < 2 && error("First argument to @testset_skip giving reason for "
                              * "skipping is required")

    skip_reason = args[1]

    desc, testsettype, options = parse_testset_args(args[2:end-1])

    ex = quote
        # record the reason for the skip in the description, and mark the tests as
        # broken, but don't run tests
        local ts = DefaultTestSet(string($desc, " - ", $skip_reason))
        push!(ts.results, Broken(:skipped, "skipped tests"))
        local ret = finish(ts)
        ret
    end

    return ex
end

I’d still vote for something like this as a feature of the standard-library @testset and DefaultTestSet.

5 Likes