Selective testing

General Usage
,
1

I’m wondering if there is a way in Julia to run a subset of a package’s tests. The use case is pretty simple: I have a failing test, and I’m working my way to fixing the bug; however, I don’t want to have to run the whole suite of tests to check if my fixes work, since that may take a few minutes. Instead, I want to run only the test sets defined in a file, or whose name matches some regexp.

Is there anything like this in Julia?

4 Likes
2

Maybe just: in the REPL using Test and copy+paste that one test?

3

That would work I guess, but I’m wondering if any Pkg has any option for that, and if not, whether there are any plans for such a feature.

I’ll have to admit: this is mainly motivated by being able to do so in Python (with pytest) but not (?) in Julia.

My current solution is going to runtests.jl, commenting out all the includes except the one, running the tests. Not quite the smoothest flow I’d say.

4

I was looking for this just the other day, and found this other discussion about other testing packages. The @includetests macro from TestSetExtensions.jl kinda does what you want. I haven’t seen anything as nice as pytest’s features.

I think there’s definitely space in the ecosystem for more testing tools.

3 Likes
5

I usually place my tests in functions, as in

# foo_tests.jl
function footests()
  @testset "Foo related test" begin
    @test foo() == 1
  end
end

@testset "All"
  footest()
end

# runtests.jl
@testset "All tests" begin
  include("foo_tests.jl")
end

Then I can simply include("test/foo_tests.jl") or even more granular footest().

It also provides some insulation of the different tests.

1 Like
6

One option is to use Juno’s inline evaluation and evaluate single top level testsets at a time directly from inside the runtests.jl file. I do this alot and it works well, although its not as convenient for doing multiple testsets, especially if they’re spread across different files.

2 Likes
7

Perhaps a little late but this PR #1226 has been merged allows us to do what you want.

That PR allows us to inspect the ARGS variable in our runtests.jl file, this would allow us to selectively register @testsets, for example, suppose that we have 2 tests; "simple" and "complex":

using Test, MyPackage

if isempty(ARGS) ||  "all" in ARGS
    all_tests = true
else
    all_tests = false
end

if all_tests || "simple" in ARGS
   @testset "simple" begin
       @test 1 == 1 # Dummy test
   end
end

if all_tests || "complex" in ARGS
   @testset "complex" begin
       @test 2 == 2 # Dummy test
   end
end


# .. Other tests ...

You would then execute the "simple" test only like so from the CLI:

julia --project -e 'import Pkg;Pkg.test(test_args=["simple"])'

To run all tests just don’t pass the test_args or pass only "all":

julia --project -e 'import Pkg;Pkt.test(test_args=["all"])'
julia --project -e 'import Pkg;Pkt.test()'

I created this simple zshell function to shorten the command:

function jltest {
    if [ $# -eq 0 ]; then # Test all packages
        julia --project -e "import Pkg;Pkg.test()" 
        return $?  
    fi

    local quoted_test_names=""
    for arg in "${@}"; do
        quoted_test_names="$quoted_test_names\"$arg\""
        if [ $# -gt 1 ]; then
            quoted_test_names="$quoted_test_names, "
        fi
    done

    julia --project -e "import Pkg;Pkg.test(test_args=[$quoted_test_names])"
}

Which you can then use like so:

jltest simple             # Test only simple
jltest simple complex     # Test "simple" and "complex"
jltest all                # Test all tests
jltest                    # Test all tests

Hope it helps!

2 Likes
8

Or ReTest.jl. From the documentation:

@testset "a" begin
    @test true
    @testset "b" begin
    end
    @testset "c" begin
    end
end

running retest(M, "a") will run everything, retest(M, "b") will run @test true and @testset "b" but not @testset "c" .

7 Likes