Pkg test run only one testset

General Usage
1

I’m developing a package and I have a test folder with a runtest.jl file and other test files.
The runtest.jl is the following

cd(@__DIR__)

using Pkg

using Test, TestSetExtensions
using Wasabi
using Mocking

@testset ExtendedTestSet "Wasabi tests" begin
    @includetests ARGS
end

If I run test from Pkg in the julia repl everything works fine.
Some tests take lot of time to run, so I want to be able to run only a specific testset and ignore others.
Running the runtest.jl directly I can do

julia test/runtest.jl nameoftest

But that doesn’t work since I get

LoadError: ArgumentError: Package TestSetExtensions not found in current path.

Following the doc I don’t have any test specific dependency in the project but I have a project.toml file under test/

I’ve also tried

julia --project=./test test/runtests.jl builder

but I got

LoadError: ArgumentError: Package Wasabi not found in current path

where Wasabi is the package I’m currently working on.

How can I execute only a specific testset?

2

I don’t know about TestSetExtensions, but with TestItems within VSCode that is very practical:

1 Like
3

Take a look at ReTest.jl. It is semi-compatible with the native test structure (via hijack), and it offers selective test running as a service. I find it useful occasionallly, even though it fails to cope sometimes.

1 Like
4

FWIW, it’s pretty easy to roll your own, also. I always use this chunk of code I found in StaticArrays:

# Hook into Pkg.test so that tests from a single file can be run.  For example,
# to run only the MVector and SVector tests, use:
#
#   Pkg.test("StaticArrays", test_args=["MVector", "SVector"])
#
enabled_tests = lowercase.(ARGS)
function addtests(fname)
    key = lowercase(splitext(fname)[1])
    if isempty(enabled_tests) || key in enabled_tests
        Random.seed!(42)
        include(fname)
    end
end

Then, write your tests in separate files and add calls in runtests.jl like addtests("test_file1.jl").

I expect to move to TestItems and TestItemRunner soon, though.

3 Likes
5

Thanks for the suggestion.

6

This is a cool idea! I enhanced it slightly, now my runtests.jl file looks like this:

smartcase = any(isuppercase, join(ARGS)) ? identity : lowercase
filter(readdir()) do file
    return endswith(file, ".jl") &&
        !startswith(file, ['.', '#']) &&
        file != "runtests.jl" &&
        (isempty(ARGS) || any(occursin(smartcase(file)), ARGS))
end .|> include

It basically auto-discovers test files in the test directory, but runs only those which contain any of the CLI arguments as substrings. The script uses smartcase matching, i.e., if the CLI arguments contain uppercase letters, then substring matching is case-sensitive, otherwise it is case-insensitive.

And then tests are invoked with the following command:

$ alias jlt='julia --project -e "using Pkg; Pkg.test(test_args = ARGS)"'
$ jlt sub str
...

Works like a charm!

2 Likes
7

@iskyd, have you added TestSetExtensions (as well as Mocking) to the dependencies of test/Project.toml? I’m guessing that is what you are missing.

8

I elaborated this test auto-discovery idea in a new Julia package I just released, see the announcement here.

2 Likes
9

Thanks, I think that this is a really good solution.

1 Like