Run test subsets with `Pkg.test`

It doesn’t seem to be possible to define groups of tests and then run only a given group of tests with Pkg.test. Is there an alternative recommended mechanism to achieve this goal?

Here is the use-case. I have a large package where running all the tests takes a long time. I am only working on a small part of it, so I would like to only run the tests related to the part that I am working on.

I could define test functions within the package, then import the package and call the test directly. This way I can organize things how I want, but I wonder if there is a more canonical way of doing it.

6 Likes

I think this would be a great feature. I usually separate out my tests into smaller file, and include them all into runtests.jl. It’s pretty easy to comment out the slow tests while developing.

2 Likes

Of course, this is a workaround. But this is not very robust. You might forget that you commented something, and I don’t think constantly changing a source file without need is good practice.

1 Like

I agree, I just wanted to share it in case it would help temporarily.

2 Likes

Ideally, it shows up when you review the code before committing.

1 Like

You can easily structure runtests.jl to select particular test groups depending on the environment.
Either write your own logic (following, e.g. DataStructures), or use @includetests from TestSetExtensions.

3 Likes

What do you mean by “environment”? Environment variables?

Both of the approaches I mentioned use ARGS which can be set interactively or on a command line, but other environment variables could also work.

2 Likes

+1 for an “official” way to do this!

2 Likes

Is there any update on this? Most languages I used supports running single test or groups. It is really convenient and productive to be able to integrate with modern editor like vscode to click a button above the test code block to run single tests or save multiple run configurations of tests. Which is the repo to open a feature request for this, julia or Pkg?

1 Like

you could run the test files in a specific directory and path using Jive
visit https://github.com/wookay/Jive.jl

I think that this is implemented in

2 Likes

Could we pretty please get this in Pkg itself? It’s super handy to be able to just run one test when developing the test or the code being tested.

In the latest Pkg version there is https://github.com/JuliaLang/Pkg.jl/pull/1226. This can be used to customize Pkg.test(...). I am not sure if this landed in julia 1.5.

Has that landed in 1.5? Is the final form documented somewhere? I tried to follow that thread, but I could not find out how to do it at the end.

It allows Pkg to pass arguments to the test process, but Pkg itself is not a testing framework so it is up to the testing framework you use to make use of those arguments.

1 Like