# Pkg test run only one testset

**URL:** <https://discourse.julialang.org/t/pkg-test-run-only-one-testset/96356>\
**Category:** General Usage\
**Tags:** testing\
**Created:** [March 20, 2023, 5:27pm UTC](https://discourse.julialang.org/t/pkg-test-run-only-one-testset/96356 "2023-03-20T17:27:10Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![iskyd](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/iskyd/32/44797_2.png) [@iskyd](https://discourse.julialang.org/u/iskyd)\
**Post date:** [March 20, 2023, 5:27pm UTC](https://discourse.julialang.org/t/pkg-test-run-only-one-testset/96356/1 "2023-03-20T17:27:10Z")

</div>

I’m developing a package and I have a test folder with a runtest.jl file and other test files.  
The runtest.jl is the following

```julia
cd(@ __DIR__ )

using Pkg

using Test, TestSetExtensions
using Wasabi
using Mocking

@testset ExtendedTestSet "Wasabi tests" begin
    @includetests ARGS
end

```

If I run test from Pkg in the julia repl everything works fine.  
Some tests take lot of time to run, so I want to be able to run only a specific testset and ignore others.  
Running the runtest.jl directly I can do

```julia
julia test/runtest.jl nameoftest

```

But that doesn’t work since I get

> LoadError: ArgumentError: Package TestSetExtensions not found in current path.

Following [the doc](https://pkgdocs.julialang.org/v1/creating-packages/#Test-specific-dependencies-1) I don’t have any test specific dependency in the project but I have a project.toml file under test/

I’ve also tried

```julia
julia --project=./test test/runtests.jl builder

```

but I got

> LoadError: ArgumentError: Package Wasabi not found in current path

where Wasabi is the package I’m currently working on.

How can I execute only a specific testset?

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [March 20, 2023, 5:53pm UTC](https://discourse.julialang.org/t/pkg-test-run-only-one-testset/96356/2 "2023-03-20T17:53:56Z")

</div>

I don’t know about `TestSetExtensions`, but with `TestItems` within VSCode that is very practical:

> [@Prerelease of new testing framework and test run UI in VS Code](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355):
>
> I just pushed out a new release of the Julia VS Code extension (1.7.6) to everyone that includes a preview of our new test UI and a general new testing framework for Julia. For a short demo, take a look at our Juliacon talk [https://youtu.be/Okn\_HKihWn8?t=1268](https://youtu.be/Okn_HKihWn8?t=1268). This new set of features consists of two new packages ([TestItems.jl](https://github.com/julia-vscode/TestItems.jl) and [TestItemRunner.jl](https://github.com/julia-vscode/TestItemRunner.jl)) and the new UI in the Julia VS Code extension. A good way to refer to this new test framework is by the “test item” name. How to write test items …

---

<div class="post-metadata">

**Author:** ![HanD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hand/32/213908_2.png) [@HanD](https://discourse.julialang.org/u/HanD)\
**Post date:** [March 21, 2023, 8:23am UTC](https://discourse.julialang.org/t/pkg-test-run-only-one-testset/96356/3 "2023-03-21T08:23:06Z")

</div>

Take a look at [ReTest.jl](https://github.com/JuliaTesting/ReTest.jl). It is semi-compatible with the native test structure (via hijack), and it offers selective test running as a service. I find it useful occasionallly, even though it fails to cope sometimes.

---

<div class="post-metadata">

**Author:** ![moble](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/moble/32/23535_2.png) [@moble](https://discourse.julialang.org/u/moble)\
**Post date:** [March 21, 2023, 2:11pm UTC](https://discourse.julialang.org/t/pkg-test-run-only-one-testset/96356/4 "2023-03-21T14:11:30Z")

</div>

FWIW, it’s pretty easy to roll your own, also. I always use this chunk of code I [found in `StaticArrays`](https://github.com/JuliaArrays/StaticArrays.jl/blob/7ca6faab7fb8863c13f2e957d54424567980d97e/test/runtests.jl#L10-L22):

```julia
# Hook into Pkg.test so that tests from a single file can be run. For example,
# to run only the MVector and SVector tests, use:
#
# Pkg.test("StaticArrays", test_args=["MVector", "SVector"])
#
enabled_tests = lowercase.(ARGS)
function addtests(fname)
    key = lowercase(splitext(fname)[1])
    if isempty(enabled_tests) || key in enabled_tests
        Random.seed!(42)
        include(fname)
    end
end

```

Then, write your tests in separate files and add calls in `runtests.jl` like `addtests("test_file1.jl")`.

I expect to move to `TestItems` and `TestItemRunner` soon, though.

---

<div class="post-metadata">

**Author:** ![iskyd](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/iskyd/32/44797_2.png) [@iskyd](https://discourse.julialang.org/u/iskyd)\
**Post date:** [March 22, 2023, 2:18pm UTC](https://discourse.julialang.org/t/pkg-test-run-only-one-testset/96356/5 "2023-03-22T14:18:02Z")

</div>

Thanks for the suggestion.

---

<div class="post-metadata">

**Author:** ![HanD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hand/32/213908_2.png) [@HanD](https://discourse.julialang.org/u/HanD)\
**Post date:** [March 30, 2023, 9:30am UTC](https://discourse.julialang.org/t/pkg-test-run-only-one-testset/96356/6 "2023-03-30T09:30:57Z")

</div>

This is a cool idea! I enhanced it slightly, now my `runtests.jl` file looks like this:

```julia
smartcase = any(isuppercase, join(ARGS)) ? identity : lowercase
filter(readdir()) do file
    return endswith(file, ".jl") &&
        !startswith(file, ['.', '#']) &&
        file != "runtests.jl" &&
        (isempty(ARGS) || any(occursin(smartcase(file)), ARGS))
end .|> include

```

It basically auto-discovers test files in the `test` directory, but runs only those which contain any of the CLI arguments as substrings. The script uses smartcase matching, i.e., if the CLI arguments contain uppercase letters, then substring matching is case-sensitive, otherwise it is case-insensitive.

And then tests are invoked with the following command:

```julia
$ alias jlt='julia --project -e "using Pkg; Pkg.test(test_args = ARGS)"'
$ jlt sub str
...

```

Works like a charm!

---

<div class="post-metadata">

**Author:** ![HanD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hand/32/213908_2.png) [@HanD](https://discourse.julialang.org/u/HanD)\
**Post date:** [March 30, 2023, 9:42am UTC](https://discourse.julialang.org/t/pkg-test-run-only-one-testset/96356/7 "2023-03-30T09:42:27Z")

</div>

@iskyd, have you added `TestSetExtensions` (as well as `Mocking`) to the dependencies of `test/Project.toml`? I’m guessing that is what you are missing.

---

<div class="post-metadata">

**Author:** ![HanD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hand/32/213908_2.png) [@HanD](https://discourse.julialang.org/u/HanD)\
**Post date:** [April 6, 2023, 3:32pm UTC](https://discourse.julialang.org/t/pkg-test-run-only-one-testset/96356/8 "2023-04-06T15:32:35Z")

</div>

I elaborated this test auto-discovery idea in a new Julia package I just released, see the announcement [here](https://discourse.julialang.org/t/tidytest-jl-test-execution-with-auto-discovery-and-a-neater-test-summary/97183).

---

<div class="post-metadata">

**Author:** ![iskyd](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/iskyd/32/44797_2.png) [@iskyd](https://discourse.julialang.org/u/iskyd)\
**Post date:** [April 6, 2023, 3:49pm UTC](https://discourse.julialang.org/t/pkg-test-run-only-one-testset/96356/9 "2023-04-06T15:49:38Z")

</div>

Thanks, I think that this is a really good solution.