# Prerelease of new testing framework and test run UI in VS Code

**URL:** <https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355>\
**Category:** VS Code\
**Created:** [August 25, 2022, 7:37pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355 "2022-08-25T19:37:13Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [August 25, 2022, 7:37pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/1 "2022-08-25T19:37:13Z")

</div>

I just pushed out a new release of the Julia VS Code extension (1.7.6) to everyone that includes a preview of our new test UI and a general new testing framework for Julia.

For a short demo, take a look at our Juliacon talk [https://youtu.be/Okn\_HKihWn8?t=1268](https://youtu.be/Okn_HKihWn8?t=1268).

This new set of features consists of two new packages ([TestItems.jl](https://github.com/julia-vscode/TestItems.jl) and [TestItemRunner.jl](https://github.com/julia-vscode/TestItemRunner.jl)) and the new UI in the Julia VS Code extension. A good way to refer to this new test framework is by the “test item” name.

### How to write test items

The core feature in this new framework is that you can structure tests into `@testitem` blocks and then individually run those, rather than having to run all your tests at once. A typical `@testitem` might look like this:

```julia
@testitem "First tests" begin
    x = foo("bar")

    @test length(x)==3
    @test x == "bar"
end

```

A `@testitem` always has a name (here “First tests”) and then some code in a `begin ... end` block. The code inside a `@testitem` must be executable by itself, i.e. it can _not_ depend on code that appears outside of the `@testitem`, unless that code is somehow explicitly imported or included from within the `@testitem`. There is one exception to this: the code inside the `@testitem` will run inside a temporary module where `using Test` and `using MYPACKAGENAME` was already executed, so anything exported from either the `Test` module or the package your are developing can be directly used. In the example above this applies to the `foo` function (presumably defined in the package that is being tested) and the `@test` macro.

`@testitem`s can appear _anywhere_ in a package. They do _not_ have to be in the `test` folder, nor do they have to be in a file that is included by `test/runtests.jl`. In fact, `@testitem`s can even be located inside your regular package code, for example next to the code they are testing. In that case you just need to take a dependency on the [TestItems.jl](https://github.com/julia-vscode/TestItems.jl) package so that you have access to the `@testitem` macro. If you have a package `MyPackage`, then the file `src/MyPackage.jl` could look like this:

```julia
module MyPackage

using TestItems

export foo

foo(x) = x

@testitem "First tests" begin
    x = foo("bar")

    @test length(x)==3
    @test x == "bar"
end

end

```

If you don’t like this inline `@testitem` style, you can also just put `@testitem` blocks into Julia files in your `test` folder.

### Running test items inside VS Code

When you open a Julia package inside VS Code and have the [Julia extension](https://www.julia-vscode.org/) installed it will constantly (after every keypress!) look for any and all `@testitem`s in your Julia files. If any are found, they will appear in various places in the UI.

You can find all detected `@testitem`s in the Testing activity bar in VS Code:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/4/0/408868a33ac183d1e0a99211ec87b59f316e9ea5.png)

The testing activity area then provides you with options to run individual `@testitem`s, look at results etc.

VS Code will also place a small little run button next to each detected `@testitem` in the text editor itself:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/e/2/e2b266e9043066bd8002f2ffcf6486bd86b5f42f.png)

In addition to all these UI elements that allow you to run tests, there is also UI to display test results. For example, when you run tests and some of them fail, the extension will collect all these test failures and then display them in a structured way, directly at the place in the code where a specific test failed:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/8/c/8c3ed490414566d103a32017b0fe86f22688e3b0.png)

Especially when you run a lot of tests with large test files this makes it much easier to find the specific test that failed, no more hunting in the REPL for file and line information!

### Running tests from the command line

This part is a little less fleshed out, but you can use the [TestItemRunner.jl](https://github.com/julia-vscode/TestItemRunner.jl) package to run `@testitem`s as part of a traditional `Pkg.test` workflow. This makes it easy to integrate these new types of tests with for example a continuous integration setup.

To enable integration with `Pkg.test` for a package that uses `@testitem`, you just have to do two things:

1. Add [TestItemRunner.jl](https://github.com/julia-vscode/TestItemRunner.jl) as a test dependency to your package
2. Put the following code into the package’s `test/runtests.jl` file:

```julia
using TestItemRunner

@run_package_tests

```

I hope that in the future we can make the [TestItemRunner.jl](https://github.com/julia-vscode/TestItemRunner.jl) package much more feature complete, for example add the ability to only run a subset of `@testitem`s (as you can already do in VS Code), add support for parallel execution etc. Help is most welcome!

### Under the hood

We already have some great testing packages in Julialand, and some of them (like the excellent [ReTest.jl](https://github.com/JuliaTesting/ReTest.jl)) seem to provide a lot of similar functionality already, so why create yet another testing framework?

The core reason is that we have a very different requirement for the VS Code extension: we need to detect test items at every keystroke. To do this, I used a completely different test detection strategy than any of the existing testing frameworks. In the test item design, no user code is run at all for test item detection. Test item detection is purely done by syntactic analysis of Julia source files. This kind of approach integrates _really_ well with the existing analysis we have in the [LanguageServer.jl](https://github.com/julia-vscode/LanguageServer.jl) that powers the rest of the Julia extension.

One nice side benefit is that the design of this test item framework is quite simple. For example, the definition of the `@testitem` macro is almost hilariously simple, you can see it [here](https://github.com/julia-vscode/TestItems.jl/blob/e710519d9ad4115440781d96fc297ddabd722dc4/src/TestItems.jl). Having it so simple and not doing anything is ideal, because that means that adding inline tests into the actual package code should really not add any runtime overhead at all to a package (it does probably add a tiny bit of precompile time).

I also want to point out some more details about the test runner in VS Code. The design there is that we first detect in which environment a given `@testitem` should run, then we spin up a test process per individual environment. These test processes are long run, i.e. once they have started they are reused whenever you execute another new `@testitem`. By reusing these processes, we can cut out a huge amount of delay between running tests, rerunning an individual test item is often completely instantaneous. The Julia extension now also ships [Revise.jl](https://github.com/timholy/Revise.jl) as part of the extension, and uses that to detect any code changes you make to the package code you are testing. The integration with [Revise.jl](https://github.com/timholy/Revise.jl) is completely under the hood and automatic. For example, if you make a change to your package that `Revise` cannot track (like redefining a struct), the extension will recognize that and automatically restart the test process instead of relying on `Revise` to handle this update. The net effect of that is that you can just freely edit the code you are testing and the test code itself, rerun small parts of it all the time and there should be minimal delays in all of that.

### Roadmap

While this feature ships in the regular Julia VS Code extension, we are declaring it a preview at the moment. We want to collect feedback on the design and usability for a while, potentially address design issues that might crop up, and only then will we declare it stable and released.

So, please try this out and let us know what you think! And if some folks want to help improve [TestItemRunner.jl](https://github.com/julia-vscode/TestItemRunner.jl), that would be especially fantastic.

# 10/3/2022 Update

I just pushed out a new build of the extension to ~~the insider channel~~ everyone that brings a lot of new features along, and there is also a new corresponding [TestItemRunner.jl](https://github.com/julia-vscode/TestItemRunner.jl) version tagged. Please give these new versions a try and report back here or (especially for specific bugs) over in the issues on GitHub.

Here is a rundown of the new features:

#### Tags

You can now add tags to `@testitem`s. Tags can be used both in the VS Code UI and via the `TestItemRunner` to filter which test items you want to run.

The syntax for adding tags is this:

```julia
@testitem "My testitem" tags=[:skipci, :important] begin
    x = foo("bar")

    @test length(x)==3
    @test x == "bar"
end

```

You can then filter the test list in the VS Code UI with these same tags:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/1/b/1badedcd6bf98a547dc212aa1175c26b3bb1b1c6.png)

And you can also use tags in `test/runtests.jl` to filter down the list of tests that will run via the traditional `Pkg.test` entry point:

```julia
using TestItemRunner

@run_package_tests filter=ti->!(:skipci in ti.tags)

```

Scroll down for a more complete description of the new `filter` keyword for the `@run_package_tests` macro.

#### Parallel test execution in VS Code

The VS Code extension has a new setting that controls how many Julia processes you want to use for parallel test execution:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/1/1/1133645131feb401c00cd83cc4ba513c87ecc7d3.png)

The default value (for now) is 1, so you have to change that to use the parallel test execution feature. A value of `0` will use as many test processes as you have processors.

Once you configured more than one test process, individual `@testitem`s will run in parallel, for example here there are four `@testitem`s that are running at the same time:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/f/9/f9fa46608e1bbbc74153bbc1c179d15b62d8fc25.png)

There is a trade-off here: more test processes mean more memory is needed, and there is also potentially additional overhead to get all processes to spin up and be ready to actually run `@testitem`s. I’m still playing around with this, any reports how this feature is working for you would be great!

#### Cancel support

The test UI in VS Code always had a button to cancel a test run:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/f/0/f0c88889e69e9c942bce4d43d1dca6f78b43d236.png)

The new feature is that it now works 🙂

#### Test duration

The extension now keeps track how long it took to run each individual `@testitem` and displays that in the UI. You can even sort the `@testitem`s by duration, very convenient to identify specific tests that are slow! This is how it looks:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/e/8/e878629742c5ca3bc84f8b4f40f12d066cb6c235.png)

#### Workspace integration

Test processes that are launched via this new test UI in VS Code are not automatically terminated, i.e. they hang around and take up memory and other resources. That of course has many benefits, namely that `@testitem`s can be executed very quickly once the test process is up and running, but in some situations one might still want to simply terminate all currently running test processes.

To enable this, all test processes now show up in the Julia Workspace, alongside any REPL or Notebook processes that might also be running. And you can now terminate Julia test processes via this UI by clicking on the `Stop Test Process` button. In this screenshot I have four test processes running:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/c/9/c9d4b2c3321523c97bb75333a5ca407d8682a767.png)

#### Filtering support in [TestItemRunner.jl](https://github.com/julia-vscode/TestItemRunner.jl)

I had mentioned this feature already briefly in the section on tags: you can now pass a generic filter function to the `@run_package_tests` macro to select which `@testitem`s you want to execute. The example from above used tags to select which tests to run, but you can also filter based on the filename where a `@testitem` is defined or the name of the `@testitem`.

The way this works is that you can pass a filter function to the `@run_package_tests` macro. This filter function will be called once for each `@testitem` that is detected in your project, and the function must either return `true` if this test item should be run or `false` to not run it. `@run_package_tests` will pass a named tuple with three fields to your filter function that contains meta information about the specific test item, namely the field `filename` (the full path of the file where the `@testitem` is defined), `name` (the name of the `@testitem` that you defined) and `tags` (a vector of `Symbol`s). With this information you can write arbitrarily complex filter conditions. For example, here I’m filtering out any `@testitem` that has the `:skipci` tag and I’m also only running tests that are defined in one specific file:

```julia
@run_package_tests filter=ti->( !(:skipci in ti.tags) && endswith(ti.filename, "test_foo.jl") )

```

#### Option for default imports

When you write a `@testitem`, by default the package being tested and the `Test` package are imported via an invisible `using` statement. In some cases this might not be desirable, so one can control this behavior on a per `@testitem` level via the `default_imports` option, which accepts a `Bool` value. To disable these default imports you, you would write:

```julia
@testitem "Another test for foo" default_imports=false begin
    using MyPackage, Test

    x = foo("bar")

    @test x != "bar"
end

```

Note how we now need to add the line `using MyPackage, Test` manually to our `@testitem` so that we have access to the `foo` function and `@test` macro.

#### TestItemRunner.jl dependencies are vendored

The `TestItemRunner` package uses packages like `CSTParser` and `Tokenize` internally. This can become a problem if you want to test a package that itself uses one of these packages. In particular, if your package required a different version of say `CSTParser` than `TestItemRunner`, then that did not work in the previous release. But, no more! The latest version of `TestItemRunner` vendors all of its dependencies, so this problem is just gone. Julia’s package manager might not give us private packages, but the VS Code extension has long found a workaround for that and I’m now using the same technique here.

#### Smaller improvements

There were a number of smaller improvements as well:

- The message that the test module is being replaced is no longer shown.
- Test output is labeled a bit better.
- A lot of bug fixes for issues that you all reported, thanks so much for that!

#### Roadmap

All of this is still considered prerelease. I’m also slightly worried that I might have moved a bit too fast with this last batch of new features, i.e. I wouldn’t be too surprised if some new bugs have slipped in. Please test and report back over on GitHub or here!

I do think that in terms of features this is probably roughly what it will be when I declare all of this released. So at the moment I’m thinking we will just do testing from here on out, bug fixes and then declare a first version done. But of course, I might change my mind based on feedback or the weather or whatever 😉

---

<div class="post-metadata">

**Author:** ![phlavenk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/phlavenk/32/1873_2.png) [@phlavenk](https://discourse.julialang.org/u/phlavenk)\
**Post date:** [August 26, 2022, 9:09am UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/2 "2022-08-26T09:09:26Z")

</div>

Hi David,  
I’ve just tried the new test framework functionality and it is incredible - finally, I don’t envy my colleagues that use rust with all the fantastic and real-time test functionality.

Really minimal changes to a package bring you an incredible boost of efficiency:  
(EMProject) pkg\> add TestItemRunner

```julia
# (test) pkg> add TestItems TestItemRunner

# runtest.jl:
using TestItems, TestItemRunner

@testitem "VisualLogging" begin
    include("test_visuallogging.jl")
end
...
@testitem "Lestrade" begin
    include("test_lestrade.jl")
end

@run_package_tests

```

and that are the only changes to the runtest.jl file. All the testing functionality of Base.Test remains intact, however, immediately the new integrated test framework kicks in and re-running your test after a change in code takes seconds - no environment setup, no more wasted minutes for CairoMakie to compile and initialize every time you need to verify your small code change didn’t brake the tests.  
And it is so rewarding, to write the test first, and then just see the red light on the given test item turn green…

Thank you again for the amazing work.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [August 26, 2022, 9:56am UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/3 "2022-08-26T09:56:41Z")

</div>

This test functionality looks great, looking forward to using it!

The requirement to replace `@testset` with `@testitem` does bring some friction, though. Together with the two extra dependencies. I imagine this would significantly slow down the potential adoption among existing packages.  
Can it be made to work with top-level `@testset` entries as well? Let it assume that individual `@testset`s are independent, aside from imports at the top of `runtests.jl`.

---

<div class="post-metadata">

**Author:** ![phlavenk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/phlavenk/32/1873_2.png) [@phlavenk](https://discourse.julialang.org/u/phlavenk)\
**Post date:** [August 26, 2022, 10:42am UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/4 "2022-08-26T10:42:21Z")

</div>

In my example, I’ve replaced code

```julia
# runtest.jl:
include("test_visuallogging.jl")
...
include("test_lestrade.jl")

```

by the code above. All the `@testset`s hierarchy is in the included files - I’ve not touched them. Yes - you can have a much nicer granularity of the `@testset`s in the UI if you replace them in those concrete test files with `@testitem`, but even my example brings already the productivity boost.

And the dependencies `TestItems, TestItemRunner` are only in the project of the test environment, so no change of the dependencies of the package are nescessary.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [August 26, 2022, 11:43am UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/5 "2022-08-26T11:43:55Z")

</div>

This is amazing. But I’m having some trouble here. When I click “play” on this test, it doesn’t do anything (the thing there keeps spinning forever). Even if the test does nothing:

![image](https://global.discourse-cdn.com/julialang/original/3X/5/5/5598bfaac653ecc686afcc86c22ef809add9ba16.png)

I don´t see any output anywhere to try to debug what’s going on.

~~And if I run `] test` it complains `SVector not defined` (reason why I put the `using StaticArrays` inside the `@testitem`, but it didn’t work.~~ Solved that.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [August 26, 2022, 6:57pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/6 "2022-08-26T18:57:08Z")

</div>

> [@aplavin](#):
>
> The requirement to replace `@testset` with `@testitem` does bring some friction, though. Together with the two extra dependencies. I imagine this would significantly slow down the potential adoption among existing packages.

I should probably clarify this a bit: I think users will really only have to deal with the `TestItems` package if they want to write tests inline with their normal package code. In that case you have to add `TestItems` as a regular package dependency so that the `@testitem` macro is available inside your regular package code. But, if you do not plan to use these inline tests, then you don’t need the dependency on `TestItems` at all.

`TestItemRunner` is the package you need as a test dependency, and is essentially only used inside `test/runtests.jl`. When converting an existing package, really one will start by adding that to the `Project.toml` `[test]` section. `TestItemRunner` reexports `@testitem`, so then that is available also in say `test/runtests.jl`.

> [@aplavin](#):
>
> Can it be made to work with top-level `@testset` entries as well?

It thought a fair bit about it, and in the end I decided against it for a couple of reasons:

1. the real work in converting a test suite from `@testset` to `@testitem` is to make sure the `@testitem`s are self-contained, the renaming part seemed very little work to me, literally search-replace should do it.
2. If we start to just pretend that `@testset`s can be run individually and surface that in the VS Code UI, then I can bet that thousands of users will start to use that on their existing test suite and nothing will work. And I think they would rightly say “but hey, my tests are written in the style that Base tells me to”, and then we would have to say “yeah, but if you want to run them inside VS Code, you have to remember…” So I think it would create too much confusion, support requests etc
3. One really nice thing about this syntactic detection story is how absolutely trivial the `@testitem` macro is. The whole system is actually really, really simple. In terms of dev work for me/us that is a major benefit and I didn’t want to complicate things by brittle integration with an existing test framework that is just not a super good match.

> [@phlavenk](#):
>
> All the `@testset`s hierarchy is in the included files - I’ve not touched them. Yes - you can have a much nicer granularity of the `@testset`s in the UI if you replace them in those concrete test files with `@testitem`, but even my example brings already the productivity boost.

Yep, that is an important point! You can still use `@testset`s inside `@testitem`s and slowly and gradually split things into more granular and smaller `@testitem`s over time!

> [@lmiq](#):
>
> When I click “play” on this test, it doesn’t do anything (the thing there keeps spinning forever).
> 
> I don´t see any output anywhere to try to debug what’s going on.

You can click the button in a red circle in this screenshot:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/e/4/e4beb9cb9dcff82ebd4349472c51c98c98eacc71.png)

That should show a terminal with all output from the test run process. Does that show anything informative? Generally what you describe sounds like a bug on our end to me 🙂

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [August 26, 2022, 8:00pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/7 "2022-08-26T20:00:38Z")

</div>

Great stuff! Unfortunately, I wasn’t able to run a first simple test on a remote server with a custom julia executable path setting (it just hangs forever with a spinning circle). Is the latter being respected when spinning up the test processes?

FWIW, the test runner output (obtained by clicking on the small square highlighted in red above) is just empty.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [August 26, 2022, 9:46pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/8 "2022-08-26T21:46:39Z")

</div>

> [@carstenbauer](#):
>
> Unfortunately, I wasn’t able to run a first simple test on a remote server with a custom julia executable path setting (it just hangs forever with a spinning circle). Is the latter being respected when spinning up the test processes?

It should in theory work, but it sounds like it doesn’t. Could you maybe paste the exact setting you have for your executable here? Maybe something jumps out. Or probably even better open an issue on the extension Github for this, clearly a bug.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [August 26, 2022, 9:53pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/9 "2022-08-26T21:53:22Z")

</div>

> [@davidanthoff](#):
>
> That should show a terminal with all output from the test run process. Does that show anything informative?

No, it doesn’t show anything. I’ll try to get more info. Is there any verbose mode or log where I may find anything?

Also I’ll try to build a mwe and then file an issue appropriately.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [August 27, 2022, 2:45pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/10 "2022-08-27T14:45:35Z")

</div>

@davidanthoff

I have the problem of the “play” button not doing anything in this package, which is only the template plus the tests: [GitHub - lmiq/TestTests.jl](https://github.com/lmiq/TestTests.jl)

I’m on Linux x86 - with Julia 1.8.0.

There is no output at all, as the figure above shows, the spinning stuff just keeps spinning.

 ![image](https://global.discourse-cdn.com/julialang/original/3X/0/a/0aaa1b2127939dd65a04e27e869336f8d69e186f.png)

Opened the issue here: [@testitem does not run · Issue #19 · julia-vscode/TestItemRunner.jl · GitHub](https://github.com/julia-vscode/TestItemRunner.jl/issues/19)

To try to reproduce the problem, do:

```julia
git clone https://github.com/lmiq/TestTests.jl
cd TestTests.jl
code .

```

VSCode will open. Wait for the tests to appear at the Erlenmeyer tab, and click play.

---

<div class="post-metadata">

**Author:** ![brianguenter](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/brianguenter/32/29519_2.png) [@brianguenter](https://discourse.julialang.org/u/brianguenter)\
**Post date:** [August 27, 2022, 10:47pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/11 "2022-08-27T22:47:43Z")

</div>

This will be _fantastic_ when all the bugs get ironed out. Thank you for developing this.

Unfortunately, it doesn’t seem to be working for me. Using this MWE

```julia
module Differentiation
using TestItems, TestItemRunner

@testitem "First tests" begin
    @test true
end

end # module 

```

I also get the circle spinning, though not forever. Eventually it crashes:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/c/6/c6a842d6f276b4b7650a7423d9f155e8a2c1df13.png)

System info:

Julia language server v1.7.6

> **VSCode version**
>
> ```julia
> Version: 1.70.2 (user setup)
> Commit: e4503b30fc78200f846c62cf8091b76ff5547662
> Date: 2022-08-16T05:35:13.448Z
> Electron: 18.3.5
> Chromium: 100.0.4896.160
> Node.js: 16.13.2
> V8: 10.0.139.17-electron.0
> OS: Windows_NT x64 10.0.22000
> 
> ```

> **Julia version**
>
> ```julia
> julia> versioninfo()
> Julia Version 1.8.0
> Commit 5544a0fab7 (2022-08-17 13:38 UTC)
> Platform Info:
> OS: Windows (x86_64-w64-mingw32)
> CPU: 16 × 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
> WORD_SIZE: 64
> LIBM: libopenlibm
> LLVM: libLLVM-13.0.1 (ORCJIT, tigerlake)
> Threads: 8 on 16 virtual cores
> Environment:
> JULIA_EDITOR = code.cmd
> JULIA_NUM_THREADS = 8
> 
> ```

---

<div class="post-metadata">

**Author:** ![TI36XPro](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ti36xpro/32/33658_2.png) [@TI36XPro](https://discourse.julialang.org/u/TI36XPro)\
**Post date:** [August 31, 2022, 12:09am UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/12 "2022-08-31T00:09:13Z")

</div>

I am getting the same behavior. Endless spinning with nothing in the terminal output. I am running Ubuntu on Windows Subsystem for Linux 1 (WSL1).

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [August 31, 2022, 1:54am UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/13 "2022-08-31T01:54:54Z")

</div>

The bug is being worked on. A workaround is to change the Julia extension version to “Julia Insiders” _and_ make the path to the executable of Julia point to the actual application in the extension settings (I. E. with the full path, not just “julia”).

That said, the feature is amazing. I am completely into it, even in this experimental scenario. I’m just waiting for the blessing of David to release package versions with the new testing framework.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [September 7, 2022, 6:13pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/14 "2022-09-07T18:13:58Z")

</div>

Alright, I pushed out a new build that should fix the problem of tests never running. Thanks to @lmiq who helped debug this!

If you had tried this before and weren’t able to run tests, please try again with this new build, version 1.7.10.

If things still don’t work for you, please let me know, then you’ve found a new bug! Ideally open a new issue in that case over at [GitHub - julia-vscode/julia-vscode: Julia extension for Visual Studio Code](https://github.com/julia-vscode/julia-vscode).

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [September 7, 2022, 6:50pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/15 "2022-09-07T18:50:56Z")

</div>

For me this is one of the best additions in terms of development workflow and code quality. Having the possibility of adding tests just besides the code and run this tests independently is awesome. Thank you very much for that.

---

<div class="post-metadata">

**Author:** ![brianguenter](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/brianguenter/32/29519_2.png) [@brianguenter](https://discourse.julialang.org/u/brianguenter)\
**Post date:** [September 8, 2022, 2:06am UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/16 "2022-09-08T02:06:37Z")

</div>

I just ran this again and still get the wheel spinning a long time which eventually causes a crash. Same environment as [before](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/11) , but Julia language server is now 1.7.10 and VSCode version is 1.71.0.

---

<div class="post-metadata">

**Author:** ![stephenll](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stephenll/32/28751_2.png) [@stephenll](https://discourse.julialang.org/u/stephenll)\
**Post date:** [September 8, 2022, 2:42am UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/17 "2022-09-08T02:42:42Z")

</div>

I just installed and tried for the first time and receive a crash after waiting for a bit. I’m on Julia 1.8.0, MacOS M1 Pro. I also tried setting the Julia path manually with no success.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [September 8, 2022, 9:05am UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/18 "2022-09-08T09:05:04Z")

</div>

@brianguenter @stephenll

There are now, at the `Output` tab, the option to choose `Julia TestServer`, `Julia`, and `Julia Language Server` logs. If you can post the content of those, that helps debugging the issues.

Like this, for example:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/b/6/b67abbba66b786bc3107c8477b8421226f61d14e.png)

---

<div class="post-metadata">

**Author:** ![stephenll](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stephenll/32/28751_2.png) [@stephenll](https://discourse.julialang.org/u/stephenll)\
**Post date:** [September 8, 2022, 2:48pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/19 "2022-09-08T14:48:53Z")

</div>

Before seeing this, I tried a few more changes to see if I can get this working. Maybe this is something I should know but I’m a very occasional VS Code user: **I had to add Test and TestItems to my package to get this to work.**

It might also be relevant: I don’t have the `Test` package in the package directory I am working on. I add it to a project in the Test directory. To be clear, the Test package exists in a project.toml in the Test directory. To get this to work, I had to add both `Test` and `TestItems` to the pkg directory.

I hope this helps.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [September 8, 2022, 2:54pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/20 "2022-09-08T14:54:40Z")

</div>

As far as I understand, `Test` shouldn’t be needed. `TestItems` yes, if you want tests mixed with the package code. If all tests are in the `runtests.jl` file, then there you should load `TestItemsRunner` (and `Tests`, I’m not completely sure about this one, though).

[Next page](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355.md?page=2)