# New stable features for the test item framework

**URL:** <https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512>\
**Category:** VS Code\
**Tags:** announcement\
**Created:** [August 23, 2024, 1:51am UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512 "2024-08-23T01:51:54Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [August 23, 2024, 1:51am UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512/1 "2024-08-23T01:51:54Z")

</div>

I released a couple of new stable features for the test item framework in the last weeks. This post will describe them.

But before I dive into the new features, a quick recap what the test item framework is! The main benefit of the test item framework is that you can very easily split your tests into self-contained small test items, and then run them individually on demand. You can read my previous post about the framework [here](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355), watch a short (outdated!) YouTube demo [here](https://youtu.be/Okn_HKihWn8?t=1268) or take a look at the documentation [here](https://www.julia-vscode.org/docs/stable/userguide/testitems/).

I should also note that the framework is no longer in preview mode. It has been used heavily for a number of packages and has been production ready for many years now. You can and should use it for your “real” projects!

Ok, and with that, here are the new features:

#### Sharing code across `@testitem`s

By default `@testitem`s do not share any code between each other and have no dependencies between each other. These properties make it feasible to run `@testitem`s by themselves, but sometimes one wants to share common code between multiple `@testitem`s. The test item framework provides two macros for this purpse: `@testsnippet` and `@testmodule`. These two macros can appear in any `.jl` file in a package.

##### Test snippets

A `@testsnippet` is a block of code that individual `@testitem`s can run before their own code runs. If a `@testitem` takes a dependency on a particular `@testsnippet`, that snippet will run every time the `@testitem` runs.

The definition of a `@testsnippet` might look like this

```julia
@testsnippet MySnippet begin
    foo = "Hello world"
end

```

A `@testitem` can utilize this snippet by using the `setup` keyword like this:

```julia
@testitem "My test item" setup=[MySnippet] begin
    @test foo == "Hello world"
end

```

##### Test modules

A `@testmodule` defines a Julia module that can be accessed from `@testitem`s. Such a module will only be run _once_ per Julia test process. If for example two `@testitem`s depend on a `@testmodule`, it will only be evaluated once, and then the entire module will be made available to both `@testitem`s.

The definition of a `@testmodule` might look like this

```julia
@testmodule MyModule begin
    foo = "Hello world"
end

```

A `@testitem` can utilize this module by again using the `setup` keyword. Unlike with `@testsnippet`s, the content of a `@testmodule` is run inside a regular Julia `module`, so to access content inside there one needs to prefix the module name to any name defined in the test module. A `@testitem` that utilizes the `@testmodule` just defined might look like this:

```julia
@testitem "My test item" setup=[MyModule] begin
    @test MyModule.foo == "Hello world"
end

```

Note how we access `foo` with the expression `MyModule.foo` here.

#### Debugging of `@testitem`s

`@testitem`s can be run in the debugger by launching them via the `Debug Test` command. This command can be access in various places in the VS Code UI. In the test main testing view it is available here:

 ![](https://global.discourse-cdn.com/julialang/original/3X/3/0/30feebbf4a44577b215a592f92c2b35e733c959d.png)

One can also right click on the run test icon in the text editor to select the debug option:

 ![](https://global.discourse-cdn.com/julialang/original/3X/f/3/f39ece0da7206f2573bfc850b321cc2881f4ef7a.png)

When a test item is run in the debugger, one can set breakpoints both in the code that is being tested or in the `@testitem` itself and then utilize all the regular features of the Julia VS Code debugger.

#### Code coverage

On Julia 1.11 and newer one can run test items in a code coverage mode and display code coverage results directly in VS Code.

To run test items in code coverage mode one launches them with the command `Run Tests with Coverage`. This command is availble both in the main testing view

 ![](https://global.discourse-cdn.com/julialang/original/3X/1/1/111b9336b092ff58a7c9142a39c2f4b5c17e6c99.png)

as well as in the context menu in the text editor:

 ![](https://global.discourse-cdn.com/julialang/original/3X/e/0/e0170da6ab04f757500393b529b6ac8263da83c2.png)

The coverage results are then displayed in various ways in the VS Code UI. For example a summary view shows coverage per file:

![](https://global.discourse-cdn.com/julialang/original/3X/8/5/85e0bf04783cebdc0f69514b39acc967384c4d66.png)

One can see detailed line coverage information inside the text editor:

 ![](https://global.discourse-cdn.com/julialang/original/3X/6/5/6549d2d0ef7edf99307a1d6d8010b89dbb305234.png)

Coverage results are also displayed inline in the regular explorer part of the VS Code UI.

#### Documentation

I wrote documentation 🙂 For the whole thing, mostly copy paste from this and past discourse announcements, you can find it [here](https://www.julia-vscode.org/docs/stable/userguide/testitems/).

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [August 23, 2024, 2:21am UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512/2 "2024-08-23T02:21:45Z")

</div>

Super exciting. Thank you for working on this 👏🏼

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [August 23, 2024, 3:27am UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512/3 "2024-08-23T03:27:03Z")

</div>

Awesome stuff!

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [August 23, 2024, 6:10pm UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512/4 "2024-08-23T18:10:05Z")

</div>

Oh, and I forgot to mention one other new thing: I fixed a bug where in the past the test runner system would not pick up any changes to your project files. The only solution around that was to restart VS Code entirely. That should be fixed now, i.e. any edits to project files should be picked up automatically and shouldn’t require any kind of restart.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [August 23, 2024, 6:58pm UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512/5 "2024-08-23T18:58:06Z")

</div>

@davidanthoff is there a solution for when a test suite needs a setup that is a function of a variable?

Suppose we have `setup(T)` where `T` can be either `Float32` or `Float64`. We want to run tests in both setups:

```julia
T = Float32
for testfile in testfiles
  include(testfile)
end

T = Float64
for testfile in testfiles
  include(testfile)
end

```

where each `testfile` is full of

```julia
@testitem "basic" setup=[setup(T)] begin
  # tests go here
end

```

---

<div class="post-metadata">

**Author:** ![kellertuer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kellertuer/32/220707_2.png) [@kellertuer](https://discourse.julialang.org/u/kellertuer)\
**Post date:** [August 24, 2024, 5:29am UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512/6 "2024-08-24T05:29:47Z")

</div>

Woah! I think I really have to try this on one of my packages and “move over” to this from my dull plain tests; especially the sharing of code and the coverage looks awesome!

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [August 24, 2024, 5:28pm UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512/7 "2024-08-24T17:28:10Z")

</div>

> [@juliohm](#):
>
> is there a solution for when a test suite needs a setup that is a function of a variable?

So you can’t parameterize the code that gets included, but you can of course for example define functions in both `@testsnippet`s and in `@testmodule`s that take arguments and then call them. Would that do the trick?

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [August 24, 2024, 5:55pm UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512/8 "2024-08-24T17:55:27Z")

</div>

We managed to update to TestItemRunners.jl in Meshes.jl using GitHub actions with different env variables to vary the settings. It is working nicely! ❤

---

<div class="post-metadata">

**Author:** ![visr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/visr/32/17204_2.png) [@visr](https://discourse.julialang.org/u/visr)\
**Post date:** [August 24, 2024, 7:50pm UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512/9 "2024-08-24T19:50:32Z")

</div>

These are great new features, I especially appreciate the test item debugging.

I have been using a `@testitem` macro for a while, not from TestItems.jl but ReTestItems.jl. These macros seem mostly compatible, so I can still use the VS Code test UI. My main motivation for using ReTestItems.jl was running testitems in parallel on multiple worker processes. Does TestItemRunner.jl support parallel execution? I don’t see it in the README, though the julia-vscode docs do mention it [here](https://www.julia-vscode.org/docs/stable/userguide/testitems/#Parallel-test-execution-in-VS-Code).

I could only find this issue about unification [Figure out how to unify with VS Code testitems · Issue #18 · JuliaTesting/ReTestItems.jl · GitHub](https://github.com/JuliaTesting/ReTestItems.jl/issues/18) but it’s labeled speculative.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [August 24, 2024, 9:02pm UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512/10 "2024-08-24T21:02:39Z")

</div>

> [@davidanthoff](#):
>
> On Julia 1.11 and newer one can run test items in a code coverage mode and display code coverage results directly in VS Code.

Haven’t tried it yet because testitem coverage will only be supported for the next Julia version, but wonder:

- How coverage works when running one testitem at a time? And when rerunning a single testitem after changing it?
- How does it compare to the `Run Test task -> Run tests with coverage` feature that has been present for years already? Is it the same but for individual testitems, or something different?

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [August 24, 2024, 11:17pm UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512/11 "2024-08-24T23:17:51Z")

</div>

> [@visr](#):
>
> Does [TestItemRunner.jl](https://juliahub.com/ui/Packages/General/TestItemRunner) support parallel execution?

It’s a somewhat complicated answer 😉

TestItemRunner.jl does _not_ support parallel execution, and most likely never will.

The VS Code extension has supported parallel execution for a long time. It does speed things up relative to sequential, _but_ I should also note that the algorithm that is shipping right now in the extension is not the most efficient one. I’m in the process of implementing something better there right now, it makes a _huge_ difference. No promise when that will ship 🙂

There is a _very_ experimental [TestItemRunner2.jl](https://github.com/julia-vscode/TestItemRunner2.jl) that supports parallel execution. But be warned, that package will go away over time and be replaced with something different. It is _not_ part of the stuff that I consider “production ready”. But there will be a non-UI way to run things in parallel hopefully soon in that spirit.

You can also take a sneak-peek at [GitHub - julia-vscode/testitem-workflow](https://github.com/julia-vscode/testitem-workflow). Also not ready for prime-time, but it already provides parallel test execution on GitHub workflows (and many more benefits!). Will get its own post soon.

I don’t really know what the plans are for ReTestItems.jl, but my sense is that it has diverged from the original test item framework, it is probably best seen as a fork that is going in a different direction. Probably would have been better to give it a different name to avoid confusion 🙂

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [August 24, 2024, 11:19pm UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512/12 "2024-08-24T23:19:38Z")

</div>

> [@aplavin](#):
>
> How coverage works when running one testitem at a time? And when rerunning a single testitem after changing it?

Julia 1.11 adds an API to clear collected coverage data during runtime. So we clear the entire coverage data before each test run, and then only collect new coverage data.

I’m not 100% sure how reliable that really is. There is probably some stuff like constant propagation etc where this won’t work a 100%, so we might have to tweak a bit as we test this in the wild.

> [@aplavin](#):
>
> How does it compare to the `Run Test task -> Run tests with coverage` feature

Completely different, the two implementations share nothing.

---

<div class="post-metadata">

**Author:** ![disberd](https://avatars.discourse-cdn.com/v4/letter/d/8edcca/32.png) [@disberd](https://discourse.julialang.org/u/disberd)\
**Post date:** [August 25, 2024, 6:40am UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512/13 "2024-08-25T06:40:05Z")

</div>

> [@davidanthoff](#):
>
> So we clear the entire coverage data before each test run, and then only collect new coverage data.

Hi @davidanthoff, is this the actual cause of this issue?

> <https://github.com/julia-vscode/julia-vscode/issues/3685>
>
> I have been trying the new test coverage features and it seems extremely cool, k…udos for making this work!
> 
> I have noticed that in my tests so far, the coverage results is only shown for the last \`@testitem\` that was ran.
> Is this something that can be tweaked via a setting, or is it a bug that I am experiencing?
> 
> Edit: To clarify, this also happens when I click the button to run all tests with coverage, not only when running test items manually one by one

And is there then a way to see total coverage for a package testsuite without having to put everything in a single `@testitem`?

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [August 25, 2024, 1:20pm UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512/14 "2024-08-25T13:20:36Z")

</div>

> [@davidanthoff](#):
>
> > [@aplavin](#):
> >
> > How does it compare to the `Run Test task -> Run tests with coverage` feature
> 
> Completely different, the two implementations share nothing.

Interesting! I like the `Run tests with coverage` – works with any Julia version, with testitems or with regular testsets, shows coverage for the full testsuite (re @disberd’s question).  
That’ll remain in the extension, right?

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [August 25, 2024, 3:59pm UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512/15 "2024-08-25T15:59:31Z")

</div>

> [@disberd](#):
>
> And is there then a way to see total coverage for a package testsuite without having to put everything in a single `@testitem`?

So, coverage is always per test run. Every time you click the “Run” button, it starts a new test run. If you run test items individually, then you start a new test run every time, and coverage will be for each run and only include the results for that one test item. But when you run multiple test items in one test run, then coverage for all of them should be aggregated already. You can run multiple test items in one test run via the UI in the Testing pane, either by clicking on the button “Run tests” at the top, or by clicking the run tests button on any folder in the test tree. If coverage isn’t aggregated in those cases, then it would be a bug (but I think it is).

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [August 25, 2024, 4:02pm UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512/16 "2024-08-25T16:02:45Z")

</div>

> [@aplavin](#):
>
> I like the `Run tests with coverage` – works with any Julia version, with testitems or with regular testsets, shows coverage for the full testsuite (re @disberd’s question).  
> That’ll remain in the extension, right?

Yes, no plans to remove that. A few other random points on that:

- the test item framework also works with any Julia version starting at 1.0
- we should actually probably add a hook that if a package is using the test item framework, that these old commands trigger runs in the new framework…
- I think we could actually also integrate the results of these old-style coverage runs with the new coverage UI in VS Code. If someone wants to tackle that, join the `vscode-dev` channel on Slack and ask for help!

---

<div class="post-metadata">

**Author:** ![disberd](https://avatars.discourse-cdn.com/v4/letter/d/8edcca/32.png) [@disberd](https://discourse.julialang.org/u/disberd)\
**Post date:** [August 25, 2024, 5:34pm UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512/17 "2024-08-25T17:34:11Z")

</div>

Thanks for the reply, so it must be a bug as I found no way to aggregate across testitems within a single run, but we can continue the discussion on the linked issue

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [August 25, 2024, 8:58pm UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512/18 "2024-08-25T20:58:07Z")

</div>

> [@davidanthoff](#):
>
> the test item framework also works with any Julia version starting at 1.0

But not coverage, right? When trying to run it on latest released versions of everything, I get the message that only 1.11 is supported.

> [@davidanthoff](#):
>
> I think we could actually also integrate the results of these old-style coverage runs with the new coverage UI in VS Code

Oh, there’s another coverage UI in VSCode? The `Run tests with coverage` function plays nicely with the “Coverage Gutters” extension that supports basically any language (through lcov files).

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [August 25, 2024, 9:36pm UTC](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512/19 "2024-08-25T21:36:56Z")

</div>

> [@aplavin](#):
>
> But not coverage, right? When trying to run it on latest released versions of everything, I get the message that only 1.11 is supported.

Ah, yes, coverage in VS Code for test items only works on Julia 1.11 and newer. Coverage on GitHub for test items works for any Julia version.

> [@aplavin](#):
>
> Oh, there’s another coverage UI in VSCode? The `Run tests with coverage` function plays nicely with the “Coverage Gutters” extension that supports basically any language (through lcov files).

Yes, VS Code added a native coverage UI/API a couple of releases ago. I’m using that already for showing the test item coverage, and it should be pretty simple to actually also utilize that API for the `Run tests with coverage` task that has been around forever. Main benefit would just be that one could see coverage without having to install another extension.