# How does the Testing pane work?

**URL:** <https://discourse.julialang.org/t/how-does-the-testing-pane-work/115301>\
**Category:** VS Code\
**Created:** [June 6, 2024, 9:13pm UTC](https://discourse.julialang.org/t/how-does-the-testing-pane-work/115301 "2024-06-06T21:13:25Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ken\_Williams](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ken_williams/32/5289_2.png) [@Ken\_Williams](https://discourse.julialang.org/u/Ken_Williams)\
**Post date:** [June 6, 2024, 9:13pm UTC](https://discourse.julialang.org/t/how-does-the-testing-pane-work/115301/1 "2024-06-06T21:13:25Z")

</div>

I haven’t been able to find a reference about how the Testing pane in VS Code works for a Julia project’s unit tests. I’ve fiddled around a bit and tried various of

- file layouts (`test_*.jl` vs. `*_test.jl`, putting in a `MyTestModule`, etc.)
- runners (`TestItemRunner`, `ReTest`, `ReTestItems`, etc.)
- macros (`@testset` vs. `@testitem`)
- setup code (add stuff in `runtests.jl`, put it in a file that you `include()` in each test, use some runner-specific thing like `@testsetup`, etc.)

and I’ve converged on something that works sort of okay, but I feel like it should be documented somewhere what the Testing pane actually does, and how it finds its tests (is it finding them itself, or are the various runner frameworks feeding it the test info) and what happens when you click the “run” buttons.

Of course, feel free to point me to docs if they exist - I looked in [Home · Julia in VS Code](https://www.julia-vscode.org/docs/latest/) .

Thanks!

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [June 6, 2024, 9:31pm UTC](https://discourse.julialang.org/t/how-does-the-testing-pane-work/115301/2 "2024-06-06T21:31:44Z")

</div>

> [@Prerelease of new testing framework and test run UI in VS Code](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355):
>
> I just pushed out a new release of the Julia VS Code extension (1.7.6) to everyone that includes a preview of our new test UI and a general new testing framework for Julia. For a short demo, take a look at our Juliacon talk [https://youtu.be/Okn\_HKihWn8?t=1268](https://youtu.be/Okn_HKihWn8?t=1268). This new set of features consists of two new packages ([TestItems.jl](https://github.com/julia-vscode/TestItems.jl) and [TestItemRunner.jl](https://github.com/julia-vscode/TestItemRunner.jl)) and the new UI in the Julia VS Code extension. A good way to refer to this new test framework is by the “test item” name. How to write test items …

---

<div class="post-metadata">

**Author:** ![Ken\_Williams](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ken_williams/32/5289_2.png) [@Ken\_Williams](https://discourse.julialang.org/u/Ken_Williams)\
**Post date:** [June 6, 2024, 9:42pm UTC](https://discourse.julialang.org/t/how-does-the-testing-pane-work/115301/3 "2024-06-06T21:42:19Z")

</div>

I’ve seen `TestItemRunner` and tried it but couldn’t get it to work with the GUI. That’s one reason I’m asking what the Testing pane is actually doing, so I can see if I’m complying with its expectations.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [June 6, 2024, 9:44pm UTC](https://discourse.julialang.org/t/how-does-the-testing-pane-work/115301/4 "2024-06-06T21:44:25Z")

</div>

The testing pane will only show `@testitem`s, i.e. the stuff I described in the linked post.

Can you describe what didn’t work when you tried it? And maybe show a code example?

---

<div class="post-metadata">

**Author:** ![Ken\_Williams](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ken_williams/32/5289_2.png) [@Ken\_Williams](https://discourse.julialang.org/u/Ken_Williams)\
**Post date:** [June 6, 2024, 9:57pm UTC](https://discourse.julialang.org/t/how-does-the-testing-pane-work/115301/5 "2024-06-06T21:57:24Z")

</div>

It’s a little hard for me to remember everything that wasn’t working, because I was flailing around trying to get things working, trying things kind of at random. And unfortunately this is a proprietary codebase so it would take some time to distill it down to something I could share.

But I think the main things I was having trouble with were

- Couldn’t get tests to appear in the Testing pane
- Couldn’t get initialization code to run (I commented on [Doc for `testsetup` and cross-module testing by epolack · Pull Request #57 · julia-vscode/TestItemRunner.jl · GitHub](https://github.com/julia-vscode/TestItemRunner.jl/pull/57#issuecomment-2148251174))
- Different results (e.g. different working directory) running from Testing pane vs. `]test` in the REPL
- Initialization code seems to only support code in a module, so `using` statements don’t propagate to test code

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [June 6, 2024, 10:29pm UTC](https://discourse.julialang.org/t/how-does-the-testing-pane-work/115301/6 "2024-06-06T22:29:32Z")

</div>

Take a look at this post: [Prerelease of new testing framework and test run UI in VS Code - #22 by lmiq](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/22)

With that step by step you should get things working. There are some things there that are not needed (`using package` inside each test item), but do no harm.

---

<div class="post-metadata">

**Author:** ![Ken\_Williams](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ken_williams/32/5289_2.png) [@Ken\_Williams](https://discourse.julialang.org/u/Ken_Williams)\
**Post date:** [June 6, 2024, 10:31pm UTC](https://discourse.julialang.org/t/how-does-the-testing-pane-work/115301/7 "2024-06-06T22:31:33Z")

</div>

Another thing that confused me is that (because I had read @davidanthoff 's post) I thought that the only way to get the Testing pane to show tests (and be able to run them individually, etc.) was to start using the “test items” stuff.

But then yesterday a co-worker showed me his setup where he’s using `ReTestItems` and that populates the Testing pane (but only if you name your files a certain way…), so I experimented with that a bit today and verified that I can get it working. But I’m left a bit confused about what’s going on. For example - when using `ReTestItems`, does my `runtests.jl` enter into the picture, or is it just running tests directly?

And what’s all this I see in the console output about a “test server instance”? Is that explained anywhere I could read about? I wouldn’t really need to know much about it, except that it sometimes seems to run tests from the root directory `/`, causing them to fail when loading files using relative paths.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [June 6, 2024, 10:52pm UTC](https://discourse.julialang.org/t/how-does-the-testing-pane-work/115301/8 "2024-06-06T22:52:45Z")

</div>

> [@Ken\_Williams](#):
>
> Couldn’t get tests to appear in the Testing pane

If you could reproduce that it would be great! At the moment I’m not aware of any bugs in the test detection logic.

> [@Ken\_Williams](#):
>
> Couldn’t get initialization code to run

Yes, that feature is just not finished. For now, the only real option to share code between different `@testitem`s is to put it into a separate file, and then `include` that file from each `@testitem`.

> [@Ken\_Williams](#):
>
> Different results (e.g. different working directory) running from Testing pane vs. `]test` in the REPL

Hm, I think that is a bug, just looked at the code. Fix incoming.

> [@Ken\_Williams](#):
>
> Initialization code seems to only support code in a module, so `using` statements don’t propagate to test code

I don’t understand that, could you expand a bit?

> [@Ken\_Williams](#):
>
> But then yesterday a co-worker showed me his setup where he’s using `ReTestItems` and that populates the Testing pane (but only if you name your files a certain way…)

So `ReTestItems` is an independent test runner developed by Relational AI, that also happens to run `@testitem`s. Think of it as an alternative to `TestItemRunner`. Now, `ReTestItem` has _slightly_ different features than what I introduced as the “test item framework” originally… Some extra ones and some things are just a bit different relative to `TestItemRunner` (for example you have to put tests into files with a specific filename pattern). I think it would probably have been better to just give it a different name completely distinct from “test item” to make it clear that it is something different, but hey, here we are 🙂

As far as I know `ReTestItems` has no VS Code integration at all. So whatever you are seeing showing up in the testing UI in VS Code is just the official “test item” feature in VS Code.

> [@Ken\_Williams](#):
>
> And what’s all this I see in the console output about a “test server instance”?

That is an implementation detail, it is just the name of the Julia process in which the test code is running. Re working directory, see my response above.

---

<div class="post-metadata">

**Author:** ![Ken\_Williams](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ken_williams/32/5289_2.png) [@Ken\_Williams](https://discourse.julialang.org/u/Ken_Williams)\
**Post date:** [June 19, 2024, 9:49pm UTC](https://discourse.julialang.org/t/how-does-the-testing-pane-work/115301/9 "2024-06-19T21:49:19Z")

</div>

> [@davidanthoff](#):
>
> > [@Ken\_Williams](#):
> >
> > Couldn’t get tests to appear in the Testing pane
> 
> If you could reproduce that it would be great! At the moment I’m not aware of any bugs in the test detection logic.

Well, I’m not sure if it was a bug or not - since there didn’t seem to be any docs for the Testing Pane I couldn’t tell what was supposed to happen, if I was doing anything wrong or not. I’ve reorganized my tests a bunch and now it seems to be behaving well for the past couple weeks, so it must have been on my end.

> [@davidanthoff](#):
>
> For now, the only real option to share code between different `@testitem`s is to put it into a separate file, and then `include` that file from each `@testitem`.

That’s what I’ve switched to, it seems to be working fine, thanks.

> [@davidanthoff](#):
>
> > [@Ken\_Williams](#):
> >
> > Different results (e.g. different working directory) running from Testing pane vs. `]test` in the REPL
> 
> Hm, I think that is a bug, just looked at the code. Fix incoming.

Has that fix been released? Would it be a fix in the VS Code extension? I don’t see it in the Changelog but I’m not sure if that’s kept up to date.

> [@davidanthoff](#):
>
> > [@Ken\_Williams](#):
> >
> > Initialization code seems to only support code in a module, so `using` statements don’t propagate to test code
> 
> I don’t understand that, could you expand a bit?

It might be a moot point until the initialization stuff is ironed out more. But what I meant was that since the mechanism for initialization code (`@testsetup`) is to put code in a module, any `using` statements in that module won’t transitively export their symbols to the test code. So while I can stick `using Foo` in a file that’s `include`d in a test and that gives me access to `Foo.*` symbols, that’s not true for stuff in a `@testsetup` module and that makes it a bit less useful.

> [@davidanthoff](#):
>
> > [@Ken\_Williams](#):
> >
> > And what’s all this I see in the console output about a “test server instance”?
> 
> That is an implementation detail, it is just the name of the Julia process in which the test code is running.

I think it needs to be mentioned - since it wasn’t mentioned anywhere, I didn’t know that it ran using a persistent server, and in parallel, and was therefore an order of magnitude faster than using `] test`, so I didn’t know that I should try to get my tests visible in that pane.

It’s also an important fact to understand because sometimes the server seems to get out of sync with the code being tested, and things need to be torn down & rebuilt to get fixed. The `Developer: Reload Window` action has been useful for this, not sure if there’s a better way.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [June 19, 2024, 10:12pm UTC](https://discourse.julialang.org/t/how-does-the-testing-pane-work/115301/10 "2024-06-19T22:12:50Z")

</div>

> [@Ken\_Williams](#):
>
> sometimes the server seems to get out of sync with the code being tested, and things need to be torn down & rebuilt to get fixed.

There is one known problem where we can get out of sync (namely updates to project or manifest files, I have a branch where that is fixed, but it will take a while to get that really ready), other than that there should never be a need to manually tear or restart anything, that should all happen automatically without any user intervention.

---

<div class="post-metadata">

**Author:** ![ianfiske](https://avatars.discourse-cdn.com/v4/letter/i/58f4c7/32.png) [@ianfiske](https://discourse.julialang.org/u/ianfiske)\
**Post date:** [June 20, 2024, 3:44pm UTC](https://discourse.julialang.org/t/how-does-the-testing-pane-work/115301/11 "2024-06-20T15:44:17Z")

</div>

> [@Ken\_Williams](#):
>
> It might be a moot point until the initialization stuff is ironed out more. But what I meant was that since the mechanism for initialization code (`@testsetup`) is to put code in a module, any `using` statements in that module won’t transitively export their symbols to the test code. So while I can stick `using Foo` in a file that’s `include`d in a test and that gives me access to `Foo.*` symbols, that’s not true for stuff in a `@testsetup` module and that makes it a bit less useful.

For this one, you can `export` individual symbols needed from the `testsetup` modules. If you want to re-export entire packages, there is Reexport.jl. I haven’t tested this.

---

<div class="post-metadata">

**Author:** ![Ken\_Williams](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ken_williams/32/5289_2.png) [@Ken\_Williams](https://discourse.julialang.org/u/Ken_Williams)\
**Post date:** [June 20, 2024, 5:23pm UTC](https://discourse.julialang.org/t/how-does-the-testing-pane-work/115301/12 "2024-06-20T17:23:33Z")

</div>

> [@ianfiske](#):
>
> you can `export` individual symbols needed from the `testsetup` modules. If you want to re-export entire packages, there is Reexport.jl. I haven’t tested this.

Oh nice, it looks like that would work well for this, thanks for pointing it out.

---

<div class="post-metadata">

**Author:** ![Ken\_Williams](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ken_williams/32/5289_2.png) [@Ken\_Williams](https://discourse.julialang.org/u/Ken_Williams)\
**Post date:** [June 24, 2024, 6:45pm UTC](https://discourse.julialang.org/t/how-does-the-testing-pane-work/115301/13 "2024-06-24T18:45:49Z")

</div>

> [@davidanthoff](#):
>
> Hm, I think that is a bug, just looked at the code. Fix incoming.

Is this fix (wrong `pwd` while running tests) likely to be released soon? I’ve been keeping an ad-hoc `cd(...)` in my Git stash for a while, but if it’s going to be a while for the fix, I’ll probably need to come up with a better workaround that the whole team can use.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [June 24, 2024, 8:10pm UTC](https://discourse.julialang.org/t/how-does-the-testing-pane-work/115301/14 "2024-06-24T20:10:13Z")

</div>

It might take a while, I’m on vacation 🙂 Having said that, I would generally try to write tests in a way that they don’t depend on the working directory. Using things like `@ __DIR__ ` and `@ __FILE__ ` is good practice for tests.