# Prerelease of new testing framework and test run UI in VS Code

**URL:** <https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355>\
**Category:** VS Code\
**Created:** [August 25, 2022, 7:37pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355 "2022-08-25T19:37:13Z")\
**Posts on this page:** 14\
**Page:** 3

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [September 21, 2022, 1:36am UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/41 "2022-09-21T01:36:36Z")

</div>

> [@lmiq](#):
>
> I’m keeping a submodule in my package called “Testing”, containing functions and variables for testing. The I use this module inside the test items.

That works, but is really not ideal because now you keep that state around in a non-test situation as well, right?

> [@juliohm](#):
>
> I wonder if the Julia extension could do something like that for us automatically with macros.

Yes, I think something like that makes sense. We’ll just have to think hard about a) how is given `@testsetup` linked to a given `@testitem`? Maybe a `@testsetup` should have a name, and then it needs to explicitly be specified in a `@testitem`? Or maybe `@testsetup` can have a flag a la `default` that would then load it into all `@testitem`s. Or something like that 🙂 The second b) question is whether introducing this global shared stated will cause real problems in terms of `@testitem` runs no longer being really truly independent…

---

<div class="post-metadata">

**Author:** ![suavesito](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/suavesito/32/34386_2.png) [@suavesito](https://discourse.julialang.org/u/suavesito)\
**Post date:** [September 21, 2022, 1:48am UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/42 "2022-09-21T01:48:24Z")

</div>

For the first, maybe something along the lines of

```julia
@testsetup "setup1" begin
# setup code
end

@testitem "item1" begin
@setup "setup1"
# test code
end

```

For the second part, maybe you can do more than just importing, maybe you can inline the code, that way every `@testitem` is its separate environment.

If the user do want a module then it could be something along the lines of

```julia
@testsetup "setup1" begin
# setup code
end

@testitem "item1" begin
@setup "setup1" module=true # inlining the code wraps it in a module and does `using`
# test code
end

```

Also, setup code can be run along the test as its individual test.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [September 21, 2022, 8:31am UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/43 "2022-09-21T08:31:38Z")

</div>

> [@davidanthoff](#):
>
> That works, but is really not ideal because now you keep that state around in a non-test situation as well, right?

The module has functions and constants, thus no big deal. What’s not ideal is the dependency management of that module. I can’t have test only dependencies there.

edit: Now I remember doing something like modifying my test functions to receive as parameters the functions of the dependencies, such that the module is not anymore dependent on those. For a small test set with those properties, that’s ok, for larger sets it becomes cumbersome, for sure.

---

<div class="post-metadata">

**Author:** ![lrnv](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lrnv/32/19373_2.png) [@lrnv](https://discourse.julialang.org/u/lrnv)\
**Post date:** [September 27, 2022, 3:34am UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/44 "2022-09-27T03:34:30Z")

</div>

Thanks for the work, this is really nice. It is a shame that the old syntax cannot be conserved and overloaded, but I understand why you did that this way.

What about a piece of template in [PkgTemplate.jl](https://github.com/invenia/PkgTemplates.jl) that will produce directly the infrastructure needed in the package ? This might help people install this correctly on new packages on the fly (with the packages to add to the test environement and not the package environement, etc, already pre-done for you).

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [September 27, 2022, 5:37am UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/45 "2022-09-27T05:37:09Z")

</div>

> [@lrnv](#):
>
> What about a piece of template in [PkgTemplate.jl](https://github.com/invenia/PkgTemplates.jl) that will produce directly the infrastructure needed in the package ?

That would be great! In particular, just adding the `TestItemRunner` to `Project.toml` and the correct default `runtests.jl` would probably go a long way. I have too much on my plate right now to tackle this, but if someone else wants to take the initiative it would be great.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [October 4, 2022, 2:00am UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/46 "2022-10-04T02:00:50Z")

</div>

I just released a new build to the insider channel of the VS Code extension. I added a pretty complete writeup of what was added to my initial post at the top of this thread, see [here](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355#h-1032022-update-6).

---

<div class="post-metadata">

**Author:** ![Daniel\_Gonzalez](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/daniel_gonzalez/32/52412_2.png) [@Daniel\_Gonzalez](https://discourse.julialang.org/u/Daniel_Gonzalez)\
**Post date:** [October 4, 2022, 2:16pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/47 "2022-10-04T14:16:50Z")

</div>

Thank you David, this extension is great! I have one question: if I’m developing a package and a test item fails at the package precompilation stage, is there currently any way to see the “inner” stack trace (the one describing the actual reason for the precompilation failure) in the test output instead of (or in addition to) the “outer” stack trace (the one that just tells you that precompilation failed)?

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [October 4, 2022, 4:54pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/48 "2022-10-04T16:54:06Z")

</div>

> [@Daniel\_Gonzalez](#):
>
> I have one question: if I’m developing a package and a test item fails at the package precompilation stage, is there currently any way to see the “inner” stack trace (the one describing the actual reason for the precompilation failure) in the test output instead of (or in addition to) the “outer” stack trace (the one that just tells you that precompilation failed)?

Ah, excellent point! I’ll try to fix soon, agreed that it would make much more sense to show the inner error in that case.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [October 4, 2022, 8:06pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/49 "2022-10-04T20:06:49Z")

</div>

Actually, turns out that I’m not sure we can fix this… The error we are interested in is happening in a different Julia instance that is spun up just for the precompile, and apparently the error information is not transferred back at all to the calling process, so as far as I understand it right now, we just don’t have access to it ☹

---

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [October 4, 2022, 9:00pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/50 "2022-10-04T21:00:33Z")

</div>

As an alternative to the `@testsetup` suggestion for common test code, how about we make test items nested, and each test item runs all the code from parent test items too.

For example

```julia
@testitem "outer" begin
    setup()
    @testitem "inner1" ...
    @testitem "inner2" ...
end

```

has two “leaf” test items, which each run `setup()` first.

And if you didn’t want to syntactically nest then there could be a `parent` argument so you could instead do

```julia
@testitem "inner1" parent="outer" ...
@testitem "inner2" parent="outer" ...

```

which is approximately the same as the `@testsetup` idea.

Nesting would also be nice just from a GUI point of view, to organise large test suites a bit.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [October 5, 2022, 1:08pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/51 "2022-10-05T13:08:08Z")

</div>

@davidanthoff any chance to include the “global setup” macro in this release already? It is the only missing feature that inhibits us from migrating to the extension in big test suites like the one in Meshes.jl: [Meshes.jl/runtests.jl at master · JuliaGeometry/Meshes.jl · GitHub](https://github.com/JuliaGeometry/Meshes.jl/blob/master/test/runtests.jl)

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [October 5, 2022, 5:28pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/52 "2022-10-05T17:28:58Z")

</div>

I have two design issues where I would welcome feedback, ideally on the linked GitHub issues.

1. Some thoughts on the [code reuse question](https://github.com/julia-vscode/TestItemRunner.jl/issues/11), CC @juliohm
2. Some thoughts on hierarchies, [here](https://github.com/julia-vscode/TestItemRunner.jl/issues/5)

---

<div class="post-metadata">

**Author:** ![brianguenter](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/brianguenter/32/29519_2.png) [@brianguenter](https://discourse.julialang.org/u/brianguenter)\
**Post date:** [February 8, 2023, 7:18pm UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/53 "2023-02-08T19:18:27Z")

</div>

I finally figured out why it appeared that the test framework was running out of date code. For reasons that are still unclear sometimes VSCode takes much longer than it should, minutes instead of a fraction of a second, to save small julia files. I was changing code in the test cases and then running them immediately but VSCode hadn’t saved them yet so the old versions were run.

I believe this is related to file formatting. I have VSCode set to autosave on focus change _and_ to format on save.

Ordinarily formatting and saving the files takes a fraction of a second. But, the longer my REPL session is active the longer formatting and saving takes, until eventually it takes minutes to format and save a 200 line file. Then I have to restart VSCode to get acceptable response.

I posted a question about this [here](https://discourse.julialang.org/t/vscode-taking-a-very-long-time-to-format-saved-file/92531/2) but did not get a resolution.

@davidanthoff do you have any ideas about how I can fix the slow saving problem?

> **Version and OS info**
>
> ```julia
> julia> versioninfo()
> Julia Version 1.8.5
> Commit 17cfb8e65e (2023-01-08 06:45 UTC)
> Platform Info:
> OS: Windows (x86_64-w64-mingw32)
> CPU: 32 × AMD Ryzen 9 7950X 16-Core Processor
> WORD_SIZE: 64
> LIBM: libopenlibm
> LLVM: libLLVM-13.0.1 (ORCJIT, znver3)
> Threads: 32 on 32 virtual cores
> Environment:
> JULIA_EDITOR = code.cmd
> 
> ```

> **VSCode settings**
>
> ```julia
> {
> "terminal.integrated.commandsToSkipShell": [
> "language-julia.interrupt"
> ],
> 
> "editor.fontFamily": "JuliaMono,Consolas,Courier New ,monospace",
> "editor.fontLigatures": "'zero', 'ss02', 'ss04', 'ss05', 'ss08'",
> "julia.symbolCacheDownload": true,
> "julia.enableTelemetry": true,
> "files.autoSave": "onFocusChange",
> "workbench.editor.enablePreview": false,
> "workbench.editor.revealIfOpen": true,
> "git.confirmSync": false,
> "editor.accessibilitySupport": "off",
> "terminal.integrated.tabs.enabled": false,
> "editor.minimap.enabled": false,
> "git.enableSmartCommit": true,
> "editor.wordWrap": "on",
> "editor.defaultFormatter": "julialang.language-julia",
> "editor.formatOnSave": true,
> "julia.NumThreads": "auto",
> "julia.useCustomSysimage": true
> }
> 
> ```

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [February 9, 2023, 6:24am UTC](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/54 "2023-02-09T06:24:23Z")

</div>

Hm, that is very strange. I don’t think it can be related to the length of the REPL session, code formatting should be entirely independent of that…

[Previous page](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355.md?page=2)