Prerelease of new testing framework and test run UI in VS Code

I just pushed out a new release of the Julia VS Code extension (1.7.6) to everyone that includes a preview of our new test UI and a general new testing framework for Julia.

For a short demo, take a look at our Juliacon talk Julia in VS Code - What's New | David Anthoff, Sebastian Pfitzner | JuliaCon 2022 - YouTube.

This new set of features consists of two new packages (TestItems.jl and TestItemRunner.jl) and the new UI in the Julia VS Code extension. A good way to refer to this new test framework is by the “test item” name.

How to write test items

The core feature in this new framework is that you can structure tests into @testitem blocks and then individually run those, rather than having to run all your tests at once. A typical @testitem might look like this:

@testitem "First tests" begin
    x = foo("bar")

    @test length(x)==3
    @test x == "bar"
end

A @testitem always has a name (here “First tests”) and then some code in a begin ... end block. The code inside a @testitem must be executable by itself, i.e. it can not depend on code that appears outside of the @testitem, unless that code is somehow explicitly imported or included from within the @testitem. There is one exception to this: the code inside the @testitem will run inside a temporary module where using Test and using MYPACKAGENAME was already executed, so anything exported from either the Test module or the package your are developing can be directly used. In the example above this applies to the foo function (presumably defined in the package that is being tested) and the @test macro.

@testitems can appear anywhere in a package. They do not have to be in the test folder, nor do they have to be in a file that is included by test/runtests.jl. In fact, @testitems can even be located inside your regular package code, for example next to the code they are testing. In that case you just need to take a dependency on the TestItems.jl package so that you have access to the @testitem macro. If you have a package MyPackage, then the file src/MyPackage.jl could look like this:

module MyPackage

using TestItems

export foo

foo(x) = x

@testitem "First tests" begin
    x = foo("bar")

    @test length(x)==3
    @test x == "bar"
end

end

If you don’t like this inline @testitem style, you can also just put @testitem blocks into Julia files in your test folder.

Running test items inside VS Code

When you open a Julia package inside VS Code and have the Julia extension installed it will constantly (after every keypress!) look for any and all @testitems in your Julia files. If any are found, they will appear in various places in the UI.

You can find all detected @testitems in the Testing activity bar in VS Code:

image

The testing activity area then provides you with options to run individual @testitems, look at results etc.

VS Code will also place a small little run button next to each detected @testitem in the text editor itself:

image

In addition to all these UI elements that allow you to run tests, there is also UI to display test results. For example, when you run tests and some of them fail, the extension will collect all these test failures and then display them in a structured way, directly at the place in the code where a specific test failed:

Especially when you run a lot of tests with large test files this makes it much easier to find the specific test that failed, no more hunting in the REPL for file and line information!

Running tests from the command line

This part is a little less fleshed out, but you can use the TestItemRunner.jl package to run @testitems as part of a traditional Pkg.test workflow. This makes it easy to integrate these new types of tests with for example a continuous integration setup.

To enable integration with Pkg.test for a package that uses @testitem, you just have to do two things:

  1. Add TestItemRunner.jl as a test dependency to your package
  2. Put the following code into the package’s test/runtests.jl file:
using TestItemRunner

@run_package_tests

I hope that in the future we can make the TestItemRunner.jl package much more feature complete, for example add the ability to only run a subset of @testitems (as you can already do in VS Code), add support for parallel execution etc. Help is most welcome!

Under the hood

We already have some great testing packages in Julialand, and some of them (like the excellent ReTest.jl) seem to provide a lot of similar functionality already, so why create yet another testing framework?

The core reason is that we have a very different requirement for the VS Code extension: we need to detect test items at every keystroke. To do this, I used a completely different test detection strategy than any of the existing testing frameworks. In the test item design, no user code is run at all for test item detection. Test item detection is purely done by syntactic analysis of Julia source files. This kind of approach integrates really well with the existing analysis we have in the LanguageServer.jl that powers the rest of the Julia extension.

One nice side benefit is that the design of this test item framework is quite simple. For example, the definition of the @testitem macro is almost hilariously simple, you can see it here. Having it so simple and not doing anything is ideal, because that means that adding inline tests into the actual package code should really not add any runtime overhead at all to a package (it does probably add a tiny bit of precompile time).

I also want to point out some more details about the test runner in VS Code. The design there is that we first detect in which environment a given @testitem should run, then we spin up a test process per individual environment. These test processes are long run, i.e. once they have started they are reused whenever you execute another new @testitem. By reusing these processes, we can cut out a huge amount of delay between running tests, rerunning an individual test item is often completely instantaneous. The Julia extension now also ships Revise.jl as part of the extension, and uses that to detect any code changes you make to the package code you are testing. The integration with Revise.jl is completely under the hood and automatic. For example, if you make a change to your package that Revise cannot track (like redefining a struct), the extension will recognize that and automatically restart the test process instead of relying on Revise to handle this update. The net effect of that is that you can just freely edit the code you are testing and the test code itself, rerun small parts of it all the time and there should be minimal delays in all of that.

Roadmap

While this feature ships in the regular Julia VS Code extension, we are declaring it a preview at the moment. We want to collect feedback on the design and usability for a while, potentially address design issues that might crop up, and only then will we declare it stable and released.

So, please try this out and let us know what you think! And if some folks want to help improve TestItemRunner.jl, that would be especially fantastic.

68 Likes

Hi David,
I’ve just tried the new test framework functionality and it is incredible - finally, I don’t envy my colleagues that use rust with all the fantastic and real-time test functionality.

Really minimal changes to a package bring you an incredible boost of efficiency:
(EMProject) pkg> add TestItemRunner

# (test) pkg> add TestItems TestItemRunner

# runtest.jl:
using TestItems, TestItemRunner

@testitem "VisualLogging" begin
    include("test_visuallogging.jl")
end
...
@testitem "Lestrade" begin
    include("test_lestrade.jl")
end

@run_package_tests

and that are the only changes to the runtest.jl file. All the testing functionality of Base.Test remains intact, however, immediately the new integrated test framework kicks in and re-running your test after a change in code takes seconds - no environment setup, no more wasted minutes for CairoMakie to compile and initialize every time you need to verify your small code change didn’t brake the tests.
And it is so rewarding, to write the test first, and then just see the red light on the given test item turn green…

Thank you again for the amazing work.

7 Likes

This test functionality looks great, looking forward to using it!

The requirement to replace @testset with @testitem does bring some friction, though. Together with the two extra dependencies. I imagine this would significantly slow down the potential adoption among existing packages.
Can it be made to work with top-level @testset entries as well? Let it assume that individual @testsets are independent, aside from imports at the top of runtests.jl.

In my example, I’ve replaced code

# runtest.jl:
include("test_visuallogging.jl")
...
include("test_lestrade.jl")

by the code above. All the @testsets hierarchy is in the included files - I’ve not touched them. Yes - you can have a much nicer granularity of the @testsets in the UI if you replace them in those concrete test files with @testitem, but even my example brings already the productivity boost.

And the dependencies TestItems, TestItemRunner are only in the project of the test environment, so no change of the dependencies of the package are nescessary.

3 Likes

This is amazing. But I’m having some trouble here. When I click “play” on this test, it doesn’t do anything (the thing there keeps spinning forever). Even if the test does nothing:

image

I don´t see any output anywhere to try to debug what’s going on.

And if I run ] test it complains SVector not defined (reason why I put the using StaticArrays inside the @testitem, but it didn’t work. Solved that.

I should probably clarify this a bit: I think users will really only have to deal with the TestItems package if they want to write tests inline with their normal package code. In that case you have to add TestItems as a regular package dependency so that the @testitem macro is available inside your regular package code. But, if you do not plan to use these inline tests, then you don’t need the dependency on TestItems at all.

TestItemRunner is the package you need as a test dependency, and is essentially only used inside test/runtests.jl. When converting an existing package, really one will start by adding that to the Project.toml [test] section. TestItemRunner reexports @testitem, so then that is available also in say test/runtests.jl.

It thought a fair bit about it, and in the end I decided against it for a couple of reasons:

  1. the real work in converting a test suite from @testset to @testitem is to make sure the @testitems are self-contained, the renaming part seemed very little work to me, literally search-replace should do it.
  2. If we start to just pretend that @testsets can be run individually and surface that in the VS Code UI, then I can bet that thousands of users will start to use that on their existing test suite and nothing will work. And I think they would rightly say “but hey, my tests are written in the style that Base tells me to”, and then we would have to say “yeah, but if you want to run them inside VS Code, you have to remember…” So I think it would create too much confusion, support requests etc
  3. One really nice thing about this syntactic detection story is how absolutely trivial the @testitem macro is. The whole system is actually really, really simple. In terms of dev work for me/us that is a major benefit and I didn’t want to complicate things by brittle integration with an existing test framework that is just not a super good match.

Yep, that is an important point! You can still use @testsets inside @testitems and slowly and gradually split things into more granular and smaller @testitems over time!

You can click the button in a red circle in this screenshot:

image

That should show a terminal with all output from the test run process. Does that show anything informative? Generally what you describe sounds like a bug on our end to me :slight_smile:

2 Likes

Great stuff! Unfortunately, I wasn’t able to run a first simple test on a remote server with a custom julia executable path setting (it just hangs forever with a spinning circle). Is the latter being respected when spinning up the test processes?

FWIW, the test runner output (obtained by clicking on the small square highlighted in red above) is just empty.

It should in theory work, but it sounds like it doesn’t. Could you maybe paste the exact setting you have for your executable here? Maybe something jumps out. Or probably even better open an issue on the extension Github for this, clearly a bug.

No, it doesn’t show anything. I’ll try to get more info. Is there any verbose mode or log where I may find anything?

Also I’ll try to build a mwe and then file an issue appropriately.

2 Likes

@davidanthoff

I have the problem of the “play” button not doing anything in this package, which is only the template plus the tests: GitHub - lmiq/TestTests.jl

I’m on Linux x86 - with Julia 1.8.0.

There is no output at all, as the figure above shows, the spinning stuff just keeps spinning.

Opened the issue here: @testitem does not run · Issue #19 · julia-vscode/TestItemRunner.jl · GitHub

To try to reproduce the problem, do:

git clone https://github.com/lmiq/TestTests.jl
cd TestTests.jl
code .

VSCode will open. Wait for the tests to appear at the Erlenmeyer tab, and click play.

This will be fantastic when all the bugs get ironed out. Thank you for developing this.

Unfortunately, it doesn’t seem to be working for me. Using this MWE

module Differentiation
using TestItems, TestItemRunner

@testitem "First tests" begin
    @test true
end

end # module 

I also get the circle spinning, though not forever. Eventually it crashes:

System info:

Julia language server v1.7.6

VSCode version
Version: 1.70.2 (user setup)
Commit: e4503b30fc78200f846c62cf8091b76ff5547662
Date: 2022-08-16T05:35:13.448Z
Electron: 18.3.5
Chromium: 100.0.4896.160
Node.js: 16.13.2
V8: 10.0.139.17-electron.0
OS: Windows_NT x64 10.0.22000
Julia version
julia> versioninfo()
Julia Version 1.8.0
Commit 5544a0fab7 (2022-08-17 13:38 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 16 × 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, tigerlake)
  Threads: 8 on 16 virtual cores
Environment:
  JULIA_EDITOR = code.cmd
  JULIA_NUM_THREADS = 8

I am getting the same behavior. Endless spinning with nothing in the terminal output. I am running Ubuntu on Windows Subsystem for Linux 1 (WSL1).

The bug is being worked on. A workaround is to change the Julia extension version to “Julia Insiders” and make the path to the executable of Julia point to the actual application in the extension settings (I. E. with the full path, not just “julia”).

That said, the feature is amazing. I am completely into it, even in this experimental scenario. I’m just waiting for the blessing of David to release package versions with the new testing framework.

3 Likes

Alright, I pushed out a new build that should fix the problem of tests never running. Thanks to @lmiq who helped debug this!

If you had tried this before and weren’t able to run tests, please try again with this new build, version 1.7.10.

If things still don’t work for you, please let me know, then you’ve found a new bug! Ideally open a new issue in that case over at GitHub - julia-vscode/julia-vscode: Julia extension for Visual Studio Code.

7 Likes

For me this is one of the best additions in terms of development workflow and code quality. Having the possibility of adding tests just besides the code and run this tests independently is awesome. Thank you very much for that.

5 Likes

I just ran this again and still get the wheel spinning a long time which eventually causes a crash. Same environment as before , but Julia language server is now 1.7.10 and VSCode version is 1.71.0.

I just installed and tried for the first time and receive a crash after waiting for a bit. I’m on Julia 1.8.0, MacOS M1 Pro. I also tried setting the Julia path manually with no success.

@brianguenter @stephenll

There are now, at the Output tab, the option to choose Julia TestServer, Julia, and Julia Language Server logs. If you can post the content of those, that helps debugging the issues.

Like this, for example:

Before seeing this, I tried a few more changes to see if I can get this working. Maybe this is something I should know but I’m a very occasional VS Code user: I had to add Test and TestItems to my package to get this to work.

It might also be relevant: I don’t have the Test package in the package directory I am working on. I add it to a project in the Test directory. To be clear, the Test package exists in a project.toml in the Test directory. To get this to work, I had to add both Test and TestItems to the pkg directory.

I hope this helps.

As far as I understand, Test shouldn’t be needed. TestItems yes, if you want tests mixed with the package code. If all tests are in the runtests.jl file, then there you should load TestItemsRunner (and Tests, I’m not completely sure about this one, though).