I just pushed out a new release of the Julia VS Code extension (1.7.6) to everyone that includes a preview of our new test UI and a general new testing framework for Julia.
For a short demo, take a look at our Juliacon talk Julia in VS Code - What's New | David Anthoff, Sebastian Pfitzner | JuliaCon 2022 - YouTube.
This new set of features consists of two new packages (TestItems.jl and TestItemRunner.jl) and the new UI in the Julia VS Code extension. A good way to refer to this new test framework is by the “test item” name.
The core feature in this new framework is that you can structure tests into
@testitem blocks and then individually run those, rather than having to run all your tests at once. A typical
@testitem might look like this:
@testitem "First tests" begin x = foo("bar") @test length(x)==3 @test x == "bar" end
@testitem always has a name (here “First tests”) and then some code in a
begin ... end block. The code inside a
@testitem must be executable by itself, i.e. it can not depend on code that appears outside of the
@testitem, unless that code is somehow explicitly imported or included from within the
@testitem. There is one exception to this: the code inside the
@testitem will run inside a temporary module where
using Test and
using MYPACKAGENAME was already executed, so anything exported from either the
Test module or the package your are developing can be directly used. In the example above this applies to the
foo function (presumably defined in the package that is being tested) and the
@testitems can appear anywhere in a package. They do not have to be in the
test folder, nor do they have to be in a file that is included by
test/runtests.jl. In fact,
@testitems can even be located inside your regular package code, for example next to the code they are testing. In that case you just need to take a dependency on the TestItems.jl package so that you have access to the
@testitem macro. If you have a package
MyPackage, then the file
src/MyPackage.jl could look like this:
module MyPackage using TestItems export foo foo(x) = x @testitem "First tests" begin x = foo("bar") @test length(x)==3 @test x == "bar" end end
If you don’t like this inline
@testitem style, you can also just put
@testitem blocks into Julia files in your
When you open a Julia package inside VS Code and have the Julia extension installed it will constantly (after every keypress!) look for any and all
@testitems in your Julia files. If any are found, they will appear in various places in the UI.
You can find all detected
@testitems in the Testing activity bar in VS Code:
The testing activity area then provides you with options to run individual
@testitems, look at results etc.
VS Code will also place a small little run button next to each detected
@testitem in the text editor itself:
In addition to all these UI elements that allow you to run tests, there is also UI to display test results. For example, when you run tests and some of them fail, the extension will collect all these test failures and then display them in a structured way, directly at the place in the code where a specific test failed:
Especially when you run a lot of tests with large test files this makes it much easier to find the specific test that failed, no more hunting in the REPL for file and line information!
This part is a little less fleshed out, but you can use the TestItemRunner.jl package to run
@testitems as part of a traditional
Pkg.test workflow. This makes it easy to integrate these new types of tests with for example a continuous integration setup.
To enable integration with
Pkg.test for a package that uses
@testitem, you just have to do two things:
- Add TestItemRunner.jl as a test dependency to your package
- Put the following code into the package’s
using TestItemRunner @run_package_tests
I hope that in the future we can make the TestItemRunner.jl package much more feature complete, for example add the ability to only run a subset of
@testitems (as you can already do in VS Code), add support for parallel execution etc. Help is most welcome!
We already have some great testing packages in Julialand, and some of them (like the excellent ReTest.jl) seem to provide a lot of similar functionality already, so why create yet another testing framework?
The core reason is that we have a very different requirement for the VS Code extension: we need to detect test items at every keystroke. To do this, I used a completely different test detection strategy than any of the existing testing frameworks. In the test item design, no user code is run at all for test item detection. Test item detection is purely done by syntactic analysis of Julia source files. This kind of approach integrates really well with the existing analysis we have in the LanguageServer.jl that powers the rest of the Julia extension.
One nice side benefit is that the design of this test item framework is quite simple. For example, the definition of the
@testitem macro is almost hilariously simple, you can see it here. Having it so simple and not doing anything is ideal, because that means that adding inline tests into the actual package code should really not add any runtime overhead at all to a package (it does probably add a tiny bit of precompile time).
I also want to point out some more details about the test runner in VS Code. The design there is that we first detect in which environment a given
@testitem should run, then we spin up a test process per individual environment. These test processes are long run, i.e. once they have started they are reused whenever you execute another new
@testitem. By reusing these processes, we can cut out a huge amount of delay between running tests, rerunning an individual test item is often completely instantaneous. The Julia extension now also ships Revise.jl as part of the extension, and uses that to detect any code changes you make to the package code you are testing. The integration with Revise.jl is completely under the hood and automatic. For example, if you make a change to your package that
Revise cannot track (like redefining a struct), the extension will recognize that and automatically restart the test process instead of relying on
Revise to handle this update. The net effect of that is that you can just freely edit the code you are testing and the test code itself, rerun small parts of it all the time and there should be minimal delays in all of that.
While this feature ships in the regular Julia VS Code extension, we are declaring it a preview at the moment. We want to collect feedback on the design and usability for a while, potentially address design issues that might crop up, and only then will we declare it stable and released.
So, please try this out and let us know what you think! And if some folks want to help improve TestItemRunner.jl, that would be especially fantastic.