It has been about two years since my last update on the test item framework, and a lot has happened in the meantime, so here is a roundup.
A quick recap for anyone who has not come across it: the test item framework lets you write your tests as independent @testitem blocks instead of one large nested @testset tree. Each test item runs in its own module, declares its own dependencies and tags, and — this is the whole point — can be run on its own, in parallel with others, and from wherever you happen to be working.
That last part is what changed the most. The framework is no longer a VS Code feature that happens to be usable elsewhere: you write your tests once, and they run in the editor, on the command line and in CI without any changes. There is now a julia-testitems GitHub org that is gradually becoming the home for all of this, and a documentation site at julia-testitems.org that replaces the user guide page that used to live under julia-vscode.org.
The rest of this post follows that order: first what is new in the editor, then on the command line, then in CI, and finally a section on the machinery underneath for anyone who is curious.
Part 1: New features
In the VS Code extension
These are the new features in the VS Code extension related to test items.
A test profile per Juliaup channel. Every Julia version you have installed through Juliaup now shows up as its own run profile in the Testing view. You can run the same test item against the release, the LTS and a nightly without changing anything about your project, and without leaving the editor.
Stack traces on test failures. When a test fails, the stack trace is now attached to the failure message and rendered in VS Code’s own stack trace UI, so frames are clickable and take you straight to the line that threw.
Cancelling works, and the whole thing is much more robust. Cancelling a run now stops it promptly and leaves you in a clean state, and the extension recovers properly when a test process or the controller itself goes down. This was the single most common source of complaints, and it should simply not come up any more.
Control over parallelism. The julia.numTestProcesses setting controls how many test processes are used: 1 runs your test items serially, 0 uses one process per core, and any other number is taken literally. Test processes persist between runs and show up in the Julia Workspace panel, and there are Stop Test Process and Stop Test Controller commands if you want to get rid of one.
Linting around tests. @testitems now introduce their own scopes, @test_throws bodies are no longer linted, and test files now wait for their test environment to be indexed rather than being analyzed against the wrong environment. Note that these linter changes are only in the prerelease version of the extension at the moment.
A command line test runner
The other half of “run your test items wherever you are working” is finally there: juliati, a command line test runner. No editor, no test/runtests.jl, and no julia --project -e '...' complications required. You point it at a folder, and it finds every @testitem in it, groups them by package, launches parallel test processes and reports the results.
It uses exactly the same discovery and execution engines as the VS Code extension, so it finds the same test items the editor shows you, and it honors the same configuration.
juliati is a Julia app and requires Julia 1.12 or newer:
using Pkg
Pkg.Apps.add(url="https://github.com/julia-vscode/TestItemApp.jl")
That installs a juliati executable into ~/.julia/bin, which needs to be on your PATH. Then:
juliati
or, if you are somewhere else:
juliati path/to/MyPackage
which gives you something like
Discovered 24 test item run(s) in 3 file(s)
Launching test processes....
Progress: 24/24 (23 passed, 1 failed)
24 tests ran, 23 passed, 1 failed.
There are no subcommands — running tests is the default action, and the only other things it does are --help and --version. The exit code is 0 when everything passed, 1 on test failures or definition errors, and 2 on usage errors, so it drops straight into a shell chain or a CI script.
The options worth knowing about:
--filtertakes a Julia expression evaluated for each test item withname,tags,filenameandpackage_namein scope, sojuliati --filter ':fast in tags && !(:windows in tags)'does what it looks like.--max-workerscontrols how many test processes run in parallel (by default the number of CPU threads, capped at 8), and--timeoutsets a per-test-item timeout.--coverageruns the test processes in coverage mode.--progress bar|log|nonepicks between a progress bar, one line per finished test item, or silence.--results-jsonwrites the complete run — every test item, its status, duration, failure messages with stack traces and captured output — to a JSON file for further processing.--env,--julia-cmdand--juliaup-channelcontrol the environment and which Julia the test processes use, so--juliaup-channel ltsruns your tests on the LTS.--check-boundsis described in the last section of this post.
One caveat: TestItemApp.jl is a prerelease and is not yet registered, which is why you install it from a URL. The command line interface may still change before the first stable release. Please do try it out and complain loudly about anything that feels wrong, that is exactly the point of announcing it now.
Configuring what gets discovered
There is a new optional configuration file, JuliaTestItems.toml, that controls which files are searched for test items. Every surface reads it — VS Code, juliati and CI — so a single file keeps them all in agreement.
# Only look for test items in these folders.
include = ["src/**", "test/**"]
# ...but never in these.
exclude = ["test/manual/**"]
The patterns are gitignore-style and relative to the folder containing the config file, and exclude always wins over include. Without a config file every .jl file in your project is searched, which is the right behavior for almost every package. It is worth having one when you have vendored or generated code that contains test items which are not yours to run, or scratch files you keep around for interactive debugging and never want to see in the test explorer.
The nearest JuliaTestItems.toml governs a file, and only that one — settings are not merged across nested config files, so to know how a folder is configured you read exactly one file.
Right now the file only covers discovery. Execution settings — worker counts, timeouts, environment variables, default tag filters — are planned as additional sections in the same file, and the keys above will keep working when they arrive.
Test items in CI
A major new feature is testitem-workflow, a single reusable GitHub Actions workflow for packages that use the test item framework.
Every Julia package ends up carrying the same pile of CI YAML, and almost none of it is about the package itself. A matrix over Julia versions and platforms, a coverage upload, a documentation deployment, a tagging job. It gets copied over from a neighboring package, and from that day on every copy drifts on its own.
testitem-workflow replaces all of that with one file. You get linting, format checking, a test matrix, coverage, documentation deployment and TagBot, and you maintain none of it. Add this as .github/workflows/juliaci.yml:
name: Julia CI
on:
push: {branches: [main,master]}
pull_request: {types: [opened,synchronize,reopened,ready_for_review,converted_to_draft]}
issue_comment: {types: [created]}
workflow_dispatch: {inputs: {feature: {type: choice, description: What to run, options: [DocDeploy,LintAndTest,TagBot]}}}
jobs:
julia-ci:
uses: julia-testitems/testitem-workflow/.github/workflows/juliaci.yml@v2
permissions: write-all
secrets:
codecov_token: ${{ secrets.CODECOV_TOKEN }}
That is the entire configuration. Out of the box it tests the current release, the LTS and the smallest Julia version your Project.toml is compatible with, on Windows, Linux and macOS.
Note what is not in there: a list of Julia versions. You never write version numbers into the workflow. The matrix is constructed fresh on every run, from the julia bound in your Project.toml together with whichever of the options below you have set. Widen the bound and the next run tests the wider range. A new Julia version is released, or a nightly moves, and it is picked up without anyone touching the file.
The options that control the matrix are include-release-versions, include-lts-versions, include-smallest-compatible-minor-versions, include-all-compatible-minor-versions, include-rc-versions, include-beta-versions, include-alpha-versions and include-nightly-versions, plus one per platform: include-windows-x64, include-windows-x86, include-linux-x64, include-linux-x86, include-macos-x64 and include-macos-aarch64.
Any of those can be overridden for a specific trigger by prefixing it with draft-pr-, pr-, main- or manual-trigger-. That is how you get a quick signal on work in progress without giving up the full matrix everywhere else:
jobs:
julia-ci:
uses: julia-testitems/testitem-workflow/.github/workflows/juliaci.yml@v2
with:
draft-pr-include-lts-versions: false
draft-pr-include-windows-x64: false
draft-pr-include-windows-x86: false
draft-pr-include-linux-x86: false
draft-pr-include-macos-x64: false
draft-pr-include-macos-aarch64: false
permissions: write-all
secrets:
codecov_token: ${{ secrets.CODECOV_TOKEN }}
The same prefixes work on the other options. filter takes the same kind of Julia expression as juliati does, so pr-filter: '!(:slow in tags)' skips everything tagged :slow on pull requests. testitem-timeout (1200 seconds by default) terminates a single runaway test item and reports it as errored, instead of letting it hang until GitHub kills the whole job twenty minutes later with nothing to show for it. env takes a JSON string, for example env: '{"FOO": "BAR"}', and github_job_prep_script points at a Julia file that is run once on each worker before any tests. Coverage in CI works on any Julia version, unlike the VS Code path, which needs 1.11 or newer.
For reporting, the whole matrix is merged into a single job summary. Identical failures across legs are deduplicated, so a test that fails on one platform only is reported once, with the platforms it failed on — you do not have to open dozens of job logs to find it. Lint results land in the same summary, and there is an artifact with the full untruncated output of every test process, which is where you look when something failed during precompilation and therefore belongs to no test item at all.
One breaking change
There is one change that can affect an existing suite: setup modules are now loaded with using by default. Everything a setup module exports is therefore in scope in the test items that depend on it, without qualification. If a test item defines a name that a setup module also exports, that is now a conflict where it previously was not, and you will see it as an error rather than as silently different behavior.
Part 2: Under the hood
None of what follows is something you have to act on. It is here because a fair number of these changes explain why the things above became possible.
TestItemControllers.jl
This is the biggest change of the last two years. All the machinery that actually runs test items used to live inside the VS Code extension. It is now a standalone package, and the extension has been using it since 1.140.0 — as do juliati and the CI actions.
There are two public APIs, both documented: a native Julia one (TestItemController, ControllerCallbacks, execute_testrun) and a JSONRPC wire protocol. If you want to build your own test item runner, or integrate test items into a different editor, you no longer have to reimplement any of this. The VS Code extension is simply one consumer of the same public API that is available to you.
Event-driven architecture. All state mutations flow through a single-threaded reactor loop, with explicit state machines for the controller, process and test run lifecycles, and guarded transitions between states. That is a dull thing to read about, but it is what finally cleared out the long tail of race conditions and hangs when you cancelled a run, restarted a process or changed the environment mid-run.
Process pooling with Revise. Idle test processes are kept around and reused. The pool is keyed on a hash of the test environment: if the environment has not changed, code is hot-reloaded with Revise.revise() instead of starting a fresh process; if it has, the process is restarted. This is what makes rerunning a single test item feel instantaneous.
Multi-environment test runs. A single run can execute the same test items against several configurations — different Julia versions, thread counts, environment variables, and coverage or debug settings. That is what surfaces as the per-Juliaup-channel test profiles in VS Code.
Work stealing
Parallelism is where the scheduling gets interesting, so it is worth spelling out how work is handed to test processes.
When a run starts, the test items for a given environment are divided into chunks — roughly the number of items left divided by the number of processes left — and each process is given its whole chunk up front. That keeps the common case cheap: no round trip to the controller for every single test item.
The catch with handing out work up front is that test items are not equally expensive. One process can draw a chunk full of slow items while another burns through its share and then sits idle for the rest of the run. So a process that runs out of work does not simply stop. It looks at the other processes working in the same environment, picks the one with the longest remaining queue, and takes the back half of that queue for itself. That repeats until there is nothing worth stealing, at which point the idle process goes back to the pool and is available — warm, and reusable via Revise — for the next run.
The practical effect is that a parallel run finishes when the work is actually done, rather than when the unluckiest chunk is done.
Test runs no longer touch your working tree
Activating a test environment could previously resolve and write a Manifest.toml straight into an environment that did not have one — in your source folder, as a side effect of running a test. Test runs now build a throwaway environment that mirrors yours and activate that instead. An existing manifest is honored where the run used to re-resolve from scratch, and Base.active_project() inside a test item no longer points at your folder.
Related: preferences set in your environment, whether in LocalPreferences.toml or a [preferences] section in the project file, now reach the test process, including while it precompiles. They used to be dropped before anything was compiled, which among other things made PrecompileTools’ precompile_workload = false escape hatch have no effect at all on a test run.
Bounds checking
Pkg.test runs your tests with --check-bounds=yes, which forces bounds checks everywhere regardless of @inbounds. That is a reasonable default for a release check, but it has a cost that is easy to miss: it means the whole environment gets precompiled into a separate cache slot, so none of the precompilation from your normal development session can be reused.
Test runs are therefore now configurable. The auto mode respects @inbounds annotations and shares the precompile caches of your regular Julia sessions, so runs start fast. The yes mode matches Pkg.test semantics at the cost of a slow first run after switching. juliati defaults to auto, because the thing you want locally is a fast inner loop, and the CI action defaults to yes, because the thing you want in CI is the strict check.
—
The documentation is at julia-testitems.org. As always, questions and bug reports are welcome here or on GitHub — and juliati in particular is new enough that I would really like to hear how it behaves on other people’s projects.