# Two years of test item framework updates
**URL:** https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792
**Category:** Tooling
**Tags:** announcement
**Created:** [August 13, 2026, 8:16pm UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792 "2026-08-13T20:16:15Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)
#### Post date: [August 13, 2026, 8:16pm UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/1 "2026-08-13T20:16:15Z")
It has been about two years since my [last update](https://discourse.julialang.org/t/new-stable-features-for-the-test-item-framework/118512) on the test item framework, and a lot has happened in the meantime, so here is a roundup.
A quick recap for anyone who has not come across it: the test item framework lets you write your tests as independent `@testitem` blocks instead of one large nested `@testset` tree. Each test item runs in its own module, declares its own dependencies and tags, and — this is the whole point — can be run on its own, in parallel with others, and from wherever you happen to be working.
That last part is what changed the most. The framework is no longer a VS Code feature that happens to be usable elsewhere: you write your tests once, and they run in the editor, on the command line and in CI without any changes. There is now a [`julia-testitems` GitHub org](https://github.com/julia-testitems) that is gradually becoming the home for all of this, and a documentation site at [julia-testitems.org](https://julia-testitems.org) that replaces the user guide page that used to live under [julia-vscode.org](http://julia-vscode.org).
The rest of this post follows that order: first what is new in the editor, then on the command line, then in CI, and finally a section on the machinery underneath for anyone who is curious.
## Part 1: New features
### In the VS Code extension
These are the new features in the VS Code extension related to test items.
**A test profile per Juliaup channel.** Every Julia version you have installed through Juliaup now shows up as its own run profile in the Testing view. You can run the same test item against the release, the LTS and a nightly without changing anything about your project, and without leaving the editor.
**Stack traces on test failures.** When a test fails, the stack trace is now attached to the failure message and rendered in VS Code’s own stack trace UI, so frames are clickable and take you straight to the line that threw.
**Cancelling works, and the whole thing is much more robust.** Cancelling a run now stops it promptly and leaves you in a clean state, and the extension recovers properly when a test process or the controller itself goes down. This was the single most common source of complaints, and it should simply not come up any more.
**Control over parallelism.** The `julia.numTestProcesses` setting controls how many test processes are used: `1` runs your test items serially, `0` uses one process per core, and any other number is taken literally. Test processes persist between runs and show up in the Julia Workspace panel, and there are `Stop Test Process` and `Stop Test Controller` commands if you want to get rid of one.
**Linting around tests.** `@testitem`s now introduce their own scopes, `@test_throws` bodies are no longer linted, and test files now wait for their test environment to be indexed rather than being analyzed against the wrong environment. Note that these linter changes are only in the prerelease version of the extension at the moment.
### A command line test runner
The other half of “run your test items wherever you are working” is finally there: `juliati`, a command line test runner. No editor, no `test/runtests.jl`, and no `julia --project -e '...'` complications required. You point it at a folder, and it finds every `@testitem` in it, groups them by package, launches parallel test processes and reports the results.
It uses exactly the same discovery and execution engines as the VS Code extension, so it finds the same test items the editor shows you, and it honors the same configuration.
`juliati` is a [Julia app](https://pkgdocs.julialang.org/dev/apps/) and requires Julia 1.12 or newer:
```julia
using Pkg
Pkg.Apps.add(url="https://github.com/julia-vscode/TestItemApp.jl")
```
That installs a `juliati` executable into `~/.julia/bin`, which needs to be on your `PATH`. Then:
```julia-auto
juliati
```
or, if you are somewhere else:
```julia-auto
juliati path/to/MyPackage
```
which gives you something like
```julia-auto
Discovered 24 test item run(s) in 3 file(s)
Launching test processes....
Progress: 24/24 (23 passed, 1 failed)
24 tests ran, 23 passed, 1 failed.
```
There are no subcommands — running tests is the default action, and the only other things it does are `--help` and `--version`. The exit code is `0` when everything passed, `1` on test failures or definition errors, and `2` on usage errors, so it drops straight into a shell chain or a CI script.
The options worth knowing about:
- `--filter` takes a Julia expression evaluated for each test item with `name`, `tags`, `filename` and `package_name` in scope, so `juliati --filter ':fast in tags && !(:windows in tags)'` does what it looks like.
- `--max-workers` controls how many test processes run in parallel (by default the number of CPU threads, capped at 8), and `--timeout` sets a per-test-item timeout.
- `--coverage` runs the test processes in coverage mode.
- `--progress bar|log|none` picks between a progress bar, one line per finished test item, or silence.
- `--results-json` writes the complete run — every test item, its status, duration, failure messages with stack traces and captured output — to a JSON file for further processing.
- `--env`, `--julia-cmd` and `--juliaup-channel` control the environment and which Julia the test processes use, so `--juliaup-channel lts` runs your tests on the LTS.
- `--check-bounds` is described in the last section of this post.
One caveat: TestItemApp.jl is a prerelease and is not yet registered, which is why you install it from a URL. The command line interface may still change before the first stable release. Please do try it out and complain loudly about anything that feels wrong, that is exactly the point of announcing it now.
### Configuring what gets discovered
There is a new optional configuration file, `JuliaTestItems.toml`, that controls which files are searched for test items. Every surface reads it — VS Code, `juliati` and CI — so a single file keeps them all in agreement.
```toml
# Only look for test items in these folders.
include = ["src/ **", "test/**"]
# ...but never in these.
exclude = ["test/manual/**"]
```
The patterns are gitignore-style and relative to the folder containing the config file, and `exclude` always wins over `include`. Without a config file every `.jl` file in your project is searched, which is the right behavior for almost every package. It is worth having one when you have vendored or generated code that contains test items which are not yours to run, or scratch files you keep around for interactive debugging and never want to see in the test explorer.
The nearest `JuliaTestItems.toml` governs a file, and only that one — settings are not merged across nested config files, so to know how a folder is configured you read exactly one file.
Right now the file only covers discovery. Execution settings — worker counts, timeouts, environment variables, default tag filters — are planned as additional sections in the same file, and the keys above will keep working when they arrive.
### Test items in CI
A major new feature is `testitem-workflow`, a single reusable GitHub Actions workflow for packages that use the test item framework.
Every Julia package ends up carrying the same pile of CI YAML, and almost none of it is about the package itself. A matrix over Julia versions and platforms, a coverage upload, a documentation deployment, a tagging job. It gets copied over from a neighboring package, and from that day on every copy drifts on its own.
`testitem-workflow` replaces all of that with one file. You get linting, format checking, a test matrix, coverage, documentation deployment and TagBot, and you maintain none of it. Add this as `.github/workflows/juliaci.yml`:
```yml
name: Julia CI
on:
push: {branches: [main,master]}
pull_request: {types: [opened,synchronize,reopened,ready_for_review,converted_to_draft]}
issue_comment: {types: [created]}
workflow_dispatch: {inputs: {feature: {type: choice, description: What to run, options: [DocDeploy,LintAndTest,TagBot]}}}
jobs:
julia-ci:
uses: julia-testitems/testitem-workflow/.github/workflows/juliaci.yml@v2
permissions: write-all
secrets:
codecov_token: ${{ secrets.CODECOV_TOKEN }}
```
That is the entire configuration. Out of the box it tests the current release, the LTS and the smallest Julia version your `Project.toml` is compatible with, on Windows, Linux and macOS.
Note what is _not_ in there: a list of Julia versions. You never write version numbers into the workflow. The matrix is constructed fresh on every run, from the `julia` bound in your `Project.toml` together with whichever of the options below you have set. Widen the bound and the next run tests the wider range. A new Julia version is released, or a nightly moves, and it is picked up without anyone touching the file.
The options that control the matrix are `include-release-versions`, `include-lts-versions`, `include-smallest-compatible-minor-versions`, `include-all-compatible-minor-versions`, `include-rc-versions`, `include-beta-versions`, `include-alpha-versions` and `include-nightly-versions`, plus one per platform: `include-windows-x64`, `include-windows-x86`, `include-linux-x64`, `include-linux-x86`, `include-macos-x64` and `include-macos-aarch64`.
Any of those can be overridden for a specific trigger by prefixing it with `draft-pr-`, `pr-`, `main-` or `manual-trigger-`. That is how you get a quick signal on work in progress without giving up the full matrix everywhere else:
```yml
jobs:
julia-ci:
uses: julia-testitems/testitem-workflow/.github/workflows/juliaci.yml@v2
with:
draft-pr-include-lts-versions: false
draft-pr-include-windows-x64: false
draft-pr-include-windows-x86: false
draft-pr-include-linux-x86: false
draft-pr-include-macos-x64: false
draft-pr-include-macos-aarch64: false
permissions: write-all
secrets:
codecov_token: ${{ secrets.CODECOV_TOKEN }}
```
The same prefixes work on the other options. `filter` takes the same kind of Julia expression as `juliati` does, so `pr-filter: '!(:slow in tags)'` skips everything tagged `:slow` on pull requests. `testitem-timeout` (1200 seconds by default) terminates a single runaway test item and reports it as errored, instead of letting it hang until GitHub kills the whole job twenty minutes later with nothing to show for it. `env` takes a JSON string, for example `env: '{"FOO": "BAR"}'`, and `github_job_prep_script` points at a Julia file that is run once on each worker before any tests. Coverage in CI works on any Julia version, unlike the VS Code path, which needs 1.11 or newer.
For reporting, the whole matrix is merged into a single job summary. Identical failures across legs are deduplicated, so a test that fails on one platform only is reported once, with the platforms it failed on — you do not have to open dozens of job logs to find it. Lint results land in the same summary, and there is an artifact with the full untruncated output of every test process, which is where you look when something failed during precompilation and therefore belongs to no test item at all.
### One breaking change
There is one change that can affect an existing suite: setup modules are now loaded with `using` by default. Everything a setup module exports is therefore in scope in the test items that depend on it, without qualification. If a test item defines a name that a setup module also exports, that is now a conflict where it previously was not, and you will see it as an error rather than as silently different behavior.
## Part 2: Under the hood
None of what follows is something you have to act on. It is here because a fair number of these changes explain _why_ the things above became possible.
### TestItemControllers.jl
This is the biggest change of the last two years. All the machinery that actually runs test items used to live inside the VS Code extension. It is now a standalone package, and the extension has been using it since 1.140.0 — as do `juliati` and the CI actions.
There are two public APIs, both documented: a native Julia one (`TestItemController`, `ControllerCallbacks`, `execute_testrun`) and a JSONRPC wire protocol. If you want to build your own test item runner, or integrate test items into a different editor, you no longer have to reimplement any of this. The VS Code extension is simply one consumer of the same public API that is available to you.
**Event-driven architecture.** All state mutations flow through a single-threaded reactor loop, with explicit state machines for the controller, process and test run lifecycles, and guarded transitions between states. That is a dull thing to read about, but it is what finally cleared out the long tail of race conditions and hangs when you cancelled a run, restarted a process or changed the environment mid-run.
**Process pooling with Revise.** Idle test processes are kept around and reused. The pool is keyed on a hash of the test environment: if the environment has not changed, code is hot-reloaded with `Revise.revise()` instead of starting a fresh process; if it has, the process is restarted. This is what makes rerunning a single test item feel instantaneous.
**Multi-environment test runs.** A single run can execute the same test items against several configurations — different Julia versions, thread counts, environment variables, and coverage or debug settings. That is what surfaces as the per-Juliaup-channel test profiles in VS Code.
### Work stealing
Parallelism is where the scheduling gets interesting, so it is worth spelling out how work is handed to test processes.
When a run starts, the test items for a given environment are divided into chunks — roughly the number of items left divided by the number of processes left — and each process is given its whole chunk up front. That keeps the common case cheap: no round trip to the controller for every single test item.
The catch with handing out work up front is that test items are not equally expensive. One process can draw a chunk full of slow items while another burns through its share and then sits idle for the rest of the run. So a process that runs out of work does not simply stop. It looks at the other processes working in the same environment, picks the one with the longest remaining queue, and takes the back half of that queue for itself. That repeats until there is nothing worth stealing, at which point the idle process goes back to the pool and is available — warm, and reusable via Revise — for the next run.
The practical effect is that a parallel run finishes when the work is actually done, rather than when the unluckiest chunk is done.
### Test runs no longer touch your working tree
Activating a test environment could previously resolve and write a `Manifest.toml` straight into an environment that did not have one — in your source folder, as a side effect of running a test. Test runs now build a throwaway environment that mirrors yours and activate that instead. An existing manifest is honored where the run used to re-resolve from scratch, and `Base.active_project()` inside a test item no longer points at your folder.
Related: preferences set in your environment, whether in `LocalPreferences.toml` or a `[preferences]` section in the project file, now reach the test process, including while it precompiles. They used to be dropped before anything was compiled, which among other things made PrecompileTools’ `precompile_workload = false` escape hatch have no effect at all on a test run.
### Bounds checking
`Pkg.test` runs your tests with `--check-bounds=yes`, which forces bounds checks everywhere regardless of `@inbounds`. That is a reasonable default for a release check, but it has a cost that is easy to miss: it means the whole environment gets precompiled into a separate cache slot, so none of the precompilation from your normal development session can be reused.
Test runs are therefore now configurable. The `auto` mode respects `@inbounds` annotations and shares the precompile caches of your regular Julia sessions, so runs start fast. The `yes` mode matches `Pkg.test` semantics at the cost of a slow first run after switching. `juliati` defaults to `auto`, because the thing you want locally is a fast inner loop, and the CI action defaults to `yes`, because the thing you want in CI is the strict check.
**—**
The documentation is at [julia-testitems.org](https://julia-testitems.org). As always, questions and bug reports are welcome here or on GitHub — and `juliati` in particular is new enough that I would really like to hear how it behaves on other people’s projects.
---
### Author:  [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)
#### Post date: [August 14, 2026, 1:10am UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/2 "2026-08-14T01:10:26Z")
This sounds great! Can’t wait to try it! Thank you for all the improvements. 🎉
---
### Author:  [@simsurace](https://discourse.julialang.org/u/simsurace)
#### Post date: [August 15, 2026, 5:51pm UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/3 "2026-08-15T17:51:44Z")
How does parallelism work in CI? Can we parallelize a package test suite over arbitrary numbers of runners?
---
### Author:  [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)
#### Post date: [August 15, 2026, 6:45pm UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/4 "2026-08-15T18:45:00Z")
If you use the [reusable workflow](https://julia-testitems.org/guide/ci), it works like this:
- There is one job that creates a platform matrix of Julia versions and OS platforms the tests will run on. The inputs into that are the min Julia version in the `[compat]` section of your `Project.toml`, and then the various configuration toggles of the workflow (things like `include-windows-x64`, `include-smallest-compatible-minor-versions` etc), and the available Julia versions from Juliaup.
- We then launch one GitHub Actions job per platform/julia-version combo, and all of those jobs run in parallel and typically on different Github runners.
- Within each of these jobs the test item framework parallel scheduling occurs: we will launch multiple test processes, and all the test items are distributed over these test processes, and then work item stealing happens between these processes.
So the short version is that the “smart” scheduling and work item stealing stuff _only_ happens within one platform/Julia-version combo, not across them.
I think one could with modest effort change the 1:1 mapping of Github jobs/runners to platform/Julia-versions, i.e. we could do something like “run all the Julia versions we want to run tests on that run on windows on one github runner/job”, or “distribute all the test items for windows x64 on Julia 1.12 over 2 github runners” or things like that. Not really clear to me, though, how useful that would be. But there would still be no work item stealing across github jobs, I think that is probably out of reach…
---
### Author:  [@simsurace](https://discourse.julialang.org/u/simsurace)
#### Post date: [August 16, 2026, 7:58am UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/5 "2026-08-16T07:58:13Z")
Looks like having a large runner and/or parallelizing over tags is more practical right now.
---
### Author:  [@Torkel](https://discourse.julialang.org/u/Torkel)
#### Post date: [August 17, 2026, 10:49am UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/6 "2026-08-17T10:49:33Z")
Looks amazing!
Do you know of any packages that have embraced these changes well that one could take a look at for reference?
---
### Author:  [@krcools](https://discourse.julialang.org/u/krcools)
#### Post date: [August 18, 2026, 8:46am UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/7 "2026-08-18T08:46:00Z")
Could you point me to where I can find information on which environment the controller and runners use to run the test in?
In particular, does it matter whether test items are defined under `src` or under `tests`?
---
### Author:  [@franckgaga](https://discourse.julialang.org/u/franckgaga)
#### Post date: [August 18, 2026, 4:59pm UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/8 "2026-08-18T16:59:25Z")
> [@krcools](#):
>
> In particular, does it matter whether test items are defined under `src` or under `tests`?
I’m not 100% sure about this, but I will use the stack overflow tactic: If I’m wrong I will be corrected promptly 😁
For local execution within VS Code, no, it does not matter where the `@testitem` are defined. The test item controller will always use the `Project.toml` environment at the base of your package, and presumably with the additional dependencies listed in `tests/Project.toml`. I just tested and it seems that the additional dependencies in `docs/Project.toml` are not installed.
---
### Author:  [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)
#### Post date: [August 18, 2026, 7:24pm UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/9 "2026-08-18T19:24:27Z")
> [@Torkel](#):
>
> Do you know of any packages that have embraced these changes well that one could take a look at for reference?
I added a chapter to the docs that shows how to make an example package use the framework: [Example | Julia Test Items](https://julia-testitems.org/guide/example).
And I also had Claude search for examples of real world usage, and it came back with a lot 🙂 I’ve added that also to the docs at [Projects Using Test Items | Julia Test Items](https://julia-testitems.org/guide/users).
> [@krcools](#):
>
> Could you point me to where I can find information on which environment the controller and runners use to run the test in?
It is complicated and simple at the same time 😉 I think simple, because I hope by default it will just do something sensible/expected and hopefully users don’t have to think too much about it. Complicated, because Julia’s environment story is complex and we try to support all the different ways one can use that. I’ve added another chapter to the docs that explains environment selection in quite some detail at [Environments | Julia Test Items](https://julia-testitems.org/guide/environments).
> [@krcools](#):
>
> In particular, does it matter whether test items are defined under `src` or under `tests`?
No 🙂 The short version is: for each test item we first find the package it is in (and for both tests under `src` and `tests` that will end up with the same package), and then we find the environment to run the tests in, but that is off relative to the package we found.
> [@franckgaga](#):
>
> I’m not 100% sure about this, but I will use the stack overflow tactic: If I’m wrong I will be corrected promptly 😁
>
> For local execution within VS Code, no, it does not matter where the `@testitem` are defined. The test item controller will always use the `Project.toml` environment at the base of you package, and presumably with the additional dependencies listed in `tests/Project.toml`. I just tested and it seems that the additional dependencies in `docs/Project.toml` are not installed.
Haha, no stack overflow approach needed, that all sounds correct to me!
---
### Author:  [@franckgaga](https://discourse.julialang.org/u/franckgaga)
#### Post date: [August 18, 2026, 8:48pm UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/10 "2026-08-18T20:48:02Z")
Just a small comment: the documentation website ([https://julia-testitems.org/](https://julia-testitems.org/)) is hard to find. I would personally add this URL somewhere in the [organization page](https://github.com/julia-testitems) and also in the [VS Code test item framework documentation](https://www.julia-vscode.org/docs/stable/userguide/testitems/#Test-item-framework).
---
### Author:  [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)
#### Post date: [August 18, 2026, 9:12pm UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/11 "2026-08-18T21:12:36Z")
Ah, yes, great idea! I’ve added links everywhere 🙂
---
### Author:  [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)
#### Post date: [August 19, 2026, 5:28pm UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/12 "2026-08-19T17:28:42Z")
A week on, and the batch of work that was in flight when I wrote the post above has landed and is released. Most of it is small enough that it does not deserve its own thread, but there are a couple of things in here that change how a test run behaves, so here is a roundup.
Everything below is out and usable today, with one exception that is marked as such.
### Skipping test items
`@testitem` takes a new `skip` keyword argument.
```julia
@testitem "not ready yet" skip=true begin
...
end
@testitem "needs a recent Julia" skip=(VERSION < v"1.11") begin
...
end
```
`skip=true` is the boring case. The interesting one is the expression form: it is evaluated **in the test process, immediately before the test item would have run, and before its setups**. So `VERSION`, `Sys.iswindows()`, `Threads.nthreads()` and anything else you check see the process the tests actually run in, not the one that discovered them — which matters as soon as you run the same test item against several Juliaup channels, or on a CI matrix. A skipped item never pays the cost of evaluating its setups. The source text of the expression that caused the skip is reported as the reason.
This is the complement to `tags`: tags are literals resolved when the file is parsed, whereas `skip` is a question that can only be answered where the test runs.
It works on the command line and in `TestItemRunner` today, and it is honoured in VS Code in the next extension release. Under `Pkg.test`, a skipped item is recorded as `Test.Broken(:skipped, ...)`, the same as `@test_skip`, so it stays visible in the summary without failing the run.
### A new scheduler
The post above describes how test items are handed to test processes: they are divided into contiguous chunks up front, and a process that runs out of work steals the back half of the longest remaining queue. That is still the shape of it, but the first half — deciding who gets what — is no longer positional.
The default is now `duration` scheduling. The controller keeps, in memory for the session, how long each test item took the last time it ran, whether it failed, and how long each `@testmodule` setup took to evaluate. A new run places test items longest-first, each on the test process where it adds the least work, counting both the item’s own duration and the cost of any setup that process does not already have. Two consequences fall out of that:
- **Test items that failed last time are dealt out one per process, first.** This duplicates setup work, deliberately: when you are iterating on a failure, how quickly the failure comes back matters more than the makespan of the whole run.
- **Items that share a `@testmodule` are routed to the process that already has it.** A warm setup on a pooled process costs nothing, so setup-heavy suites stop re-evaluating the same module on four processes. Only `@testmodule` creates this affinity — `@testsnippet`s are re-evaluated per test item by definition and must not.
Setup costs are measured, not guessed. The first run of a session has no history and falls back to chunking, so this is something you notice on the second run onwards. Work stealing is unchanged and is still the correction mechanism when the estimates are wrong.
If you ever need to rule the scheduler out while diagnosing something, `--schedule contiguous` on the command line (or the `schedule` input in CI) restores the old behaviour exactly.
### What test processes do between test items
Two options that matter for long-running suites, both documented on the new [Test Processes](https://julia-testitems.org/guide/test-processes) page:
`--gc-between-testitems` runs a full `GC.gc()` after each test item. It is on by default whenever more than one test process is used, and `--no-gc-between-testitems` turns it off.
`--memory-threshold 0.9` takes a fraction of system memory above which a test process finishes its current test item, exits cleanly, and is replaced — the controller redistributes whatever it had left. This is off by default and still experimental, but it is the escape hatch if you have a suite that allocates its way into swap.
There is also a watchdog inside each test process now. Shortly before a test item hits its timeout, it writes a memory summary, the backtraces of every live task, and a one-second CPU profile to disk. A hung test item used to give you a timeout and nothing else; now it leaves evidence about _where_ it hung. The one thing it cannot do is help a test item that never reaches a safepoint at all.
### Test item ids
Test item ids used to be derived from an item’s position in its file, which meant inserting a test item renumbered every one below it. They now look like this:
```julia-auto
MyPkg@a1b2c3d4/test/parsing_tests.jl::parses floats
```
That is the package name, the first eight hex digits of its UUID, the path relative to the package root with forward slashes, and the test item’s name. The id is stable when you edit the file around it, and it is byte-identical in a dev checkout on Windows and on a Linux CI runner. That is what makes JUnit output, failure history and failures-first scheduling work at all, none of which can be built on an id that changes when you add a test above it.
Two things follow from this that you will notice. Using the same test item name twice in one file is now reported as a definition error, and each occurrence gets its own entry rather than the duplicates silently displacing one another. And if you have the same package checked out into two folders of one workspace, results no longer get attributed to the wrong copy. The format is documented under [Test item ids](https://julia-testitems.org/guide/cli#test-item-ids).
### `juliati` output and reporting
The command line runner gained the reporting options it was missing:
- `--junit-xml ` writes JUnit XML — one `` per source file, one `` per test item, with the performance statistics attached as properties.
- `--coverage-lcov ` writes LCOV, and implies `--coverage`.
- `--output issues|all|none` controls which captured output is echoed to the console, defaulting to `issues`, and `--stream` streams output live instead of collecting it (only with `--max-workers 1`, since interleaved live output from parallel processes is not readable).
- `--threads ` sets the thread configuration of the test processes.
- `--timeout` now defaults to 1200 seconds instead of running without a timeout, matching the CI action. `--timeout none` opts out.
Test items also report performance statistics now — elapsed time, allocated memory, allocation count, GC time and compile time — measured around the test item body only, so a setup is not charged to whichever item happened to trigger it. They show up in `--progress log`, in the JSON and JUnit output, and in VS Code in the next release.
One fix worth calling out: `--coverage` was broken outright when no coverage roots were given, in a way that made every test item in the run error. It works.
`juliati` is still installed from a URL, so if you installed it when the post above went up, re-run `Pkg.Apps.add(url="https://github.com/julia-testitems/TestItemApp.jl")` to pick any of this up.
### CI
All of the above is reachable from CI. Both the `julia-run-testitems` action and the reusable `testitem-workflow` gained the same seven inputs, each defaulting to whatever `juliati` itself does: `junit-path`, `coverage-lcov-path`, `output-mode`, `threads`, `gc-between-testitems`, `memory-threshold` and `schedule`. The action also has a `junit-path` output.
These deliberately have no `pr-`, `main-` or `draft-pr-` variants. Those prefixes exist for varying _how much_ gets tested on a given trigger; these describe how the test processes behave, which does not sensibly differ between a pull request and `main`.
### If you are still on `Pkg.test`
`TestItemRunner` 1.2.1 brings two of the above to `@run_package_tests`: it honours `JuliaTestItems.toml`, so it discovers exactly the same test items as the editor and the command line rather than its own approximation, and it honours `skip`. It also no longer walks into `.git`, `node_modules` and friends looking for test items, and an error in a test setup is now recorded against the test item that needed it instead of aborting the whole run.
### In VS Code
None of this has shipped in an extension release yet — it is on `main` and lands in the next one. When it does: `skip` is honoured, performance statistics appear as a summary line in a test item’s run output, the new ids fix results being reported against the wrong copy of a package checked out twice, duplicate test item names are surfaced rather than silently swallowed, and a test run that fails to start or is cancelled stops spinning forever in the Test Explorer.
### Documentation
The documentation site grew a fair bit, including answers to a few things that came up in this thread.
There is now an [example chapter](https://julia-testitems.org/guide/example) built around [TestItemExamplePackage.jl](https://github.com/julia-testitems/TestItemExamplePackage.jl), a deliberately tiny package that exists to show, file by file, what it takes to move a freshly generated package onto the test item stack: the test target, test items with tags and setups, a `test/runtests.jl` for `Pkg.test`, the one-file CI workflow, a format check and Documenter docs. If you want a reference to copy from rather than prose to read, start there.
Which environment a test item runs in, and where files have to live for it to work, now has a chapter of its own. [Environments](https://julia-testitems.org/guide/environments) spells out the whole resolution — which package owns a file, which project supplies the manifest, when the active environment is consulted and how that differs between VS Code, the REPL and `juliati`, what the test process actually sees, and the guarantee that none of this ever writes into a folder of yours.
For anyone looking for packages to crib from, there is now a [Projects Using Test Items](https://julia-testitems.org/guide/users) page with a couple of hundred open-source packages that write their tests as `@testitem` blocks. It was assembled from a GitHub search, so it is a sample rather than a census and will drift out of date — there is an edit link on the page if yours is missing.
There is also the new Test Processes page mentioned above, and the [julia-testitems](https://github.com/julia-testitems) org finally has a front page.
As before, bug reports and complaints are welcome here or on GitHub, and I am still particularly interested in how `juliati` behaves on other people’s projects.
---
### Author:  [@franckgaga](https://discourse.julialang.org/u/franckgaga)
#### Post date: [August 20, 2026, 3:15pm UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/13 "2026-08-20T15:15:22Z")
I’m working on transitioning to the new `juliaci.yml` in ModelPredictiveControl.jl, here are my comments (most of them are my own opinions, I obviously do not represent the majority):
- One single `yaml` file for everything, with builtin parallelism and without explicit Julia version numbers, is awesome. Perfectly aligned with the “no boilerplate code”. Excellent work!
- My test runs are 1.25-1.5 faster with the new builtin parallelism on CI.
- I think that testing on x86 Linux and x86 Windows by default is overkill.
- I would personally add release candidate testing by default. That’s the whole point of release candidates, it targets package maintainer. This is also a smooth transition from the [basic CI configuration](https://juliaci.github.io/PkgTemplates.jl/stable/user/#PkgTemplates.GitHubActions) with the default `pre` setting. As a package maintainer, I like RC testing since it is proactive. An ounce of prevention is worth a pound of cure.
- For me, the default 1200 s timeout was too low. My 2 biggest tests takes longer than this. But it is a clear sign that I need to reduce their size and separate them into smaller chunks. Just wanted to mention this, I still think that 1200 s is an adequate default for CI.
Worth the migration overall!
---
### Author:  [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)
#### Post date: [August 20, 2026, 4:23pm UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/14 "2026-08-20T16:23:42Z")
> [@franckgaga](#):
>
> - I think that testing on x86 Linux and x86 Windows by default is overkill.
I kind of just want to stick with the platforms that Julia supports as tier 1. I think I saw a discussion that Julia itself might drop x86 Windows support, and then I would presumably follow that choice. I do think having at least one 32 bit system is good, though, it _very_ regularly finds bugs in my code 🙂
> [@franckgaga](#):
>
> - I would personally add release candidate testing by default.
Yup, that makes sense, I’ll change the default!
> [@franckgaga](#):
>
> - For me, the default 1200 s timeout was too loo.
Maybe we should just not have a timeout at all? At some point the github runners will terminate the job in any case… And any timeout that we pick for everyone is kind of arbitrary. Might be better to just have none and then give people an option to set one if they need it…
---
### Author:  [@franckgaga](https://discourse.julialang.org/u/franckgaga)
#### Post date: [August 20, 2026, 5:05pm UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/15 "2026-08-20T17:05:12Z")
> [@davidanthoff](#):
>
> Maybe we should just not have a timeout at all? At some point the github runners will terminate the job in any case… And any timeout that we pick for everyone is kind of arbitrary. Might be better to just have none and then give people an option to set one if they need it…
Yep, no timeout by default would slightly “safer”. But I do think that forcing us to reduce the size of our `@testitem` is a good thing (improved parallelism, quicker feedback, etc.). I reduced the size of my two large `@testitem`s and they are now okay. I also hit issues with `Aqua.test_all()`, it took longer than 1200 s. I assume that many packages do this, so it creates friction in the migration. I’m currently trying to call the ~8 individual tests of Aqua.jl in separate `@testitem`s, but some of them like method ambiguities are dangerously close to 1200 s, and this is the smallest it can get.
---
### Author:  [@franckgaga](https://discourse.julialang.org/u/franckgaga)
#### Post date: [August 21, 2026, 12:35am UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/16 "2026-08-21T00:35:29Z")
Okay I hit a weird bug. All the tests in a `@testitem` passes, but the controller does not let go the test item and it failed because my 3600 s timeout was hit. You can see it here: [changed: using the new `juliaci.yml` workflow for test items · JuliaControl/ModelPredictiveControl.jl@7255d9f · GitHub](https://github.com/JuliaControl/ModelPredictiveControl.jl/actions/runs/32420160289/attempts/1#summary-96623620550)
Because of this I tried to disable the timeout in `juliaci.yml` but I was not able to. I tried `testitem-timeout: null`. `testitem-timeout: 0`, `testitem-timeout: Inf`, `testitem-timeout: ` but nothing work. It’s not documented what should be the value to disable the timeout.
edit: I think `testitem-timeout: -1` worked! 🍾🥳🍾🥳
edit²: nope 😪
---
### Author:  [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)
#### Post date: [August 21, 2026, 3:55am UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/17 "2026-08-21T03:55:54Z")
> [@franckgaga](#):
>
> Okay I hit a weird bug.
I have something incoming that will hopefully allow us to figure out what is really going on. I don’t think it is going to fix the problem, we’ll probably have to first get this diagnostic code in. Do you want to open an issue against TestItemController for this to track?
---
### Author:  [@Krastanov](https://discourse.julialang.org/u/Krastanov)
#### Post date: [August 23, 2026, 3:43pm UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/18 "2026-08-23T15:43:22Z")
Thank you for the update! I have a package with a very hairy TestItemRunner setup and was considering moving it to ParallelTestRunner. I have done that for a few simpler packages, but doing it for QuantumClifford.jl would be painful and I hope these new updates will help me avoid that.
I still have a few (I believe common) needs that I can not figure out how to do with the new test item framework:
- I am running my tests in buildkite and I do not want to figure out how to create and maintain a new buildkite action that uses `Pkg.test`. Moreover, I want to use the standard `Pkg.test(test_args=...)` because otherwise I am becoming dependent on too much custom code that I can not trust will be maintained in the future.
- I want to be able to mark some tests need a custom `test/customproject/Project.toml` instead of the default `test/Project.toml` for various reasons. With ParallelTestRunner this is as easy as `if ARGV[...]=...; Pkg.activate(...)`, but with the current TestItemRunner I need to do `Pkg.add` instead which is much messier, because I can not mark a test item with a custom Project.toml
- this type of custom Project.toml is necessary for JET tests because if JET is a dependency of test/Project.toml, then frequently tests will altogether fail because JET is not installable, and julia crashes during instantiation
- it is also important for GPU extension tests, because I do not want my main test runner to download 5GB of CUDA (only the GPU test runner should do that)
I guess my questions are:
- Will TestItemRunner.jl ever support tests running in parallel?
- What is the correct way for TestItemRunner.jl to run a test item with a different Project.toml (not the one in test/Project.toml)?
---
### Author:  [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)
#### Post date: [August 23, 2026, 4:43pm UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/19 "2026-08-23T16:43:03Z")
> [@Krastanov](#):
>
> - Will [TestItemRunner.jl](https://juliaregistries.github.io/General/packages/redirect_to_repo/TestItemRunner) ever support tests running in parallel?
Probably not. The reason is this: Its role is to make sure test items always run in all contexts via `Pkg.test()`. That needs to run back all the way to Julia 1.0. But all the parallel test running requires a “test item controller” process that runs on Julia 1.12 (which can then dispatch test client processes that are run on older Julia versions). Squaring these requirements would be really tough. Not impossible, but I’m just worried it would get too involved.
I am also hoping that `juliati` (the command line utility) would cover the kind of situation that you have. Essentially all you would have to do in buildkite is run a very few Julia commands: 1) install the TestItemApp, then 2) launch `juliati` with say command line options to write results into a standard xml format and then that is pretty much it. It now supports exporting results as JUnit XML, which I assume buildkite has support for downstream.
Another option would be [GitHub - julia-testitems/TestItemRuns.jl · GitHub](https://github.com/julia-testitems/TestItemRuns.jl), although that seems more involved for your case.
Having said that, maybe the right thing is to just provide a buildkite support out of the box…
> [@Krastanov](#):
>
> - What is the correct way for [TestItemRunner.jl](https://juliaregistries.github.io/General/packages/redirect_to_repo/TestItemRunner) to run a test item with a different Project.toml (not the one in test/Project.toml)?
So the algorithm should work like this: for any test item we detect which package it belongs too by walking up the folder hierarchy until we find a `Project.toml` that is a package. Then we also find the env in which it should be run, by again walking up the folder hierarchy looking for a manifest (and some fallback options). So I think if you just say put a `Projec.toml`/`Manifest.toml` in a sub folder like `test/test_env_2` and then in that project you dev the top level package, and then put test files into that folder, then I _think_ it should run those test items in that env.
So that is how it is supposed to work. If not, open a bug report 🙂
---
### Author:  [@Krastanov](https://discourse.julialang.org/u/Krastanov)
#### Post date: [August 23, 2026, 5:59pm UTC](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/20 "2026-08-23T17:59:58Z")
> [@davidanthoff](#):
>
> So the algorithm should work like this: for any test item we detect which package it belongs too by walking up the folder hierarchy until we find a `Project.toml` that is a package. Then we also find the env in which it should be run, by again walking up the folder hierarchy looking for a manifest (and some fallback options). So I think if you just say put a `Projec.toml`/`Manifest.toml` in a sub folder like `test/test_env_2` and then in that project you dev the top level package, and then put test files into that folder, then I _think_ it should run those test items in that env.
Oh, that is great! I will try it out and report any issues.
[Next page](https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792.md?page=2)