Two years of test item framework updates

Tooling
23

I think there is an actual bug here. Say it is decided that a Manifest.toml file is indeed needed (I do not like this, but for the sake of this conversation I am sticking to that setup).

I can have my test runner instantiate the various test environment Project.tomls and make the Manifest.toml files. But that seems insufficient. That Manifest.toml file is indeed found, but then TestEnv.activate still disregards it. Here is a very involved PR where I am transitioning the QuantumClifford.jl test infrastructure to juliati and facing that exact problem (with a nasty workaround that uses JULIA_LOAD_PATH): QuantumClifford.jl/.buildkite/pipeline.yml at f74d0b7cc6b46b757733728b462cad57b1ec5e17 · QuantumSavory/QuantumClifford.jl · GitHub

The actual PR is Move specialized tests to juliati environments - Pull Request #820 - QuantumSavory/QuantumClifford.jl - GitHub

Here is an issue report that I asked one of my LLMs to post: Dependencies from a detected nested test project are not loadable · Issue #97 · julia-testitems/TestItemControllers.jl · GitHub

Currently it works like this:

  1. Buildkite instantiates the nested project, creating its Manifest.toml.

  2. JuliaWorkspaces finds that manifest, confirms that it path-develops QuantumClifford, and assigns the correct nested project to the test item.

  3. TestItemControllers starts the process from that project—but then calls TestEnv.activate(“QuantumClifford”).

  4. TestEnv.activate replaces the active project with a temporary environment based on QuantumClifford’s canonical test/Project.toml.

  5. The nested manifest’s packages remain resolved, but its [deps] are no longer direct dependencies of the active project. Julia does not permit using Aqua merely because Aqua appears in a
    manifest.

  6. Adding the nested project to JULIA_LOAD_PATH restores it as a fallback project, which is why the workaround succeeds.