There is something that is confusing me about the prescription of how the test environment is found for a given testitem. Finding the package by walking up and finding a Project.toml that defines a package makes sense. But why is it walking up the directory tree searching for a Manifest.toml to find the test environment – I do not want my test environment to be defined by a Manifest.toml that I have to check in my git repository, I want it to use a much simpler human readable Project.toml, e.g. test/Project.toml and test/special_test_env/Project.toml and so on, whichever is first.
The real question is what is the behaviour of normal Pkg.test() runs (without @testitem)? If there is a Manifest.toml file next to the base Project.toml file of the package, will the testing environnent uses this manifest file ? I don’t think so. It will generate its own manifest. If so, I agree with @Krastanov that @testitem should reproduce this behaviour. For package testing through Pkg.test() or CI, it would completely ignore this manifest file.
Another viable option would be to use the discovered manifest file only for @testitem runs through the VS code extension, since it’s a bit like running a block of code in the REPL (and this is likely the environnement activated in the REPL). Or, even more intuitive IMO, use the manifest of the selected environnement at the bottom of the window for VS code runs, here:
I think there is an actual bug here. Say it is decided that a Manifest.toml file is indeed needed (I do not like this, but for the sake of this conversation I am sticking to that setup).
I can have my test runner instantiate the various test environment Project.tomls and make the Manifest.toml files. But that seems insufficient. That Manifest.toml file is indeed found, but then TestEnv.activate still disregards it. Here is a very involved PR where I am transitioning the QuantumClifford.jl test infrastructure to juliati and facing that exact problem (with a nasty workaround that uses JULIA_LOAD_PATH): QuantumClifford.jl/.buildkite/pipeline.yml at f74d0b7cc6b46b757733728b462cad57b1ec5e17 · QuantumSavory/QuantumClifford.jl · GitHub
The actual PR is Move specialized tests to juliati environments - Pull Request #820 - QuantumSavory/QuantumClifford.jl - GitHub
Here is an issue report that I asked one of my LLMs to post: Dependencies from a detected nested test project are not loadable · Issue #97 · julia-testitems/TestItemControllers.jl · GitHub
Currently it works like this:
-
Buildkite instantiates the nested project, creating its Manifest.toml.
-
JuliaWorkspaces finds that manifest, confirms that it path-develops QuantumClifford, and assigns the correct nested project to the test item.
-
TestItemControllers starts the process from that project—but then calls TestEnv.activate(“QuantumClifford”).
-
TestEnv.activate replaces the active project with a temporary environment based on QuantumClifford’s canonical test/Project.toml.
-
The nested manifest’s packages remain resolved, but its [deps] are no longer direct dependencies of the active project. Julia does not permit using Aqua merely because Aqua appears in a
manifest. -
Adding the nested project to JULIA_LOAD_PATH restores it as a fallback project, which is why the workaround succeeds.
I just responded on that bug report issue. Short version is that I think I gave an incorrect explanation here in this thread, and the implementation is how it is supposed to be at the moment, namely mimicking Pkg.test env handling. But happy to discuss this more over at Dependencies from a detected nested test project are not loadable · Issue #97 · julia-testitems/TestItemControllers.jl · GitHub, probably best done there as that is a pretty in the weeds topic 
Thanks everyone for reporting this stuff!