I think now we’re getting to the root of the issue.
Pinning your test environment, as in, only testing against pinned versions, seems like a bad idea to me (hence: don’t commit Manifest files). But the fundamental ability to reproduce a particular test run, on CI or otherwise, is a good idea.
So yes, the fact that Pkg does not specify exactly how a test environment is instantiated in an issue. The docstring of Pkg.test is both insufficiently detailed and outdated: it doesn’t even mention test/Project.toml.
I don’t think I was really aware that Pkg.test takes into account the main Manifest.toml. Not that that actually makes any sense: The main Manifest.toml file is not for the test environment! It doesn’t include the test dependencies in [extras]. If anything, Pkg.test should use test/Manifest.toml when it exists. I have this vague recollection of Pkg.test doing some kind of (unspecified!) “merging” of environments when there are any Manifest files present, but that’s only ever caused errors whenever I encountered it (and the exact behavior may have changed over the years).
This underspecification of how Pkg.test actually instantiates the test environment, and the difficulty in getting it to easily test against specific versions of dependency packages has actually made me eschew Pkg.test altogether for a long time. Instead, I would use my own test() function for running test/runtests.jl in a subprocess. Alas, with the general improvements in built-in tooling, maintaining such custom workflows is becoming more trouble than it’s worth. For “standalone” packages that can use 1.11 or 1.12 for development tasks, there are “standard” workflows at this point that seem to work well enough.
But yes, Pkg.test is still a mess, and we should do better. There should be a way to exactly specify the test environment (presumably, by giving it a full Manifest; none of this “merging environments”). It should also be possible to get a Manifest out of Pkg.test. Presumably, a flag to override the temp-folder where Pkg instantiates the test project would do the trick.
Once you have the ability to get Manifests in and out of Pkg.test, there is still the question of what you do with that. I still wouldn’t commit them. Having them as downloadable artifacts in CI would probably go a long way (although CI logs on GitHub only survive three months, so for long-term archival, you’d have to figure out some archival system).
In any case, yes, you could open issues in Pkg. There probably are some already, for example Document two new ways of specifying test-specific dependencies · Issue #3953 · JuliaLang/Pkg.jl · GitHub. Of course, opening issues doesn’t mean the problem gets solved, so it would probably better to really dive into the code of Pkg.test and prepare a pull request that fully documents its actual current behavior. I’ve had a look at that code, though, and it has lots of layers of abstractions, so this is not such an easy task.