does anyone know if it is possible to add additional package dependencies to @testitems which are usually not executed due to filters in a @run_package_tests?
To clarify:
I do not want to burden users with too many tests but the unit tests. More complicated tests are included as test items, but are only ever executed by me in Vscode. Because users don’t usually need dependencies for more complicated tests, there are not included in the test Project.toml.
This, however, causes TestItemRunner to fail in Vscode for those tests.
Does anyone know of a work around, or a proper way to include such dependencies without just directly adding them even though a user might never need those packages? (I have looked into the documentation but haven’t found anything.)
Thanks in advance.
PS: If anyone requires a use case for some reason, this kind of stuff is especially useful for testing package extensions.
Addendum: Now that I finally asked the question, it kind of dawned on me that I can simply add a second test folder, or even a test folder within each individual extension folder. That said, to me that kind of defeats the purpose of filtering test items, doesn’t it? Maybe those, who extensively use test items, can enlighten me and share with me how they use it?
Hi,
I’m glad you found a workaround by making a second directory, and that makes sense to me. People put all sorts of dependencies into the Test environment because users won’t need to load it when they use the main package. As you say, that makes it harder to test different package extensions.
You said you’re using TestItems.jl. Maybe you could make a @testsnippet that checks for the presence of a set of extensions? Then individual tests could run conditionally on whether the extensions are loaded. If they are there, then run the test. If they aren’t, then skip the test. Then you could put a step into CI/CD that does testing with different extensions loaded.
Or maybe I’m missing your point. But I agree it isn’t a case that has a lot of support right now.
Hope All Is Well - Drew
The problem isn’t really about checking if an extension is loaded or not and then conditionally executing a test. Maybe I didn’t explain well enough.
It’s about not needlessly adding all the packages into the test dependencies that would be required to properly test extensions on them. Not all users are going to use all extensions, so why should they have to run an extensive test suite that covers all extensions? In particular, why should they need to setup the test environment by downloading and precompiling all those dependencies?
It would be nice if we could define separate test environments with separate dependencies, so users will be required to download and precompile only those dependencies they are already using. This would decrease their wait times on the test results.
I am not saying that is a problem currently, I don’t know how many packages are implementing how many extensions. Or even if at all, any users are actually executing the tests… probably not many? None?
In case of CI/CD, there is no restriction to adding whatever is required for a full test coverage.
Maybe this is a non-issue and it just bugs me personally.
PS: I just tried and made a tests folder parallel to test with its own dependencies as proposed in my OP. TestItems.jl always chooses the environment defined in test, I guess? Regardless of where the tests are located (inside a separate environment or even inside the module intself), ultimately causing a “package not defined” error.
Ah, I hear you. I think what you want is possible. There are two ways to specify a test environment in Julia, using either a separate Project.toml in the test/ directory or a separate dependency list in the main Project.toml. Both support what you want, but there isn’t a test runner, that I know of, that will by default call tests in a different directory. You ran into a limitation of the VSCode testing running, but Pkg.test() has the same limitation. It’s not intrinsic, though, to TestItems.jl. That package is hilariously small. It defines only a set of empty macros.
While I haven’t heard of anybody using this feature, it’s supported by the separate Project.toml environments. You could probably add a little code to TestItemRunner.jl or to ReTest.jl to make it work. Good luck! - Drew
