I haven’t been able to find a reference about how the Testing pane in VS Code works for a Julia project’s unit tests. I’ve fiddled around a bit and tried various of
file layouts (test_*.jl vs. *_test.jl, putting in a MyTestModule, etc.)
setup code (add stuff in runtests.jl, put it in a file that you include() in each test, use some runner-specific thing like @testsetup, etc.)
and I’ve converged on something that works sort of okay, but I feel like it should be documented somewhere what the Testing pane actually does, and how it finds its tests (is it finding them itself, or are the various runner frameworks feeding it the test info) and what happens when you click the “run” buttons.
Of course, feel free to point me to docs if they exist - I looked in Home · Julia in VS Code .
I’ve seen TestItemRunner and tried it but couldn’t get it to work with the GUI. That’s one reason I’m asking what the Testing pane is actually doing, so I can see if I’m complying with its expectations.
It’s a little hard for me to remember everything that wasn’t working, because I was flailing around trying to get things working, trying things kind of at random. And unfortunately this is a proprietary codebase so it would take some time to distill it down to something I could share.
But I think the main things I was having trouble with were
With that step by step you should get things working. There are some things there that are not needed (using package inside each test item), but do no harm.
Another thing that confused me is that (because I had read @davidanthoff 's post) I thought that the only way to get the Testing pane to show tests (and be able to run them individually, etc.) was to start using the “test items” stuff.
But then yesterday a co-worker showed me his setup where he’s using ReTestItems and that populates the Testing pane (but only if you name your files a certain way…), so I experimented with that a bit today and verified that I can get it working. But I’m left a bit confused about what’s going on. For example - when using ReTestItems, does my runtests.jl enter into the picture, or is it just running tests directly?
And what’s all this I see in the console output about a “test server instance”? Is that explained anywhere I could read about? I wouldn’t really need to know much about it, except that it sometimes seems to run tests from the root directory /, causing them to fail when loading files using relative paths.
If you could reproduce that it would be great! At the moment I’m not aware of any bugs in the test detection logic.
Yes, that feature is just not finished. For now, the only real option to share code between different @testitems is to put it into a separate file, and then include that file from each @testitem.
Hm, I think that is a bug, just looked at the code. Fix incoming.
I don’t understand that, could you expand a bit?
So ReTestItems is an independent test runner developed by Relational AI, that also happens to run @testitems. Think of it as an alternative to TestItemRunner. Now, ReTestItem has slightly different features than what I introduced as the “test item framework” originally… Some extra ones and some things are just a bit different relative to TestItemRunner (for example you have to put tests into files with a specific filename pattern). I think it would probably have been better to just give it a different name completely distinct from “test item” to make it clear that it is something different, but hey, here we are
As far as I know ReTestItems has no VS Code integration at all. So whatever you are seeing showing up in the testing UI in VS Code is just the official “test item” feature in VS Code.
That is an implementation detail, it is just the name of the Julia process in which the test code is running. Re working directory, see my response above.
Well, I’m not sure if it was a bug or not - since there didn’t seem to be any docs for the Testing Pane I couldn’t tell what was supposed to happen, if I was doing anything wrong or not. I’ve reorganized my tests a bunch and now it seems to be behaving well for the past couple weeks, so it must have been on my end.
That’s what I’ve switched to, it seems to be working fine, thanks.
Has that fix been released? Would it be a fix in the VS Code extension? I don’t see it in the Changelog but I’m not sure if that’s kept up to date.
It might be a moot point until the initialization stuff is ironed out more. But what I meant was that since the mechanism for initialization code (@testsetup) is to put code in a module, any using statements in that module won’t transitively export their symbols to the test code. So while I can stick using Foo in a file that’s included in a test and that gives me access to Foo.* symbols, that’s not true for stuff in a @testsetup module and that makes it a bit less useful.
I think it needs to be mentioned - since it wasn’t mentioned anywhere, I didn’t know that it ran using a persistent server, and in parallel, and was therefore an order of magnitude faster than using ] test, so I didn’t know that I should try to get my tests visible in that pane.
It’s also an important fact to understand because sometimes the server seems to get out of sync with the code being tested, and things need to be torn down & rebuilt to get fixed. The Developer: Reload Window action has been useful for this, not sure if there’s a better way.
There is one known problem where we can get out of sync (namely updates to project or manifest files, I have a branch where that is fixed, but it will take a while to get that really ready), other than that there should never be a need to manually tear or restart anything, that should all happen automatically without any user intervention.
For this one, you can export individual symbols needed from the testsetup modules. If you want to re-export entire packages, there is Reexport.jl. I haven’t tested this.
Is this fix (wrong pwd while running tests) likely to be released soon? I’ve been keeping an ad-hoc cd(...) in my Git stash for a while, but if it’s going to be a while for the fix, I’ll probably need to come up with a better workaround that the whole team can use.
It might take a while, I’m on vacation Having said that, I would generally try to write tests in a way that they don’t depend on the working directory. Using things like @__DIR__ and @__FILE__ is good practice for tests.
