TestItemRunner doesn't reload changed test case

Tooling VS Code
1

I am using Julia in VS Code and running individual tests with TestItemRunner and the @testitem macro. I am finding that if a @test case line is changed just a bit, e.g. changing a number by a single digit, the change doesn’t seem to be picked up and the test is re-run as if nothing had changed.

Very minimal example:

TestPkg.jl:

module TestPkg

export foobar

foobar() = 3

end

my_tests.jl

@testitem "My tests" begin
    @test foobar() == 3
end

When I run that test via the testing pane, it passes. But I can change the 3 to a 4 and the test will still pass. If I change it to 44 it will then fail.

Is this a bug, or perhaps something that can be configured to make it more “sensitive” to changes?

2

I also noticed something similar recently with correct functionality being recovered when deleting and then reintroducing the specific changed line

3

Glad to know it’s not just me - and I did find the same behaviour with removing the entire line. It’s a bit of a gotcha that it’s not actually referring to the test file on disk when running the test, but instead relying on a cache that clearly only updates when at least a few keystrokes have been made.