VS Code won't run tests anymore

[Ken_Williams]

My test pane (with the chemistry beaker icon) in VS Code won’t run tests anymore, as of this week. When I click on the “run” button, for all of the tests together or just a single test, it seems to just hang forever.

Running tests in a normal REPL by using ] test works fine, but is of course much slower and doesn’t allow running a single test by point-and-click.

My tests are @testitem style, so they look similar to this:

@testitem "fill_missing!()" begin
    using DataFrames

    df = DataFrame(; A = [1, 2, missing, 4], B = [missing, 2, 3, 4])

    @test_logs (:warn, r"Filling in 1 A value") begin
        MyPackage.fill_missing!(df, :A)
    end
    @test df.A == [1, 2, 0.0, 4]

    @test_logs (:warn, r"Filling in 1 B value") begin
        MyPackage.fill_missing!(df, :B; value = 1.0)
    end
    @test df.B == [1.0, 2, 3, 4]

    @test_logs (:warn, r"No C column found in my DF; using default") begin
        MyPackage.fill_missing!(df, :C; value = 2.0, name = "my DF")
    end
    @test df.C == [2.0, 2.0, 2.0, 2.0]
end

In trying to diagnose what’s going wrong, I can’t seem to find much info. In the “output” pane, I see the following for the Test Item Controller:

[ Info: Starting test item controller on Julia 1.11.5
  Activating project at `~/.vscode/extensions/julialang.language-julia-1.140.2/scripts/environments/testitemcontroller/v1.11`
Precompiling TestItemControllers...
   1632.5 ms  ✓ TestItemControllers
  1 dependency successfully precompiled in 2 seconds. 5 already precompiled.

Nothing in any of the other “output” selections seems to have anything relevant.

Is there some way to invoke the Testing pane action manually, e.g. in a vanilla REPL? Or to convince VS Code to run with debug logging turned on, or something? In general, any suggestions for how to get more information about what might be going wrong?

[adolgert]

There is a way to run it in a vanilla REPL. I’m not sure what other people do, but I use a small file I call “cli.py”:

using ArgParse
using TestEnv
using TestItemRunner
using YourPackageNameHere

# For running tests from a Bash prompt.
# julia --project=test test/cli.jl [--file <substring>] [--function <substring>]
# You need to ensure the test/Project.toml is set to Plg.develop(UnitTestDesign)
# so that it can load that in the activate() below.

function parse_testcli(env_args)
    settings = ArgParseSettings()
    @add_arg_table settings begin
        "--file"
            help = "File to test"
            arg_type = String
            required = false
        "--name"
            help = "Name of the test to run"
            arg_type = String
            required = false
    end
    return parse_args(env_args, settings)
end


TestEnv.activate("YourPackageNameHere") do
    args = parse_testcli(ARGS)
    matches = Function[ti -> true]
    if args["file"] !== nothing
        push!(matches, ti -> endswith(join(splitext(ti.filename)[1:end-1]), args["file"]))
    end
    if args["name"] !== nothing
        push!(matches, ti -> occursin(args["name"], ti.name))
    end
    @run_package_tests filter=ti -> all(map(f -> f(ti), matches))
end

Maybe somebody else knows about getting the VSCode one to work.
Cheers - Drew

[disberd]

I have also experienced a similar issue in the last week, though not on every project I have which is even weirder.

Unfortunately as in your case @Ken_Williams the output pane is not giving much information (I only have the activating part).

I wonder whether there is a way (e.g. with a setting or flag) to give more debug output.

Tagging @davidanthoff and @pfitzseb as they might have some obvious insights.

Edit: it seems there is also a related issue opened a few hours ago

TestItems testing stuck on (Queued)

[Ken_Williams]

@adolgert

There is a way to run it in a vanilla REPL. I’m not sure what other people do, but I use a small file I call “cli.py”:

I do have a way to run my tests, basically the same idea as yours but without the CLI processing:

using TestItemRunner

# Filter out the TestItemRunner tests, see
# https://github.com/julia-vscode/TestItemRunner.jl/issues/94
function my_filter(ti)
    return contains(dirname(ti.filename), r"MyPackageName(\.jl)?/test")
end
@run_package_tests filter = my_filter

However, this doesn’t exhibit the hanging behavior, it succeeds fine.

What I think I’d need for this is a way to run what VS Code itself is doing as a part of the Julia plugin.

[jClugstor]

I think TestItemRunner is what you’re looking for: Test Item Framework · Julia in VS Code

[Ken_Williams]

@jClugstor

I think TestItemRunner is what you’re looking for: Test Item Framework · Julia in VS Code

I don’t think so. See the above messages. That doesn’t appear to be what VS Code is using for the Testing pane, or at least when I run the tests using TestItemRunner, it doesn’t hang like it does in the Testing pane.

[Ken_Williams]

As an experiment, I quit VS Code, deleted my ~/Library/Application Support/Code/User/globalStorage/julialang.language-julia directory, and re-launched VS Code.

No difference, it still hangs after re-creating that directory.

[Ken_Williams]

I did a little debugging after finding out what code seems to be involved. It’s a little tough because it’s doing socket communication and I’m not sure what’s on the other end of the socket, but here’s some info.

I put ENV["JULIA_DEBUG"] = "all" into ~/.vscode/extensions/julialang.language-julia-1.140.2/scripts/apps/testitemcontroller_main.jl:

...
ENV["JULIA_DEBUG"] = "all"
using Logging
global_logger(ConsoleLogger(stderr))
...

and then put some extra debugging messages into ~/.vscode/extensions/julialang.language-julia-1.140.2/scripts/packages/TestItemControllers/src/testitemcontroller.jl:

...
function Base.run(
    controller::TestItemController,
    testprocess_created_callback,
    testprocess_terminated,
    testprocess_statuschanged,
    testprocess_output
)
    @debug "Initiated Base.run()"
    while true
        @debug "Listening for message from $(controller.msg_channel)"
        msg = take!(controller.msg_channel)
        @debug "Controller msg" msg
...

In VS Code’s output pane, I now see this:

┌ Debug: Initiated Base.run()
└ @ TestItemControllers ~/.vscode/extensions/julialang.language-julia-1.140.2/scripts/packages/TestItemControllers/src/testitemcontroller.jl:67
┌ Debug: Listening for message from Channel{Any}(9223372036854775807)
└ @ TestItemControllers ~/.vscode/extensions/julialang.language-julia-1.140.2/scripts/packages/TestItemControllers/src/testitemcontroller.jl:69

and then no more logs. So it appears to be hanging on receiving the input from the socket, but I’m not sure what that means.

[MackHarry]

Same problem. Activating any testitem style test via the gui sends the test pane into some endlessly hanging process, while running the tests from REPL is fine.Only getting the ‘activating…’ output from the test item control. Not the first idea how to go about debugging this.

[kbarros]

As a workaround, you can try installing julia-vscode extension version 1.138.1, which is from 4 months ago. From the Extensions pane, you can click “Uninstall” drop-down, then “Install Specific Version…” as shown in the image.

[MackHarry]

thank you! There only was 1.127.2 available (the newest version other than the current one), but using it tests work again, and makes dev life a lot easier

[Ken_Williams]

Ooh, reverting to 1.127.2 works for me too, thanks for looking into that @kbarros .

[disberd]

The latest vscode extension was confirmed to have broken testitems also on slack

Slack