Debugger doesn't stop at breakpoints when running tests (but does during normal execution)

Tooling VS Code
, ,
1

Julia 1.7.3
VS Code 1.74.3
Julia extension 1.38.2

I’m having a problem where, when I run tests with the debugger on the program I’m working on, the debugger doesn’t stop at any breakpoints I set. If I run the program normally (meaning executing a main function) with the debugger, the debugger works as expected and stops. I tried isolating the problem with a small example problem I can share here, however I can’t get the problem to reproduce in the example. Until I do, I suppose my question is which things in particular might be causing this issue and what I should look into.

Example

In the following the example and how I use it. I think it captures the structure and use of the full program well, but evidently I’m missing something.

project_dir
|> deps
  |> build.jl (empty)
|> src
  |> ModuleToTest.jl
  |> PackageToTest.jl
|> test
  |> runtests.jl
|> Manifest.toml
|> Project.toml

ModuleToTest.jl

module ModuleToTest

export do_something

function internal_only(x)
    return x + 1
end

function do_something(x)
    result = internal_only(x)
    return result
end

end

PackageToTest.jl

module PackageToTest

include("ModuleToTest.jl")
using .ModuleToTest

function main()
    print("The magic number is $(ModuleToTest.do_something(41))")
end
main()

end

runtests.jl

using Test
using PackageToTest.ModuleToTest

@testset "tests_are_working" begin
    @test true
end

@testset "test_do_something" begin
    @test ModuleToTest.do_something(41) == 42
end

@testset "test_internal_only" begin
    @test ModuleToTest.internal_only(41) == 42
end

Project.toml

name = "PackageToTest"
uuid = "2e0f99b1-0d8f-4296-8d0c-1c50a50d04c8"
authors = ["author"]
version = "0.1.0"

[deps]
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

Running the example

For debugging the program and testing it I have two debugger settings in launch.json:

{
    "type": "julia",
    "request": "launch",
    "name": "Run file",
    "program": "project_dir/src/PackageToTest.jl",
    "stopOnEntry": false,
    "cwd": "project_dir/",
    "juliaEnv": "${command:activeJuliaEnvironment}",
    "args": []
},
{
    "type": "julia",
    "request": "launch",
    "name": "Run tests",
    "program": "project_dir/test/runtests.jl",
    "stopOnEntry": false,
    "cwd": "project_dir/",
    "juliaEnv": ".",
    "args": []
}

The launch config Run file runs the program with its main function, and will stop at breakpoints. Launching with Run tests will stop at breakpoints in this example, but not in the full project. I don’t use the REPL other than to add dependencies to Project.toml. Running tests from the PKG REPL, by activate . then test PackageToTest, runs the tests and works as normal, but I haven’t figured out how to use the debugger in the REPL for tests.

1 Like
2

An update to our problem. It’s not solved, but I do have a new avenue to pursue. Let’s change the example by moving tests into other files that are being included:

runtests.jl

using Test

@testset "tests_are_working" begin
    @test true
end

include("included_tests.jl")

included_tests.jl

using PackageToTest.ModuleToTest

@testset "test_do_something" begin
    @test ModuleToTest.do_something(41) == 42
end

@testset "test_internal_only" begin
    @test ModuleToTest.internal_only(41) == 42
end

With this modification the example reproduces our problem. With launch config Run tests, the debugger will stop at breakpoints in file runtests.jl but not at breakpoints in any other file that is on the code path of tests, e.g. ModuleToTest.jl or included_tests.jl.

This is now beginning to sound similar to problems other users have found and solved, so I’m hoping I will stumble over a solution there.

3

I have run to what I think were similar issues in this thread

4

It seems possible the problems are related, yes. Unfortunately I don’t have much new to offer as solutions go.

Something that might be related is the “compiled mode” of the debugger. Apparently when enabled it won’t stop at breakpoints (and, perhaps, exceptions) in code files other than the file being executed. So, disabling this mode sounds like something that could solve our issue. I’d love to test it, but I cannot enable compiled mode. I don’t think the feature is broken, so it must be a problem with how we use environments and execute code / tests.

5

For what it’s worth, I have found Infiltrator.jl’s @infiltrate to work even within @testset and to be very helpful for debugging tests (in combination with @exfiltrate if necessary).

6

I tried Infiltrator.jl and while it seems useful, it leads to a much more cumbersome workflow and has limited stepping functionality. I will keep it in mind for when the speedup through compile mode becomes necessary.

However, I also tried using Debugger.jl in the REPL directly without going through the VS Code plugin. In the example above this would be by:

PKG REPL:

activate .
add Debugger

Julia REPL:

using Debugger
@enter include("test/runtests.jl")

This will stop in the include function and open the Debugger REPL:

bp add "./src/ModuleToTest.jl":6
c

This should stop at the line of internal_only’s definition when it is called during the tests, however it doesn’t. What I take from this is that the problem lies with Debugger.jl (or more likely with how we use it) and has nothing do with the plugin’s GUI for the debugger. Once I have more information I’ll take it over to the appropriate place for that.

7

This is also related, I believe. It appears that the authors of the Julia VSCode plugin are already aware of the limitations in debugging while in @testsets, and focus is on the new testitem framework.

8

I have come across the TestItem framework while looking for solutions, but haven’t found the time to check if switching to it might be worth the effort. I have already invested too much time in this, but will earmark it for future reference.

Meanwhile, as I suspected the underlying issue lies with a limitation in Debugger.jl and a solution of sorts was found that allows tests to be debugged. The debugger is intended to be used on functions, not on what is essentially deeply nested top-level code. So wrapping testsets into functions and calling them within the actual @testset macro, still works for the regular test executions, but also allows for debugging individual tests by calling them via @enter test_to_be_debugged() from the REPL.

However, I haven’t yet figured out how to connect a running REPL to the debugger panel in VS Code yet, which I want to do so I can use the GUI it provides for inspecting state at a breakpoint. Once I find out how, or find out it isn’t possible, I’ll update again and mark this as a solution. The underlying debugger issue is solved, but not the problem when using it from the VS Code plugin. I’ll be honest, I still do not care for the REPL.

1 Like
9

An (hopefully final) update to the original question of how to debug tests with the debugging functionality within the VS Code plugin for Julia:

First, as described in the previous post, let’s modify our tests so that the code lives within functions.

included_tests.jl

using Test
using PackageToTest.ModuleToTest

function test_do_something()
    @test ModuleToTest.do_something(41) == 42
end

@testset "test_do_something" begin
    test_do_something()
end

function test_internal_only()
    @test ModuleToTest.internal_only(41) == 42
end

@testset "test_internal_only" begin
    test_internal_only()
end

and we need a debug launch config that runs the active file (I think this is the default config):

launch.jl

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "julia",
            "request": "launch",
            "name": "Run active file",
            "program": "${file}",
            "stopOnEntry": false,
            "cwd": "projectdir/",
            "juliaEnv": "${command:activeJuliaEnvironment}",
            "args": []
        },
    ]
}

Now we can open file included_tests.jl, hit F5 and the debugger will stop at breakpoints in file ModuleToTest.jl and display the values of variables in the debug pane, as well as allowing for the stepping functions to work as expected.

10

I’m trying to get this solution to work on my Windows setup in VSCode, but so far I have no success. I have completely reproduced the setup as suggested above, but I cannot get the debugger to stop at breakpoints in the module code.

In fact, even a breakpoint placed inside the test function at the line where we test ...(41) == 42 is not picked up. I can get the debugger to stop when I put a breakpoint on the test_internal_only() call, but still I cannot step into the function:

function test_internal_only()
    @test ModuleToTest.internal_only(41) == 42
end

@testset "test_internal_only" begin
    test_internal_only()
end

The only way I’m able to actually step into the method under test is to move its evaluation outside of the @test macro, put a breakpoint at the actual = ... line and step in manually with F11:

@testset "test_internal_only" begin
    actual = ModuleToTest.internal_only(41)
    @test actual == 42
end

Still, the debugger doesn’t respect breakpoints in the method under test.

Ultimately, when I’m developing a bigger package I would like to be able to split up the test suite into smaller files and include them in the runtests.jl, but first I’d like the debugger be able to work on this more simple setup. Does anyone have a clue why this doesn’t work, or have a suggestion how to solve this in an alternative manner?

11

I just tried my own instructions (spread across three posts - something to improve) in a new workspace and folder and was able to replicate the correct behaviour when running launch config “Run active file” on file “included_tests.jl”.

Julia v1.8.5
VS Code v1.80.1
VS Code Julia plugin v1.47.2

Of note is that I used a new julia environment* and VS Code did ask me to switch my julia enviornments. I have to confess that I still don’t really environments, so I can’t say if that has any impact.

* By doing:
julia ]
activate .
add Test

12

Ah! I somehow got your example to work as well, I think there was a mistake in the launch.json file (just removing it completely fixed it).

It would be a nice to have to also be able to debug from a main test script (like in your runtests.jl with one or more includes), but the above is very helpful to have at least something. Thanks :slight_smile: