Prerelease of new testing framework and test run UI in VS Code

[davidanthoff]

@aplavin

The requirement to replace @testset with @testitem does bring some friction, though. Together with the two extra dependencies. I imagine this would significantly slow down the potential adoption among existing packages.

I should probably clarify this a bit: I think users will really only have to deal with the TestItems package if they want to write tests inline with their normal package code. In that case you have to add TestItems as a regular package dependency so that the @testitem macro is available inside your regular package code. But, if you do not plan to use these inline tests, then you don’t need the dependency on TestItems at all.

TestItemRunner is the package you need as a test dependency, and is essentially only used inside test/runtests.jl. When converting an existing package, really one will start by adding that to the Project.toml [test] section. TestItemRunner reexports @testitem, so then that is available also in say test/runtests.jl.

@aplavin

Can it be made to work with top-level @testset entries as well?

It thought a fair bit about it, and in the end I decided against it for a couple of reasons:

1. the real work in converting a test suite from @testset to @testitem is to make sure the @testitems are self-contained, the renaming part seemed very little work to me, literally search-replace should do it.
2. If we start to just pretend that @testsets can be run individually and surface that in the VS Code UI, then I can bet that thousands of users will start to use that on their existing test suite and nothing will work. And I think they would rightly say “but hey, my tests are written in the style that Base tells me to”, and then we would have to say “yeah, but if you want to run them inside VS Code, you have to remember…” So I think it would create too much confusion, support requests etc
3. One really nice thing about this syntactic detection story is how absolutely trivial the @testitem macro is. The whole system is actually really, really simple. In terms of dev work for me/us that is a major benefit and I didn’t want to complicate things by brittle integration with an existing test framework that is just not a super good match.

@phlavenk

All the @testsets hierarchy is in the included files - I’ve not touched them. Yes - you can have a much nicer granularity of the @testsets in the UI if you replace them in those concrete test files with @testitem, but even my example brings already the productivity boost.

Yep, that is an important point! You can still use @testsets inside @testitems and slowly and gradually split things into more granular and smaller @testitems over time!

@lmiq

When I click “play” on this test, it doesn’t do anything (the thing there keeps spinning forever).

I don´t see any output anywhere to try to debug what’s going on.

You can click the button in a red circle in this screenshot:

That should show a terminal with all output from the test run process. Does that show anything informative? Generally what you describe sounds like a bug on our end to me

[carstenbauer]

Great stuff! Unfortunately, I wasn’t able to run a first simple test on a remote server with a custom julia executable path setting (it just hangs forever with a spinning circle). Is the latter being respected when spinning up the test processes?

FWIW, the test runner output (obtained by clicking on the small square highlighted in red above) is just empty.

[davidanthoff]

@carstenbauer

Unfortunately, I wasn’t able to run a first simple test on a remote server with a custom julia executable path setting (it just hangs forever with a spinning circle). Is the latter being respected when spinning up the test processes?

It should in theory work, but it sounds like it doesn’t. Could you maybe paste the exact setting you have for your executable here? Maybe something jumps out. Or probably even better open an issue on the extension Github for this, clearly a bug.

[lmiq]

@davidanthoff

That should show a terminal with all output from the test run process. Does that show anything informative?

No, it doesn’t show anything. I’ll try to get more info. Is there any verbose mode or log where I may find anything?

Also I’ll try to build a mwe and then file an issue appropriately.

[lmiq]

@davidanthoff

I have the problem of the “play” button not doing anything in this package, which is only the template plus the tests: GitHub - lmiq/TestTests.jl

I’m on Linux x86 - with Julia 1.8.0.

There is no output at all, as the figure above shows, the spinning stuff just keeps spinning.

Opened the issue here: @testitem does not run · Issue #19 · julia-vscode/TestItemRunner.jl · GitHub

To try to reproduce the problem, do:

git clone https://github.com/lmiq/TestTests.jl
cd TestTests.jl
code .

VSCode will open. Wait for the tests to appear at the Erlenmeyer tab, and click play.

[brianguenter]

This will be fantastic when all the bugs get ironed out. Thank you for developing this.

Unfortunately, it doesn’t seem to be working for me. Using this MWE

module Differentiation
using TestItems, TestItemRunner

@testitem "First tests" begin
    @test true
end

end # module 

I also get the circle spinning, though not forever. Eventually it crashes:

System info:

Julia language server v1.7.6

VSCode version

Version: 1.70.2 (user setup)
Commit: e4503b30fc78200f846c62cf8091b76ff5547662
Date: 2022-08-16T05:35:13.448Z
Electron: 18.3.5
Chromium: 100.0.4896.160
Node.js: 16.13.2
V8: 10.0.139.17-electron.0
OS: Windows_NT x64 10.0.22000

Julia version

julia> versioninfo()
Julia Version 1.8.0
Commit 5544a0fab7 (2022-08-17 13:38 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 16 × 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, tigerlake)
  Threads: 8 on 16 virtual cores
Environment:
  JULIA_EDITOR = code.cmd
  JULIA_NUM_THREADS = 8

[TI36XPro]

I am getting the same behavior. Endless spinning with nothing in the terminal output. I am running Ubuntu on Windows Subsystem for Linux 1 (WSL1).

[lmiq]

The bug is being worked on. A workaround is to change the Julia extension version to “Julia Insiders” and make the path to the executable of Julia point to the actual application in the extension settings (I. E. with the full path, not just “julia”).

That said, the feature is amazing. I am completely into it, even in this experimental scenario. I’m just waiting for the blessing of David to release package versions with the new testing framework.

[davidanthoff]

Alright, I pushed out a new build that should fix the problem of tests never running. Thanks to @lmiq who helped debug this!

If you had tried this before and weren’t able to run tests, please try again with this new build, version 1.7.10.

If things still don’t work for you, please let me know, then you’ve found a new bug! Ideally open a new issue in that case over at GitHub - julia-vscode/julia-vscode: Julia extension for Visual Studio Code.

[lmiq]

For me this is one of the best additions in terms of development workflow and code quality. Having the possibility of adding tests just besides the code and run this tests independently is awesome. Thank you very much for that.

[brianguenter]

I just ran this again and still get the wheel spinning a long time which eventually causes a crash. Same environment as before , but Julia language server is now 1.7.10 and VSCode version is 1.71.0.

[stephenll]

I just installed and tried for the first time and receive a crash after waiting for a bit. I’m on Julia 1.8.0, MacOS M1 Pro. I also tried setting the Julia path manually with no success.

[lmiq]

@brianguenter @stephenll

There are now, at the Output tab, the option to choose Julia TestServer, Julia, and Julia Language Server logs. If you can post the content of those, that helps debugging the issues.

Like this, for example:

[stephenll]

Before seeing this, I tried a few more changes to see if I can get this working. Maybe this is something I should know but I’m a very occasional VS Code user: I had to add Test and TestItems to my package to get this to work.

It might also be relevant: I don’t have the Test package in the package directory I am working on. I add it to a project in the Test directory. To be clear, the Test package exists in a project.toml in the Test directory. To get this to work, I had to add both Test and TestItems to the pkg directory.

I hope this helps.

[lmiq]

As far as I understand, Test shouldn’t be needed. TestItems yes, if you want tests mixed with the package code. If all tests are in the runtests.jl file, then there you should load TestItemsRunner (and Tests, I’m not completely sure about this one, though).

[stephenll]

I just created a very simple package, no deps except for TestItems and didn’t add Test this time. I added using TestItems in my module and use the following test code:

@testitem "My First Test" begin
    x = [1,2,3]
    @test length(x)==3
end

In this simple pkg I am defining a few structs, and assigning values to the fields. I also don’t have a Test directory. I tried just using the TestItems feature to do a basic test.

Below is the error in Output:

┌ Error: Some Julia code in the VS Code extension crashed
└ @ Main ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/error_handler.jl:15
ERROR: ArgumentError: Package Test not found in current path.
- Run `import Pkg; Pkg.add("Test")` to install the Test package.
Stacktrace:
  [1] macro expansion
    @ ./loading.jl:1163 [inlined]
  [2] macro expansion
    @ ./lock.jl:223 [inlined]
  [3] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1144
  [4] eval
    @ ./boot.jl:368 [inlined]
  [5] run_testitem_handler(conn::VSCodeTestServer.JSONRPC.JSONRPCEndpoint{Base.PipeEndpoint, Base.PipeEndpoint}, params::VSCodeTestServer.TestserverRunTestitemRequestParams)
    @ VSCodeTestServer ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/packages/VSCodeTestServer/src/VSCodeTestServer.jl:52
  [6] dispatch_msg(x::VSCodeTestServer.JSONRPC.JSONRPCEndpoint{Base.PipeEndpoint, Base.PipeEndpoint}, dispatcher::VSCodeTestServer.JSONRPC.MsgDispatcher, msg::Dict{String, Any})
    @ VSCodeTestServer.JSONRPC ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/packages/JSONRPC/src/typed.jl:67
  [7] serve_in_env(conn::Base.PipeEndpoint)
    @ VSCodeTestServer ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/packages/VSCodeTestServer/src/VSCodeTestServer.jl:136
  [8] #13
    @ ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/packages/VSCodeTestServer/src/VSCodeTestServer.jl:147 [inlined]
  [9] (::VSCodeTestServer.TestEnv.var"#2#3"{VSCodeTestServer.var"#13#15"{Base.PipeEndpoint}})()
    @ VSCodeTestServer.TestEnv ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/packages/TestEnv/src/julia-1.8/activate_do.jl:18
 [10] withenv(::VSCodeTestServer.TestEnv.var"#2#3"{VSCodeTestServer.var"#13#15"{Base.PipeEndpoint}}, ::Pair{String, String}, ::Vararg{Pair{String}})
    @ Base ./env.jl:172
 [11] (::Pkg.Operations.var"#107#112"{String, Bool, Bool, Bool, VSCodeTestServer.TestEnv.var"#2#3"{VSCodeTestServer.var"#13#15"{Base.PipeEndpoint}}, Pkg.Types.PackageSpec})()
    @ Pkg.Operations /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1619
 [12] with_temp_env(fn::Pkg.Operations.var"#107#112"{String, Bool, Bool, Bool, VSCodeTestServer.TestEnv.var"#2#3"{VSCodeTestServer.var"#13#15"{Base.PipeEndpoint}}, Pkg.Types.PackageSpec}, temp_env::String)
    @ Pkg.Operations /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1493
 [13] (::Pkg.Operations.var"#105#110"{Nothing, Bool, Bool, Bool, VSCodeTestServer.TestEnv.var"#2#3"{VSCodeTestServer.var"#13#15"{Base.PipeEndpoint}}, Pkg.Types.Context, Pkg.Types.PackageSpec, String, Pkg.Types.Project, String})(tmp::String)
    @ Pkg.Operations /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1582
 [14] mktempdir(fn::Pkg.Operations.var"#105#110"{Nothing, Bool, Bool, Bool, VSCodeTestServer.TestEnv.var"#2#3"{VSCodeTestServer.var"#13#15"{Base.PipeEndpoint}}, Pkg.Types.Context, Pkg.Types.PackageSpec, String, Pkg.Types.Project, String}, parent::String; prefix::String)
    @ Base.Filesystem ./file.jl:764
 [15] mktempdir(fn::Function, parent::String) (repeats 2 times)
    @ Base.Filesystem ./file.jl:760
 [16] sandbox(fn::Function, ctx::Pkg.Types.Context, target::Pkg.Types.PackageSpec, target_path::String, sandbox_path::String, sandbox_project_override::Pkg.Types.Project; preferences::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool)
    @ Pkg.Operations /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1540
 [17] sandbox
    @ /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1531 [inlined]
 [18] activate(f::VSCodeTestServer.var"#13#15"{Base.PipeEndpoint}, pkg::String)
    @ VSCodeTestServer.TestEnv ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/packages/TestEnv/src/julia-1.8/activate_do.jl:16
 [19] serve(conn::Base.PipeEndpoint, project_path::String, package_path::String, package_name::String; is_dev::Bool, crashreporting_pipename::Nothing)
    @ VSCodeTestServer ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/packages/VSCodeTestServer/src/VSCodeTestServer.jl:146
 [20] serve(conn::Base.PipeEndpoint, project_path::String, package_path::String, package_name::String)
    @ VSCodeTestServer ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/packages/VSCodeTestServer/src/VSCodeTestServer.jl:140
 [21] top-level scope
    @ ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/testserver/testserver_main.jl:16
 [22] include(mod::Module, _path::String)
    @ Base ./Base.jl:419
 [23] exec_options(opts::Base.JLOptions)
    @ Base ./client.jl:303
 [24] _start()
    @ Base ./client.jl:522

[lmiq]

Here is a working case: GitHub - lmiq/TestTests.jl

• You do need to ] add Tests, yes (@davidanthoff can this be lifted? Seems redudant, isn’t Test a dependency of TestItemRunner, which could reexport the Test macros?)
• In the main module of the package you only need to put using TestItems
• In runtests.jl you need to use using TestItemRunner

The Project.toml must look like this one: TestTests.jl/Project.toml at main · lmiq/TestTests.jl · GitHub

I tested removing the test/runtests.jl file, and the test items inside the main module continue to work.

Does this work for you? You can clone that package and check, of course.

In other works, a step by step would be:

Step by step

Packages to add:

import Pkg; Pkg.add("Test", "TestItems", "TestItemRunner")

Project.toml (see example here)

• Move TestItemRunner and Test to [extras]
• On [targets], have test = ["Test", TestItemRunner"] (and other deps if needed).

Main code of the package

Add using Testitems in the main module, and @testitem(s):

module MyPackage
    using TestItems
    f(x) = 1

    @testitem "my test" begin
        using MyPackage # must be self-contained!
        @test MyPackage.f(1) == 1
    end
end

The test/runtests.jl file:

Must look like this:

using TestItemRunner

# other items can be here
@testitem "my other test" begin
    using MyPackage
    @test MyPackage.f(2) == 1
end

@run_package_tests

[stephenll]

Thanks for putting the Pkg example together! That works. I wasn’t aware that I need to add the Test and TestItems to my package. I usually have a Test project inside the ./Test folder rather than add Test package as a dep to my package. Thanks again.

[juliohm]

Thank you @davidanthoff and team for improving the development experience in Julia Looking forward to trying it out. We have very large test sets that will definitely benefit from this new workflow.

[lmiq]

@stephenll

Test package as a dep to my package

Note that you only add it to the extras, so it won’t be loaded when a user uses your package.