Prerelease of new testing framework and test run UI in VS Code

Tooling VS Code
21

I just created a very simple package, no deps except for TestItems and didn’t add Test this time. I added using TestItems in my module and use the following test code:

@testitem "My First Test" begin
    x = [1,2,3]
    @test length(x)==3
end

In this simple pkg I am defining a few structs, and assigning values to the fields. I also don’t have a Test directory. I tried just using the TestItems feature to do a basic test.

Below is the error in Output:

┌ Error: Some Julia code in the VS Code extension crashed
└ @ Main ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/error_handler.jl:15
ERROR: ArgumentError: Package Test not found in current path.
- Run `import Pkg; Pkg.add("Test")` to install the Test package.
Stacktrace:
  [1] macro expansion
    @ ./loading.jl:1163 [inlined]
  [2] macro expansion
    @ ./lock.jl:223 [inlined]
  [3] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1144
  [4] eval
    @ ./boot.jl:368 [inlined]
  [5] run_testitem_handler(conn::VSCodeTestServer.JSONRPC.JSONRPCEndpoint{Base.PipeEndpoint, Base.PipeEndpoint}, params::VSCodeTestServer.TestserverRunTestitemRequestParams)
    @ VSCodeTestServer ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/packages/VSCodeTestServer/src/VSCodeTestServer.jl:52
  [6] dispatch_msg(x::VSCodeTestServer.JSONRPC.JSONRPCEndpoint{Base.PipeEndpoint, Base.PipeEndpoint}, dispatcher::VSCodeTestServer.JSONRPC.MsgDispatcher, msg::Dict{String, Any})
    @ VSCodeTestServer.JSONRPC ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/packages/JSONRPC/src/typed.jl:67
  [7] serve_in_env(conn::Base.PipeEndpoint)
    @ VSCodeTestServer ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/packages/VSCodeTestServer/src/VSCodeTestServer.jl:136
  [8] #13
    @ ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/packages/VSCodeTestServer/src/VSCodeTestServer.jl:147 [inlined]
  [9] (::VSCodeTestServer.TestEnv.var"#2#3"{VSCodeTestServer.var"#13#15"{Base.PipeEndpoint}})()
    @ VSCodeTestServer.TestEnv ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/packages/TestEnv/src/julia-1.8/activate_do.jl:18
 [10] withenv(::VSCodeTestServer.TestEnv.var"#2#3"{VSCodeTestServer.var"#13#15"{Base.PipeEndpoint}}, ::Pair{String, String}, ::Vararg{Pair{String}})
    @ Base ./env.jl:172
 [11] (::Pkg.Operations.var"#107#112"{String, Bool, Bool, Bool, VSCodeTestServer.TestEnv.var"#2#3"{VSCodeTestServer.var"#13#15"{Base.PipeEndpoint}}, Pkg.Types.PackageSpec})()
    @ Pkg.Operations /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1619
 [12] with_temp_env(fn::Pkg.Operations.var"#107#112"{String, Bool, Bool, Bool, VSCodeTestServer.TestEnv.var"#2#3"{VSCodeTestServer.var"#13#15"{Base.PipeEndpoint}}, Pkg.Types.PackageSpec}, temp_env::String)
    @ Pkg.Operations /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1493
 [13] (::Pkg.Operations.var"#105#110"{Nothing, Bool, Bool, Bool, VSCodeTestServer.TestEnv.var"#2#3"{VSCodeTestServer.var"#13#15"{Base.PipeEndpoint}}, Pkg.Types.Context, Pkg.Types.PackageSpec, String, Pkg.Types.Project, String})(tmp::String)
    @ Pkg.Operations /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1582
 [14] mktempdir(fn::Pkg.Operations.var"#105#110"{Nothing, Bool, Bool, Bool, VSCodeTestServer.TestEnv.var"#2#3"{VSCodeTestServer.var"#13#15"{Base.PipeEndpoint}}, Pkg.Types.Context, Pkg.Types.PackageSpec, String, Pkg.Types.Project, String}, parent::String; prefix::String)
    @ Base.Filesystem ./file.jl:764
 [15] mktempdir(fn::Function, parent::String) (repeats 2 times)
    @ Base.Filesystem ./file.jl:760
 [16] sandbox(fn::Function, ctx::Pkg.Types.Context, target::Pkg.Types.PackageSpec, target_path::String, sandbox_path::String, sandbox_project_override::Pkg.Types.Project; preferences::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool)
    @ Pkg.Operations /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1540
 [17] sandbox
    @ /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1531 [inlined]
 [18] activate(f::VSCodeTestServer.var"#13#15"{Base.PipeEndpoint}, pkg::String)
    @ VSCodeTestServer.TestEnv ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/packages/TestEnv/src/julia-1.8/activate_do.jl:16
 [19] serve(conn::Base.PipeEndpoint, project_path::String, package_path::String, package_name::String; is_dev::Bool, crashreporting_pipename::Nothing)
    @ VSCodeTestServer ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/packages/VSCodeTestServer/src/VSCodeTestServer.jl:146
 [20] serve(conn::Base.PipeEndpoint, project_path::String, package_path::String, package_name::String)
    @ VSCodeTestServer ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/packages/VSCodeTestServer/src/VSCodeTestServer.jl:140
 [21] top-level scope
    @ ~/.vscode/extensions/julialang.language-julia-1.7.10/scripts/testserver/testserver_main.jl:16
 [22] include(mod::Module, _path::String)
    @ Base ./Base.jl:419
 [23] exec_options(opts::Base.JLOptions)
    @ Base ./client.jl:303
 [24] _start()
    @ Base ./client.jl:522
22

Here is a working case: GitHub - lmiq/TestTests.jl

  • You do need to ] add Tests, yes (@davidanthoff can this be lifted? Seems redudant, isn’t Test a dependency of TestItemRunner, which could reexport the Test macros?)
  • In the main module of the package you only need to put using TestItems
  • In runtests.jl you need to use using TestItemRunner

The Project.toml must look like this one: TestTests.jl/Project.toml at main · lmiq/TestTests.jl · GitHub

I tested removing the test/runtests.jl file, and the test items inside the main module continue to work.

Does this work for you? You can clone that package and check, of course.

In other works, a step by step would be:

Step by step

Packages to add:

import Pkg; Pkg.add("Test", "TestItems", "TestItemRunner")

Project.toml (see example here)

  • Move TestItemRunner and Test to [extras]
  • On [targets], have test = ["Test", TestItemRunner"] (and other deps if needed).

Main code of the package

Add using Testitems in the main module, and @testitem(s):

module MyPackage
    using TestItems
    f(x) = 1

    @testitem "my test" begin
        using MyPackage # must be self-contained!
        @test MyPackage.f(1) == 1
    end
end

The test/runtests.jl file:

Must look like this:

using TestItemRunner

# other items can be here
@testitem "my other test" begin
    using MyPackage
    @test MyPackage.f(2) == 1
end

@run_package_tests
4 Likes
23

Thanks for putting the Pkg example together! That works. I wasn’t aware that I need to add the Test and TestItems to my package. I usually have a Test project inside the ./Test folder rather than add Test package as a dep to my package. Thanks again.

1 Like
24

Thank you @davidanthoff and team for improving the development experience in Julia :juliaheartpulsing: Looking forward to trying it out. We have very large test sets that will definitely benefit from this new workflow.

2 Likes
25

Note that you only add it to the extras, so it won’t be loaded when a user uses your package.

2 Likes
26

How do you set the number of threads that the @testitem uses?

I have some multi-threaded tests that require > 1 thread. The extension by default uses 4 threads (via julia.NumThreads in the settings.json file, but this doesn’t propagate to the @testitem

27
2 Likes
28

Once @lmiq pointed out that I should look at the test server output it became clear why things weren’t working.

There were various problems with the test code I was writing that caused a load error in the test server process, which caused the test server to crash.

There wasn’t any immediate feedback from VSCode that anything was wrong. The error messages only appeared in the output of the test server process, which I didn’t see because I didn’t know this output existed.

What might cause such errors? These are some I encountered:

  • A missing package dependency inside the @testitem macro.
  • Change a test and immediately run it, not realizing you’ve introduced a syntax error.

There are many, many code errors that might cause a load error and a test server crash.

Because all you see in VSCode is “The Julia Test Server crashed”, maybe after a long wait, this makes it appear that the test framework is not working when the problem is actually in your code.

If you don’t know to look at the test server output (most users won’t) then you will assume there is a bug in the test framework. This will be the source of much confusion and unnecessary and incorrect bug reports.

Ideally the cause of the test server crash would be in the string displayed in VSCode at the test site. If that is not possible or too cumbersome maybe it would be helpful to add a note to the documentation, in bold text, telling people to look at the test server output before filing a bug report.

2 Likes
29

Effectively, as for now the use of the interface relies quite a bit in looking at that output. And it was added in the debugging process of the first issue.

For now I think that properly documenting that is fine (I was not clearly aware of those output tabs either, but at the end the errors must appear somewhere, and I am not a huge fan of them popping up in front of my code).

On the other side, there may be better alternatives, people used to these testing frameworks in other languages may have good suggestions. Worth opening issues in the repo to keep the suggestions alive.

30

@davidanthoff I’ve had a chance to use this feature more. It is terrific. It is now so much easier and faster to develop a test framework while writing code.

Thank you for this feature, and all the work you do to make VSCode better.

3 Likes
31

@davidanthoff I have just tried this feature and is working nicely, thank you for your work!

I have added some @testitem into my existing runtests.jl where I had test groups with SafeTestsets and both seem very similar, however they work mutually exclusively.
When running normal test procedure from Pkg, names from @testitem are not displayed and all tests are clumped into one category. The same is happening in Test Output in VSCode, where each @testitem is a separate entity, but no name is displayed (plus there is the issue with WARNING: replacing module Testmodule. but I’ve seen it is already reported on github).
The names are only displayed in the test pane, but no additional information like time.

Am I doing something wrong or is this not implemented yet?

32

Ah, I didn’t know about SafeTestsets, interesting! It is a little similar in that @testitem also runs each @testitem in its own module. The detection story of course is still entirely different.

Generally, the various output things still need to be improved, I think at the moment I was really more focused on getting everything to run :wink:

3 Likes
33

Overall this new feature is great. But I have noticed that the test framework occasionally gets out of sync with the most up to date information in the code files.

I’ll make a change to fix a bug, run the tests, they fail with an error message that makes it clear that out of date code is being run. If the tests are immediately run again, with no changes to the source, they pass.

Sorry don’t have a MWE, it’s intermittent and I don’t have insight into what might make it happen.

Julia Version 1.8.0
Commit 5544a0fab76 (2022-08-17 13:38 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 128 × AMD EPYC 7702P 64-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, znver2)
  Threads: 128 on 128 virtual cores
Environment:
  JULIA_NUM_THREADS = 64
  JULIA_EDITOR = code
34

Are you making a change to the code inside the @testitem or in the package? The way code changes are detected for these two scenarios is distinct.

35

Hi David, any plans to support global variables or at least some standard place where users can store global settings? We have very long and extensive tests in some packages where we need to define some global variables related to visual tests, directories with data etc. If we could move these global settings to a place where VScode can read that would be amazing. We are hesitant to migrate because of this limitation.

36

A change to the code in the package.

37

So the best way to do that right now is to put that code into a separate file, say test_shared.jl, and then at the beginning of each @testitem just do a include("test_shared.jl").

Of course, that will not add any shared state between different @testitems, it will just allow you to reuse some code in multiple @testitems. I am thinking about something where actual state is shared also. In that case a user would have to guarantee that this state is never modified by any @testitem, otherwise things will go really haywire :slight_smile: But I haven’t really decided how to do that exactly, at the moment I’m working on a parallel test execution feature in the extension.

1 Like
38

Interesting… The logic in general is that before any @testitem execution I first try to call Revise.revise(). If that works, I run the @testitem. If not, I restart the test process… I think I’ll try to add some diagnostic output in a new version that will allow you to tell me whether this kind of restart is happening in your case or not. But that will take a new build…

39

I’m keeping a submodule in my package called “Testing”, containing functions and variables for testing. The I use this module inside the test items.

1 Like
40

That is an interesting idea. I wonder if the Julia extension could do something like that for us automatically with macros. For example, if users added a section

@testsetup begin
  # global variables
  a = 1

  # global functions
  f(x) = 2x
end

to their runtests.jl file, this would then become available in all @testitems with the module trick.

1 Like