Prerelease of new testing framework and test run UI in VS Code

[davidanthoff]

@lmiq

I’m keeping a submodule in my package called “Testing”, containing functions and variables for testing. The I use this module inside the test items.

That works, but is really not ideal because now you keep that state around in a non-test situation as well, right?

@juliohm

I wonder if the Julia extension could do something like that for us automatically with macros.

Yes, I think something like that makes sense. We’ll just have to think hard about a) how is given @testsetup linked to a given @testitem? Maybe a @testsetup should have a name, and then it needs to explicitly be specified in a @testitem? Or maybe @testsetup can have a flag a la default that would then load it into all @testitems. Or something like that The second b) question is whether introducing this global shared stated will cause real problems in terms of @testitem runs no longer being really truly independent…

[suavesito]

For the first, maybe something along the lines of

@testsetup "setup1" begin
# setup code
end

@testitem "item1" begin
@setup "setup1"
# test code
end

For the second part, maybe you can do more than just importing, maybe you can inline the code, that way every @testitem is its separate environment.

If the user do want a module then it could be something along the lines of

@testsetup "setup1" begin
# setup code
end

@testitem "item1" begin
@setup "setup1" module=true # inlining the code wraps it in a module and does `using`
# test code
end

Also, setup code can be run along the test as its individual test.

[lmiq]

@davidanthoff

That works, but is really not ideal because now you keep that state around in a non-test situation as well, right?

The module has functions and constants, thus no big deal. What’s not ideal is the dependency management of that module. I can’t have test only dependencies there.

edit: Now I remember doing something like modifying my test functions to receive as parameters the functions of the dependencies, such that the module is not anymore dependent on those. For a small test set with those properties, that’s ok, for larger sets it becomes cumbersome, for sure.

[lrnv]

Thanks for the work, this is really nice. It is a shame that the old syntax cannot be conserved and overloaded, but I understand why you did that this way.

What about a piece of template in PkgTemplate.jl that will produce directly the infrastructure needed in the package ? This might help people install this correctly on new packages on the fly (with the packages to add to the test environement and not the package environement, etc, already pre-done for you).

[davidanthoff]

@lrnv

What about a piece of template in PkgTemplate.jl that will produce directly the infrastructure needed in the package ?

That would be great! In particular, just adding the TestItemRunner to Project.toml and the correct default runtests.jl would probably go a long way. I have too much on my plate right now to tackle this, but if someone else wants to take the initiative it would be great.

[davidanthoff]

I just released a new build to the insider channel of the VS Code extension. I added a pretty complete writeup of what was added to my initial post at the top of this thread, see here.

[Daniel_Gonzalez]

Thank you David, this extension is great! I have one question: if I’m developing a package and a test item fails at the package precompilation stage, is there currently any way to see the “inner” stack trace (the one describing the actual reason for the precompilation failure) in the test output instead of (or in addition to) the “outer” stack trace (the one that just tells you that precompilation failed)?

[davidanthoff]

@Daniel_Gonzalez

I have one question: if I’m developing a package and a test item fails at the package precompilation stage, is there currently any way to see the “inner” stack trace (the one describing the actual reason for the precompilation failure) in the test output instead of (or in addition to) the “outer” stack trace (the one that just tells you that precompilation failed)?

Ah, excellent point! I’ll try to fix soon, agreed that it would make much more sense to show the inner error in that case.

[davidanthoff]

Actually, turns out that I’m not sure we can fix this… The error we are interested in is happening in a different Julia instance that is spun up just for the precompile, and apparently the error information is not transferred back at all to the calling process, so as far as I understand it right now, we just don’t have access to it

[cjdoris]

As an alternative to the @testsetup suggestion for common test code, how about we make test items nested, and each test item runs all the code from parent test items too.

For example

@testitem "outer" begin
    setup()
    @testitem "inner1" ...
    @testitem "inner2" ...
end

has two “leaf” test items, which each run setup() first.

And if you didn’t want to syntactically nest then there could be a parent argument so you could instead do

@testitem "inner1" parent="outer" ...
@testitem "inner2" parent="outer" ...

which is approximately the same as the @testsetup idea.

Nesting would also be nice just from a GUI point of view, to organise large test suites a bit.

[juliohm]

@davidanthoff any chance to include the “global setup” macro in this release already? It is the only missing feature that inhibits us from migrating to the extension in big test suites like the one in Meshes.jl: Meshes.jl/runtests.jl at master · JuliaGeometry/Meshes.jl · GitHub

[davidanthoff]

I have two design issues where I would welcome feedback, ideally on the linked GitHub issues.

1. Some thoughts on the code reuse question, CC @juliohm
2. Some thoughts on hierarchies, here

[brianguenter]

I finally figured out why it appeared that the test framework was running out of date code. For reasons that are still unclear sometimes VSCode takes much longer than it should, minutes instead of a fraction of a second, to save small julia files. I was changing code in the test cases and then running them immediately but VSCode hadn’t saved them yet so the old versions were run.

I believe this is related to file formatting. I have VSCode set to autosave on focus change and to format on save.

Ordinarily formatting and saving the files takes a fraction of a second. But, the longer my REPL session is active the longer formatting and saving takes, until eventually it takes minutes to format and save a 200 line file. Then I have to restart VSCode to get acceptable response.

I posted a question about this here but did not get a resolution.

@davidanthoff do you have any ideas about how I can fix the slow saving problem?

Version and OS info

julia> versioninfo()
Julia Version 1.8.5
Commit 17cfb8e65e (2023-01-08 06:45 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 32 × AMD Ryzen 9 7950X 16-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, znver3)
  Threads: 32 on 32 virtual cores
Environment:
  JULIA_EDITOR = code.cmd

VSCode settings

{
    "terminal.integrated.commandsToSkipShell": [
        "language-julia.interrupt"
    ],

    "editor.fontFamily": "JuliaMono,Consolas,Courier New ,monospace",
    "editor.fontLigatures": "'zero', 'ss02', 'ss04', 'ss05', 'ss08'",
    "julia.symbolCacheDownload": true,
    "julia.enableTelemetry": true,
    "files.autoSave": "onFocusChange",
    "workbench.editor.enablePreview": false,
    "workbench.editor.revealIfOpen": true,
    "git.confirmSync": false,
    "editor.accessibilitySupport": "off",
    "terminal.integrated.tabs.enabled": false,
    "editor.minimap.enabled": false,
    "git.enableSmartCommit": true,
    "editor.wordWrap": "on",
    "editor.defaultFormatter": "julialang.language-julia",
    "editor.formatOnSave": true,
    "julia.NumThreads": "auto",
    "julia.useCustomSysimage": true
}

[davidanthoff]

Hm, that is very strange. I don’t think it can be related to the length of the REPL session, code formatting should be entirely independent of that…