Prerelease of new testing framework and test run UI in VS Code

That works, but is really not ideal because now you keep that state around in a non-test situation as well, right?

Yes, I think something like that makes sense. We’ll just have to think hard about a) how is given @testsetup linked to a given @testitem? Maybe a @testsetup should have a name, and then it needs to explicitly be specified in a @testitem? Or maybe @testsetup can have a flag a la default that would then load it into all @testitems. Or something like that :slight_smile: The second b) question is whether introducing this global shared stated will cause real problems in terms of @testitem runs no longer being really truly independent…

1 Like

For the first, maybe something along the lines of

@testsetup "setup1" begin
# setup code
end

@testitem "item1" begin
@setup "setup1"
# test code
end

For the second part, maybe you can do more than just importing, maybe you can inline the code, that way every @testitem is its separate environment.

If the user do want a module then it could be something along the lines of

@testsetup "setup1" begin
# setup code
end

@testitem "item1" begin
@setup "setup1" module=true # inlining the code wraps it in a module and does `using`
# test code
end

Also, setup code can be run along the test as its individual test.

The module has functions and constants, thus no big deal. What’s not ideal is the dependency management of that module. I can’t have test only dependencies there.

edit: Now I remember doing something like modifying my test functions to receive as parameters the functions of the dependencies, such that the module is not anymore dependent on those. For a small test set with those properties, that’s ok, for larger sets it becomes cumbersome, for sure.

Thanks for the work, this is really nice. It is a shame that the old syntax cannot be conserved and overloaded, but I understand why you did that this way.

What about a piece of template in PkgTemplate.jl that will produce directly the infrastructure needed in the package ? This might help people install this correctly on new packages on the fly (with the packages to add to the test environement and not the package environement, etc, already pre-done for you).

That would be great! In particular, just adding the TestItemRunner to Project.toml and the correct default runtests.jl would probably go a long way. I have too much on my plate right now to tackle this, but if someone else wants to take the initiative it would be great.

1 Like