Transition to TestItems from a semantic Test suite, but keeping the semantics

Tooling VS Code
1

Inspired by some discussions on Slack, I am considering, to test how I could improve my current workflow to @testitems, since that would give a very nice interaction in VS Code. However, I feel, that my current test setup is nearly orthogonal to @testitem and I can not find a good way to transition – though I by now have thought about it for quite a while.

For my setup, I would start some new tests in Manifolds.jl, but since the setup is a bit younger and hence a bit better structured in LieGroups.jl, let me used that as an example.
But both share the fact, that they are libraries, the first of manifolds, the second of LieGroups.

In that package, we have written a Test suite, that can test single functions on a certain Lie group, for example lie_bracket – one of the shorter tests:

and there is an overall function test_liegroup where you would provide such a group, and the functions one would like to test, cf. LieGroups.jl/test/groups/test_general_linear_group.jl at 15265899caa05d3948578bcf026f242f782b3d29 · JuliaManifolds/LieGroups.jl · GitHub, where one would before in properties specify functions to test and points and (Lie algebra) tangent vectors; expectations are meant for values to expect.

To my problem:

To me @testitem sounds like something that would be the same order or even smaller than a @testset? On the other hand it fully encapsulates, so using it within e.g. test_lie_bracket would not be possible.

On the other hand the tags would be great to use “from the very outside” to for example just doing a “short integration test” somewhere in other packages (further upstream).

So I can not see a way to have these single functionality-tests we currently have together with test items? At least to me they feel orthogonal – I can either do small encapsulated standalone tests with testitems or this test suite idea, but I can not see how to combine these.

Any ideas?

2

From how your tests are structured, you could probably make each test file a test item, e.g. one test item per group, without having to change much else. You could make it more granular, but there is probably very little benefit.

1 Like
3

Interesting, I will give that a try; The reason I did not think about that, was that most of the docs – as well as the name “item” indicated to me that it should be more fine granular.

4

Maybe there is a way to move most of these functions/testcode into a @testmodule, and then have @testitems that just call these functions but don’t contain much code, actually? The @testitems would be more a way to select which test code to run, but the code would mostly be in @testmodules.

1 Like
5

The LieGroupsTestSuite is already a module, so maybe that could be turned into a test module?

But sure, one side could be to have a testitem per test function like the one above and with that an easy way to choose, which tests to run.

Just that for this approach I am stuck on the following:
A testitem as its own environment and a very fixed setup. So I have no idea (or model in my head) yet how to use a single test item and being able to run that on different Lie groups?

Or to be more precise: To me a testitem reads like a monolithic set-in-stone thing that runs exactly one test with exactly one setup.

But my test_Lie_bracket for example should be able to run on arbitrary Lie groups, not only one.
That is where I am stuck with the orthogonal idea

  • test item runs exactly one test on one setup
  • my current functions are semantic checks of properties on a lot of setups.

I am not yet able to “mix” these two.

6

Maybe you could put the test_Lie_bracket function in a @testmodule? And then have different @testitems that exercise it with different Lie groups? Maybe one @testitem runs it with all groups, or maybe one @testitem for each group you might want to test? Not sure any of this makes sense :slight_smile:

7

I would prefer not to have a “run this function with all groups” – because that would probably mean that someone who introduces a new group to the library would have to adapt all such testitems.

And the idea of the current test suite (the current LieGroupsTestSuite module) is:
If you have defined a Lie group – either a new one in the package or one somewhere in the ecosystem – specify which functions you implemented and we check they work well.
(which is the opposite of: please add it everywhere yourself ;))

If I write one testitem for every group, I loose the idea of modularity, that I can use tags to choose which tests to run – which would be my second reason to switch; the first being coverage in VS code.

As I said, I am also totally clueless and lost on this, because it seems testitems provide tests in a way that is orthogonal to what I want to achieve (besides coverage in VS code).

8

Hm, I think really what you are looking for is a way to run a test item with a parameter value, right? If there was a way to run a test item with a custom parameter value, you could pass in the name of the group?

The VS Code API has something that might be a fit, not sure. I’ll think more about it, initially I thought we would use this for something entirely different, but maybe it could be used for this as well. But anything there will not happen anytime soon, still busy migrating the entire ecosystem to the new runner code…

9

Well not “a” parameter value, but more like a whole set around that.

The name of the group is not enough, I need a concrete instance of that type, e.g. SpecialOrthogonal(3) (since it exists in arbitrary dimensions) and 1-3 points and 1-3 Lie algebra vectors and maybe other things.
So before such a “test suite call” there is a bit of a setup as well.

And also all this here is not urgent, I already thought about it about a year ago or so, and did not find a good way to mach “my way of testing” with test items.
And maybe the test suite idea is too different from what testitems can offer,
then still a test item per group might be nice to get coverage in VS code at least.

The single Lie group test files are meant to be standalone anyways (i.e. can be run with include("test/groups/test_mygroup.jl")

10

Maybe it would be helpful to clarify why you want to switch to testitems.

For me personally, it is almost entirely about being able to run tests individually. For example, when I‘m refactoring the implementation of one unit, I want to quickly run the test for that and nothing else.

11

Sure, that is a good idea.

So “a test individually” is in my setting maybe a bit ambiguous: To run the test suite on a single Lie group, making sure the functions still “interact” correctly, sure I do that often.

Running a single of the test functions like test_lie_bracket for a very specific Lie groups and specific point – not so often. Maybe I would do that more, if I can locally test for code cov with that.
But I fear that would mean that the whole setup be run before, that is besides using LIeGroups at least

and having expectations, available like

which knows the result of the Lie bracket to expect.

In order for that to work I would probably turn all the small functions into test items(?) but I am not sure how to “switch” loading different setups then.
Maybe the setups is really my main problem. I have a whole bunch of small tests that I want to run with (currently in this example about 18) different test scenarios. And @testitems are fixed to one test-module, one setup= scenario,
so that keeps me from seeing how they would be useful for me on the fine granular level

If something like that would work, I could probably steer which tests to run with tags. That sounds nice.

Sure I could “wrap” the while test file linked about in a test item as discussed before. That brings me code cov – but nothing else of what I just said.

Well, maybe the scanarios then really do not fit. Thanks still for the inputs and clarifications.