Unit testing with generated code

General Usage
1

Problem Statement: I have a package where a number of functions are defined with multiple-dispatch signatures like f(x::A, y::B). Say that methods may be implemented for m types A and n types B and there are p functions, but not all methods will be defined. Currently two @tests are implemented for every supported combination. This leads to a support matrix where up to 2(m \times n \times p) tests should be defined. My goal is to implement something like a human-readable table where I can specify which methods are implemented and generate/run @tests for all of those.

Prior Attempts: My original solution was to manually write out all of the required tests, but this quickly became unwieldy. Eventually I consolidated things slightly using a for loop over types B. Recently I tried implementing the support table as a vector of named tuples and defined functions that implement the required tests for each named tuple, but it seems like trying to define @tests inside of a function, separate from the @testset is problematic. At a high level, it looked roughly of the form below:

function generatedtests(a,b)
    # ...
    @test ...
    @test ...
end

function test_row(nt)
    @testset “$typeof(nt.a)” begin
        foreach(generatedtest, ...)
    end
end
2

I accidentally hit the Post button early…

Basically I have a couple of questions:

  • Is there a way to tweak this existing form to avoid the function scope issue?
  • Do I need to just implement a macro instead?
  • Has anyone implemented something to solve this kind of problem before?
3

I’m doing something a little similar. If you just want to test every method available you could do something like:

for a in aTypes, b in bTypes
    hasmethod(fn, Tuple{a, b}) && @test fn(...)
end

But if you want to test that a function has methods for a particular set of type combinations I think you’d have to have a matrix or something. But then a BitMatrix could work and you could have:

typematrix = falses(m, n)
# initialise with values...

for i in 1:m, j in 1:n
    typematrix[i, j] && @test fn(aTypes[i](), bTypes[j]())
end
2 Likes
4

Yeah, one of the factors driving me toward this kind of solution is that the full matrix of combinations won’t be implemented (intentionally). Essentially, type A specifies a data structure being acted on and type B represents an algorithmic choice with settings, but not all B’s are applicable to all A’s so there’s a small number of expected/documented undefined methods.

Edit: and I’d rather not rely on hasmethod as a filter since an unexpectedly missing method just wouldn’t get tested.

1 Like
5

You’d probably also want something like:

!typematrix[i, j] && @test_throws MethodError fn(...)

to check that they are undefined.

Depending on how big n, m are, and how many are expected to be undefined, my tendency would be to define this in some external JSON/YAML/etc and load this at compile time?

6

Aside from the issues already canvased, you also mentioned concerns about having @test inside a function separate from the @testset macro.

I’m running tests that use a local HTTP testing server which has @test in the server code, running in a separate thread and these seem to work fine within the test sets I’m running.

That is, I’ve got:

function serve(...)
    ...
    @test request_is_valid
    ...
end

start_server() = HTTP.serve!(serve, ...)

in one file, then

start_server()
...
@testset fn_name begin
    response = make_request()
    @test response == expected
end
...
stop_server()

Both the @test in the server code and in the client code both register as part of the test set, as desired.

1 Like
7

On some level you’d have to specify which are or aren’t, whichever’s easier. You could write it in less text with a sparse or bit matrix, and iterate it to determine which type combinations to test for. Not great because this gets inconvenient really quickly for combinations of 3+ types, but I can’t really think of anything better.

8

Weird, it seemed to me like something about the separation of @testset from @tests defined in child functions was leading to them not registering or at least recording the results. Maybe I just need to do some debugging then. I’m looking through the source for @testset and @test macros, but I’m not an expert in Julia macros so it’s challenging for me to reason about them very precisely.

9

I’ll plan to post a link here to what I came up with when I get things working, just in case anyone else has a similar interest. Basically, I’ve got a table where the rows are keyed by A types, with boolean columns that indicate which functions can act on them and which B types are applicable. Then my test code would generate a @testset for each row and generates @tests as required for each column with a true entry and a @test_throws otherwise. There’s a little bit of setup overhead, but it allows for a nice human-readable table to drive test requirements.

1 Like