# Unit testing with generated code

**URL:** <https://discourse.julialang.org/t/unit-testing-with-generated-code/110614>\
**Category:** General Usage\
**Tags:** testjl-unittesting\
**Created:** [February 22, 2024, 11:52pm UTC](https://discourse.julialang.org/t/unit-testing-with-generated-code/110614 "2024-02-22T23:52:00Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![mike.ingold](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mike.ingold/32/203749_2.png) [@mike.ingold](https://discourse.julialang.org/u/mike.ingold)\
**Post date:** [February 22, 2024, 11:52pm UTC](https://discourse.julialang.org/t/unit-testing-with-generated-code/110614/1 "2024-02-22T23:52:01Z")

</div>

**Problem Statement:** I have a package where a number of functions are defined with multiple-dispatch signatures like `f(x::A, y::B)`. Say that methods may be implemented for m types `A` and n types `B` and there are p functions, but not all methods will be defined. Currently two `@test`s are implemented for every supported combination. This leads to a support matrix where up to 2(m \times n \times p) tests should be defined. My goal is to implement something like a human-readable table where I can specify which methods are implemented and generate/run `@test`s for all of those.

**Prior Attempts:** My original solution was to manually write out all of the required tests, but this quickly became unwieldy. Eventually I consolidated things slightly using a `for` loop over types `B`. Recently I tried implementing the support table as a vector of named tuples and defined functions that implement the required tests for each named tuple, but it seems like trying to define `@test`s inside of a function, separate from the `@testset` is problematic. At a high level, it looked roughly of the form below:

```julia
function generatedtests(a,b)
    # ...
    @test ...
    @test ...
end

function test_row(nt)
    @testset “$typeof(nt.a)” begin
        foreach(generatedtest, ...)
    end
end

```

---

<div class="post-metadata">

**Author:** ![mike.ingold](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mike.ingold/32/203749_2.png) [@mike.ingold](https://discourse.julialang.org/u/mike.ingold)\
**Post date:** [February 22, 2024, 11:56pm UTC](https://discourse.julialang.org/t/unit-testing-with-generated-code/110614/2 "2024-02-22T23:56:43Z")

</div>

I accidentally hit the Post button early…

Basically I have a couple of questions:

- Is there a way to tweak this existing form to avoid the function scope issue?
- Do I need to just implement a macro instead?
- Has anyone implemented something to solve this kind of problem before?

---

<div class="post-metadata">

**Author:** ![screw\_dog](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/screw_dog/32/48119_2.png) [@screw\_dog](https://discourse.julialang.org/u/screw_dog)\
**Post date:** [February 23, 2024, 12:51am UTC](https://discourse.julialang.org/t/unit-testing-with-generated-code/110614/3 "2024-02-23T00:51:14Z")

</div>

I’m doing something a little similar. If you just want to test every method available you could do something like:

```julia
for a in aTypes, b in bTypes
    hasmethod(fn, Tuple{a, b}) && @test fn(...)
end

```

But if you want to test that a function has methods for a particular set of type combinations I think you’d have to have a matrix or something. But then a `BitMatrix` could work and you could have:

```julia
typematrix = falses(m, n)
# initialise with values...

for i in 1:m, j in 1:n
    typematrix[i, j] && @test fn(aTypes[i](), bTypes[j]())
end

```

---

<div class="post-metadata">

**Author:** ![mike.ingold](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mike.ingold/32/203749_2.png) [@mike.ingold](https://discourse.julialang.org/u/mike.ingold)\
**Post date:** [February 23, 2024, 1:22am UTC](https://discourse.julialang.org/t/unit-testing-with-generated-code/110614/4 "2024-02-23T01:22:28Z")

</div>

Yeah, one of the factors driving me toward this kind of solution is that the full matrix of combinations won’t be implemented (intentionally). Essentially, type A specifies a data structure being acted on and type B represents an algorithmic choice with settings, but not all B’s are applicable to all A’s so there’s a small number of expected/documented undefined methods.

Edit: and I’d rather not rely on `hasmethod` as a filter since an unexpectedly missing method just wouldn’t get tested.

---

<div class="post-metadata">

**Author:** ![screw\_dog](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/screw_dog/32/48119_2.png) [@screw\_dog](https://discourse.julialang.org/u/screw_dog)\
**Post date:** [February 23, 2024, 1:30am UTC](https://discourse.julialang.org/t/unit-testing-with-generated-code/110614/5 "2024-02-23T01:30:01Z")

</div>

You’d probably also want something like:

```julia
!typematrix[i, j] && @test_throws MethodError fn(...)

```

to check that they are undefined.

Depending on how big n, m are, and how many are expected to be undefined, my tendency would be to define this in some external JSON/YAML/etc and load this at compile time?

---

<div class="post-metadata">

**Author:** ![screw\_dog](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/screw_dog/32/48119_2.png) [@screw\_dog](https://discourse.julialang.org/u/screw_dog)\
**Post date:** [February 23, 2024, 3:43am UTC](https://discourse.julialang.org/t/unit-testing-with-generated-code/110614/6 "2024-02-23T03:43:34Z")

</div>

Aside from the issues already canvased, you also mentioned concerns about having `@test` inside a function separate from the `@testset` macro.

I’m running tests that use a local HTTP testing server which has `@test` in the server code, running in a separate thread and these seem to work fine within the test sets I’m running.

That is, I’ve got:

```julia
function serve(...)
    ...
    @test request_is_valid
    ...
end

start_server() = HTTP.serve!(serve, ...)

```

in one file, then

```julia
start_server()
...
@testset fn_name begin
    response = make_request()
    @test response == expected
end
...
stop_server()

```

Both the `@test` in the server code and in the client code both register as part of the test set, as desired.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [February 23, 2024, 5:25am UTC](https://discourse.julialang.org/t/unit-testing-with-generated-code/110614/7 "2024-02-23T05:25:40Z")

</div>

> [@mike.ingold](#):
>
> the full matrix of combinations won’t be implemented (intentionally).

On some level you’d have to specify which are or aren’t, whichever’s easier. You could write it in less text with a sparse or bit matrix, and iterate it to determine which type combinations to test for. Not great because this gets inconvenient really quickly for combinations of 3+ types, but I can’t really think of anything better.

---

<div class="post-metadata">

**Author:** ![mike.ingold](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mike.ingold/32/203749_2.png) [@mike.ingold](https://discourse.julialang.org/u/mike.ingold)\
**Post date:** [February 23, 2024, 12:50pm UTC](https://discourse.julialang.org/t/unit-testing-with-generated-code/110614/8 "2024-02-23T12:50:46Z")

</div>

> [@screw\_dog](#):
>
> Both the `@test` in the server code and in the client code both register as part of the test set, as desired.

Weird, it seemed to me like something about the separation of `@testset` from `@test`s defined in child functions was leading to them not registering or at least recording the results. Maybe I just need to do some debugging then. I’m looking through [the source for `@testset` and `@test` macros](https://github.com/JuliaLang/julia/blob/923fe2d74d469d73f0285a9cd41c88f90edd04d1/stdlib/Test/src/Test.jl#L1590), but I’m not an expert in Julia macros so it’s challenging for me to reason about them very precisely.

---

<div class="post-metadata">

**Author:** ![mike.ingold](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mike.ingold/32/203749_2.png) [@mike.ingold](https://discourse.julialang.org/u/mike.ingold)\
**Post date:** [February 23, 2024, 1:02pm UTC](https://discourse.julialang.org/t/unit-testing-with-generated-code/110614/9 "2024-02-23T13:02:04Z")

</div>

I’ll plan to post a link here to what I came up with when I get things working, just in case anyone else has a similar interest. Basically, I’ve got a table where the rows are keyed by `A` types, with boolean columns that indicate which functions can act on them and which `B` types are applicable. Then my test code would generate a `@testset` for each row and generates `@test`s as required for each column with a `true` entry and a `@test_throws` otherwise. There’s a little bit of setup overhead, but it allows for a nice human-readable table to drive test requirements.