Why is `Test` so meta-programming heavy?

nrontsis
Tue, 14 Feb 2023 11:00:03 +0000
Internals & Design

I am quite puzzled by the design choices of Julia’s Test - so I thought I should write my current understanding so that people can tell me what they think, or if I am misunderstanding something.

As I understand it, Julia’s Test relies heavily on meta-programming, due to the @testset etc macros. This introduces the following difficulties:

It’s difficult (/not even supported?) to open a debugger inside @testset (but thankfully Infiltrator.jl works)

I have occasionally witnessed stacktraces being very uniformative when an error appears inside nested @testsets. To be fair I only experienced this in errors of @generated functions, but I think that this illustrates the issues that the “meta-programming complexity” that @testset can introduce.

What is the rationale of having a testing framework that relies so much on meta-programming, like Test does?

In my mind there could be much simpler alternatives like e.g. following something similar to pytest (but without its complicated fixtures) so that unit tests are simply all the functions named test_* in the test folder of a package, organised in groups by their path.

For example:

test/test_abs:

function test_abs_on_int():
      @assert abs(-1) == 1

function test_abs_on_complex():
      @assert abs(-im) = 1.0

test/test_sum:

...

Common code across tests could then be shared by non test_* functions or Modules.

30 posts - 18 participants