Why is `Test` so meta-programming heavy?

[Elrod]

@baggepinnen

this prints not only the values that were encountered and what was expected, similar to what a function could have done, but also the expression that produced the error. Only macros have access to the expression the user writes.

Here is a C++ testing library that doesn’t require macros.
Julia macros are far better than C++ macros, so we tend to try less hard to avoid them. But if we wanted to, we could perhaps draw inspiration from there.

They use overloading + operator precedence.
E.g.

struct Test end
const test = Test()

struct TestExpr # should we specialize?
     t::Any
end

Base.:(%)(::Test, x) = TestExpr(x)
function Base.:(==)(x::TestExpr, y)
    if x.t != y
        # we'd want to define a TestException
        error("$(x.t) != $y")
    end
end

I get

julia> a = 1; b = 2;

julia> test % a == b
ERROR: 1 != 2
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] ==(x::TestExpr, y::Int64)
   @ Main ./REPL[5]:4
 [3] top-level scope
   @ REPL[8]:1

Multiple dispatch (or in the case of C++, function overloading) can take you far – but as you note, it only takes us as far as the values.

Without macros, we do not have the expressions.

[nrontsis]

@ToucheSir

I feel that making an arbitrary convention of “all test functions should start with test_” is even more magical than the metaprogramming Test does. At least when I see the @ I know magic is happening. That and pytest also makes liberal use of @ decorators for parameterized tests, fixtures, etc. Those are basically equivalent to what Julia testing libraries do with metaprogramming but at runtime.

I agree that pytest’s @fixture and many other decorators are too complicated. So perhaps when the user is writing parametrised tests, using Julia’s meta-programming might be quite advantageous, as compared to pytest’s fixtures, but I believe that the use of metaprogramming would be best to be left at the discretion of the user.

However, I disagree in that I do find a rule like “unit test is every exported test_*() function defined in any test_*.jl file in the packages test folder” as very simple and powerful.

As others have mentioned, the issue with Test not being powerful enough to support features like partial runs is not because of metaprogramming

Agreed, but my original post was not about it being powerful - it was about the obscurity the metaprogramming introduces (sorry, I did get to not being powerful later on ). I mentioned the lack of debugging capabilities and difficult stack traces as my two main points - don’t you think that these relate to metaprogramming (in particular for @testset - not so much for @test which is much “lower level” and other people in this thread convinced me of its use)?

Perhaps I am too averse to metaprogramming? I try to avoid it, unless there is a particularly good reason to use, and even then I try to contain it, not put at the top-most of my whole code. This is what was suggested in this JuliaCon Keynote Talk, where Steven Johnson (an expert in code generation, I believe) spends almost twenty minutes talking about how most often using meta-programming is a mistake.

[stevengj]

@nrontsis

Steven Johnson (an expert in code generation, I believe) spends almost twenty minutes talking about how most often using meta-programming is a mistake.

That’s true about 90% of the time, but in the remaining 10% of cases it is extremely useful.

[johnmyleswhite]

@nrontsis

Perhaps I am too averse to metaprogramming? I try to avoid it, unless there is a particularly good reason to use, and even then I try to contain it, not put at the top-most of my whole code. This is what was suggested in this JuliaCon Keynote Talk, where Steven Johnson (an expert in code generation, I believe) spends almost twenty minutes talking about how most often using meta-programming is a mistake.

I think one challenge is that it’s not fully clear what you mean by meta-programming since the Python libraries you used as a contrast with Julia are built on top of techniques (e.g. introspection) that I would call meta-programming: https://2019.pycon.de/program/pyconde-xtd7te-abridged-metaprogramming-classics-this-episode-pytest-oliver-bestwalter/

[ChrisRackauckas]

I think there’s two cases to use metaprogramming:

1. When you need to write something that acts on the code naming itself, i.e. you need to know what the names of the variables the user uses in order to give better printing/feedback. Ex: @variable x.
2. Syntactic sugar for things the user knows how to write but doesn’t want to. Example: @.. or @views.

Many times you have both, for example @named x = ODESystem(sys) in ModelingToolkit performs both functions, or @model in Turing.jl names the internal variables using the DSL of Turing. Anything more than that though, I would consider using a function.

Test uses it for function (1).

[mrufsvold]

@ChrisRackauckas

Syntactic sugar for things the user knows how to write but doesn’t want to.

Your faith in my the user’s ability to correctly write out the result of syntactic sugar macros is probably misplaced

[mbauman]

@nrontsis

Perhaps I am too averse to metaprogramming? I try to avoid it, unless there is a particularly good reason to use, and even then I try to contain it, not put at the top-most of my whole code.

There’s a significant difference between using already-defined macros that Julia provides (like @testset, @test, and also @simd, @view, etc) and the trap of defining your own macros or using eval/@eval.

You definitely should be averse to the latter; that’s where the dragons lie. But using macros in the stdlibs (except @eval) shouldn’t be so scary.

[bertschi]

Don’t see much difference between macros in base, libraries or user code, i.e., in all cases they should make sense and be there for a reason (in the end, one of the nice things about Julia is that user-defined code is just like system code, e.g., allowing for efficient and composable custom data structures),
@testset constructs a protected scope around the tests executed within. Such scopes have traditionally been handled via macros, e.g., in Common Lisp. Yet, modern language often support another syntax in the form of extensible resource managers, e.g., the with statement in Python. Guess that the do notation could have been used in Julia instead of a macro:

testset("name") do
    @test 1 == 2
end

Metaprogramming has become rather wide-spread, e.g., in Python or Ruby. Yet, there its mostly done at runtime via hooking into the meta-object protocol instead of syntactic transformations, i.e., macros.

[nrontsis]

You are right - I don’t have a clear understanding on what constitutes meta-programming, and this was also reflected in the original post.

In that sense I kinda got the answer to my question.

Still, I am actually surprised that people have not commented at all in the lack of debugging (mentioned in my original post as one of the two motivating difficulties) while in @testsets and whether this is due to meta-programming (as per their definition).

[davidanthoff]

@nrontsis

lack of debugging

I might be wrong, but I think that would not be easy to pull off in the existing base test framework. I think it would probably be much easier to add debugging support to the test item framework, in fact pretty much all the pieces need to pull that off inside VS Code exist already. I “just” need to hook it all up No promise on when that is going to happen, it is on my roadmap, but it also is probably a pretty significant lift, so it might be a while. But at least the design should really lend itself very well to implementing this.