# Why is \`Test\` so meta-programming heavy?

**URL:** <https://discourse.julialang.org/t/why-is-test-so-meta-programming-heavy/94619>\
**Category:** Internals & Design\
**Tags:** testing, design\
**Created:** [February 14, 2023, 11:00am UTC](https://discourse.julialang.org/t/why-is-test-so-meta-programming-heavy/94619 "2023-02-14T11:00:03Z")\
**Posts on this page:** 10\
**Page:** 2

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [February 15, 2023, 6:57pm UTC](https://discourse.julialang.org/t/why-is-test-so-meta-programming-heavy/94619/23 "2023-02-15T18:57:55Z")

</div>

> [@baggepinnen](#):
>
> this prints not only the values that were encountered and what was expected, similar to what a function could have done, but also _the expression_ that produced the error. Only macros have access to the expression the user writes.

[Here](https://github.com/boost-ext/ut) is a C++ testing library that doesn’t require macros.  
Julia macros are far better than C++ macros, so we tend to try less hard to avoid them. But if we wanted to, we could perhaps draw inspiration from there.

They use overloading + operator precedence.  
E.g.

```julia
struct Test end
const test = Test()

struct TestExpr # should we specialize?
     t::Any
end

Base.:(%)(::Test, x) = TestExpr(x)
function Base.:(==)(x::TestExpr, y)
    if x.t != y
        # we'd want to define a TestException
        error("$(x.t) != $y")
    end
end

```

I get

```julia
julia> a = 1; b = 2;

julia> test % a == b
ERROR: 1 != 2
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] ==(x::TestExpr, y::Int64)
   @ Main ./REPL[5]:4
 [3] top-level scope
   @ REPL[8]:1

```

Multiple dispatch (or in the case of C++, function overloading) can take you far – but as you note, it only takes us as far as the values.

Without macros, we do not have the expressions.

---

<div class="post-metadata">

**Author:** ![nrontsis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nrontsis/32/38535_2.png) [@nrontsis](https://discourse.julialang.org/u/nrontsis)\
**Post date:** [February 16, 2023, 12:27am UTC](https://discourse.julialang.org/t/why-is-test-so-meta-programming-heavy/94619/24 "2023-02-16T00:27:00Z")

</div>

> [@ToucheSir](#):
>
> I feel that making an arbitrary convention of “all test functions should start with test\_” is even more magical than the metaprogramming Test does. At least when I see the `@` I know magic is happening. That and pytest also makes liberal use of `@` decorators for parameterized tests, fixtures, etc. Those are basically equivalent to what Julia testing libraries do with metaprogramming but at runtime.

I agree that `pytest`’s `@fixture` and many other decorators are too complicated. So perhaps when the user is writing parametrised tests, using Julia’s meta-programming might be quite advantageous, as compared to `pytest`’s fixtures, but I believe that the use of metaprogramming would be best to be left at the discretion of the user.

However, I disagree in that I do find a rule like “unit test is every exported `test_*()` function defined in any `test_*.jl` file in the packages test folder” as very simple and powerful.

> As others have mentioned, the issue with Test not being powerful enough to support features like partial runs is not because of metaprogramming

Agreed, but my original post was not about it being powerful - it was about the obscurity the metaprogramming introduces (sorry, I did get to not being powerful later on 😆). I mentioned the lack of debugging capabilities and difficult stack traces as my two main points - don’t you think that these relate to metaprogramming (in particular for `@testset` - not so much for `@test` which is much “lower level” and other people in this thread convinced me of its use)?

Perhaps I am too averse to metaprogramming? I try to avoid it, unless there is a particularly good reason to use, and even then I try to contain it, not put at the top-most of my whole code. This is what was suggested in [this JuliaCon Keynote Talk](https://www.youtube.com/watch?v=mSgXWpvQEHE), where Steven Johnson (an expert in code generation, I believe) spends almost twenty minutes talking about how most often using meta-programming is a mistake.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [February 16, 2023, 12:36am UTC](https://discourse.julialang.org/t/why-is-test-so-meta-programming-heavy/94619/25 "2023-02-16T00:36:38Z")

</div>

> [@nrontsis](#):
>
> Steven Johnson (an expert in code generation, I believe) spends almost twenty minutes talking about how most often using meta-programming is a mistake.

That’s true about 90% of the time, but in the remaining 10% of cases it is extremely useful. 😉

---

<div class="post-metadata">

**Author:** ![johnmyleswhite](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnmyleswhite/32/31_2.png) [@johnmyleswhite](https://discourse.julialang.org/u/johnmyleswhite)\
**Post date:** [February 16, 2023, 1:13am UTC](https://discourse.julialang.org/t/why-is-test-so-meta-programming-heavy/94619/26 "2023-02-16T01:13:09Z")

</div>

> [@nrontsis](#):
>
> Perhaps I am too averse to metaprogramming? I try to avoid it, unless there is a particularly good reason to use, and even then I try to contain it, not put at the top-most of my whole code. This is what was suggested in [this JuliaCon Keynote Talk](https://www.youtube.com/watch?v=mSgXWpvQEHE), where Steven Johnson (an expert in code generation, I believe) spends almost twenty minutes talking about how most often using meta-programming is a mistake.

I think one challenge is that it’s not fully clear what you mean by meta-programming since the Python libraries you used as a contrast with Julia are built on top of techniques (e.g. introspection) that I would call meta-programming: [https://2019.pycon.de/program/pyconde-xtd7te-abridged-metaprogramming-classics-this-episode-pytest-oliver-bestwalter/](https://2019.pycon.de/program/pyconde-xtd7te-abridged-metaprogramming-classics-this-episode-pytest-oliver-bestwalter/)

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [February 16, 2023, 2:14am UTC](https://discourse.julialang.org/t/why-is-test-so-meta-programming-heavy/94619/27 "2023-02-16T02:14:47Z")

</div>

I think there’s two cases to use metaprogramming:

1. When you need to write something that acts on the code naming itself, i.e. you need to know what the names of the variables the user uses in order to give better printing/feedback. Ex: `@variable x`.
2. Syntactic sugar for things the user knows how to write but doesn’t want to. Example: `@..` or `@views`.

Many times you have both, for example `@named x = ODESystem(sys)` in ModelingToolkit performs both functions, or `@model` in Turing.jl names the internal variables using the DSL of Turing. Anything more than that though, I would consider using a function.

Test uses it for function (1).

---

<div class="post-metadata">

**Author:** ![mrufsvold](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mrufsvold/32/31600_2.png) [@mrufsvold](https://discourse.julialang.org/u/mrufsvold)\
**Post date:** [February 16, 2023, 6:23am UTC](https://discourse.julialang.org/t/why-is-test-so-meta-programming-heavy/94619/28 "2023-02-16T06:23:29Z")

</div>

> [@ChrisRackauckas](#):
>
> Syntactic sugar for things the user knows how to write but doesn’t want to.

Your faith in ~~my~~ the user’s ability to correctly write out the result of syntactic sugar macros is probably misplaced 😉

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [February 16, 2023, 5:32pm UTC](https://discourse.julialang.org/t/why-is-test-so-meta-programming-heavy/94619/29 "2023-02-16T17:32:09Z")

</div>

> [@nrontsis](#):
>
> Perhaps I am too averse to metaprogramming? I try to avoid it, unless there is a particularly good reason to use, and even then I try to contain it, not put at the top-most of my whole code.

There’s a significant difference between _using_ already-defined macros that Julia provides (like `@testset`, `@test`, and also `@simd`, `@view`, etc) and the trap of _defining_ your own macros or using `eval`/`@eval`.

You definitely should be averse to the latter; that’s where the dragons lie. But using macros in the stdlibs (except `@eval`) shouldn’t be so scary.

---

<div class="post-metadata">

**Author:** ![bertschi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bertschi/32/33462_2.png) [@bertschi](https://discourse.julialang.org/u/bertschi)\
**Post date:** [February 16, 2023, 7:49pm UTC](https://discourse.julialang.org/t/why-is-test-so-meta-programming-heavy/94619/30 "2023-02-16T19:49:23Z")

</div>

Don’t see much difference between macros in base, libraries or user code, i.e., in all cases they should make sense and be there for a reason (in the end, one of the nice things about Julia is that user-defined code is just like system code, e.g., allowing for efficient and composable custom data structures),  
`@testset` constructs a protected scope around the tests executed within. Such scopes have traditionally been handled via macros, e.g., in Common Lisp. Yet, modern language often support another syntax in the form of extensible resource managers, e.g., the `with` statement in Python. Guess that the `do` notation could have been used in Julia instead of a macro:

```julia
testset("name") do
    @test 1 == 2
end

```

Metaprogramming has become rather wide-spread, e.g., in Python or Ruby. Yet, there its mostly done at runtime via hooking into the meta-object protocol instead of syntactic transformations, i.e., macros.

---

<div class="post-metadata">

**Author:** ![nrontsis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nrontsis/32/38535_2.png) [@nrontsis](https://discourse.julialang.org/u/nrontsis)\
**Post date:** [February 20, 2023, 5:51pm UTC](https://discourse.julialang.org/t/why-is-test-so-meta-programming-heavy/94619/31 "2023-02-20T17:51:04Z")

</div>

You are right - I don’t have a clear understanding on what constitutes meta-programming, and this was also reflected in the original post.

In that sense I kinda got the answer to my question.

Still, I am actually surprised that people have not commented at all in the lack of debugging (mentioned in my original post as one of the two motivating difficulties) while in `@testset`s and whether this is due to meta-programming (as per their definition).

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [February 20, 2023, 8:14pm UTC](https://discourse.julialang.org/t/why-is-test-so-meta-programming-heavy/94619/32 "2023-02-20T20:14:33Z")

</div>

> [@nrontsis](#):
>
> lack of debugging

I might be wrong, but I think that would not be easy to pull off in the existing base test framework. I think it would probably be much easier to add debugging support to the [test item framework](https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355), in fact pretty much all the pieces need to pull that off inside VS Code exist already. I “just” need to hook it all up 🙂 No promise on when that is going to happen, it is on my roadmap, but it also is probably a pretty significant lift, so it might be a while. But at least the design should really lend itself very well to implementing this.

[Previous page](https://discourse.julialang.org/t/why-is-test-so-meta-programming-heavy/94619.md?page=1)