Unit testing for macros?

General Usage
1

Suppose I write some complicated macro:

"Return exact same expression."
macro flawed_identity(ex)
    @assert ex.head != :(=) "oops we don't support head of :="
    ex
end

and I write some tests:

@testset "identity" begin
    ex = :(2+1)
    new = @flawed_identity 2 + 1
    @test eval(new) == ex
    @flawed_identity ex2 = 2 + 1
    @test eval(new) == ex
end

When I run the tests, I get an AssertionError, and I don’t get a Test Summary. How should I write Unit Tests for macros such that all tests run even if an error pops up?

2

Here’s how JuMP does it:
https://github.com/jump-dev/JuMP.jl/blob/7aeec3fb2dc40c365b3f9c94d559fda5743d111f/test/utilities.jl#L41-L54

julia> using Test

julia> "Return exact same expression."
       macro flawed_identity(ex)
           @assert ex.head != :(=) "oops we don't support head of :="
           ex
       end
@flawed_identity

julia> macro test_macro_throws(err_type, ex)
           return quote
               @test_throws(
                   $(esc(err_type)),
                   try
                       @eval $(esc(ex))
                   catch err
                       @show err
                       throw(err.error)
                   end
               )
           end
       end
@test_macro_throws (macro with 2 methods)

julia> @testset "identity" begin
           new = @flawed_identity 2 + 1
           @test new == 3
           @test_macro_throws AssertionError @flawed_identity ex2 = 2 + 1
       end
err = LoadError("REPL[84]", 4, AssertionError("oops we don't support head of :="))
Test Summary: | Pass  Total
identity      |    2      2
Test.DefaultTestSet("identity", Any[], 2, false, false)

there might be a better way. We haven’t tried anything else recently.

1 Like
3

Great, thank you! I wanted something slightly different, but your answer got me there:

julia> using Test
julia> macro safetest(ex)
           return quote
               @test try
                   @eval $(esc(ex))
               catch err
                   throw(err)
              end
           end
       end
@safetest (macro with 1 method)

julia> macro flawed_identity(ex)
           @assert ex.head != :(=) "oops we don't support head of :="
           ex
       end
@flawed_identity (macro with 1 method)

julia> @testset "identity" begin
                  new = @flawed_identity 2 + 1
                  @test new == 3
                  @safetest begin
                     @flawed_identity ex2 = 2 + 1
                     ex2 == 3
                  end
              end
identity: Error During Test at REPL[33]:3
  Test threw exception
  Expression: try
    #= REPL[33]:4 =# @eval $(Expr(:escape, quote
    #= REPL[35]:5 =# @flawed_identity ex2 = 2 + 1
    ex2 == 3
end))
catch err
    throw(err)
end
  LoadError: AssertionError: oops we don't support head of :=
  Stacktrace:
   [1] macro expansion
     @ REPL[33]:3 [inlined]
   [2] macro expansion
     @ REPL[35]:4 [inlined]
   [3] macro expansion
     @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [4] top-level scope
     @ REPL[35]:2
  in expression starting at REPL[35]:5
  
  caused by: LoadError: AssertionError: oops we don't support head of :=
  Stacktrace:
   [1] var"@flawed_identity"(__source__::LineNumberNode, __module__::Module, ex::Any)
     @ Main ./REPL[34]:2
   [2] eval(m::Module, e::Any)
     @ Core ./boot.jl:360
   [3] macro expansion
     @ REPL[33]:3 [inlined]
   [4] macro expansion
     @ REPL[35]:4 [inlined]
   [5] macro expansion
     @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [6] top-level scope
     @ REPL[35]:2
  in expression starting at REPL[35]:5
Test Summary: | Pass  Error  Total
identity      |    1      1      2
ERROR: Some tests did not pass: 1 passed, 0 failed, 1 errored, 0 broken.
1 Like
4

Bit late to the party, but it can sometimes also be very useful to move as much of the logic as possible into a function. The function can be tested quite easily. See also InteractiveUtils.gen_call_with_extracted_types_and_kwargs and it’s usage in, for example, Revise.jl

2 Likes