# How to test a default \`error(...)\` raising method's message

**URL:** <https://discourse.julialang.org/t/how-to-test-a-default-error-raising-methods-message/19224>\
**Category:** New to Julia\
**Created:** [January 3, 2019, 6:34am UTC](https://discourse.julialang.org/t/how-to-test-a-default-error-raising-methods-message/19224 "2019-01-03T06:34:22Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![taqtiqa-mark](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/taqtiqa-mark/32/4383_2.png) [@taqtiqa-mark](https://discourse.julialang.org/u/taqtiqa-mark)\
**Post date:** [January 3, 2019, 6:34am UTC](https://discourse.julialang.org/t/how-to-test-a-default-error-raising-methods-message/19224/1 "2019-01-03T06:34:22Z")

</div>

One way to understand the usage of a Julia package is to read the test suite…

Looking at [StatsBase.coef(…)](https://github.com/JuliaStats/StatsBase.jl/blob/88b481809cf3a4b4e381be37f4372122c2d7c361/src/statmodels.jl#L10) I wondered how/when I should expect to see that error message.

The [test suite is silent](https://github.com/JuliaStats/StatsModels.jl/blob/e03d8668bd5a8e45676fb0bb9b3b7904878f8cd3/test/statsmodel.jl).

What is the correct way to test that error message is observed?

_Aside:_  
This arose in the course of a different issue, but I thought best to isolate this specific question: the correct way to test a message that is raised by falling through to a generic definition.

---

<div class="post-metadata">

**Author:** ![tkoolen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkoolen/32/1603_2.png) [@tkoolen](https://discourse.julialang.org/u/tkoolen)\
**Post date:** [January 3, 2019, 7:23am UTC](https://discourse.julialang.org/t/how-to-test-a-default-error-raising-methods-message/19224/2 "2019-01-03T07:23:25Z")

</div>

Something like

```julia
using Test
try
    error("bla")
catch e
    buf = IOBuffer()
    showerror(buf, e)
    message = String(take!(buf))
    @test message == "bla"
end

```

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [January 3, 2019, 8:09am UTC](https://discourse.julialang.org/t/how-to-test-a-default-error-raising-methods-message/19224/3 "2019-01-03T08:09:25Z")

</div>

I’d write something like

```julia
let err = nothing
    try
        error("bla")
    catch err
    end

    @test err isa Exception
    @test sprint(showerror, err) == "bla"
end

```

so that it fails when the code inside try block does not throw. (Also I think we can use `sprint` here)

---

<div class="post-metadata">

**Author:** ![fredrikekre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fredrikekre/32/1688_2.png) [@fredrikekre](https://discourse.julialang.org/u/fredrikekre)\
**Post date:** [January 4, 2019, 6:57am UTC](https://discourse.julialang.org/t/how-to-test-a-default-error-raising-methods-message/19224/5 "2019-01-04T06:57:49Z")

</div>

There is also `@test_throws`:

```julia
julia> using Test

julia> f() = error("bla");

julia> @test_throws ErrorException("bla") f()
Test Passed
      Thrown: ErrorException

```

---

<div class="post-metadata">

**Author:** ![taqtiqa-mark](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/taqtiqa-mark/32/4383_2.png) [@taqtiqa-mark](https://discourse.julialang.org/u/taqtiqa-mark)\
**Post date:** [January 5, 2019, 10:40pm UTC](https://discourse.julialang.org/t/how-to-test-a-default-error-raising-methods-message/19224/6 "2019-01-05T22:40:44Z")

</div>

> [@fredrikekre](#):
>
> julia\> f() = error(“bla”); julia\> @test\_throws ErrorException(“bla”) f()

Just a note on why this wasn’t selected as the solution… it doesn’t seem to be the way that will generally work.

Specifically, when looping over a list of functions (such as those at the github link above):

```julia
 opcoll = (:coef, :coefnames, :coeftable, :islinear, :nobs, :params, :weights)
        for op in opcoll
            let err = nothing
                try
                    @eval $op()
                catch err
                end
                @test_throws ErrorException("MethodError: no method matching $op()") @eval($op())
                # @test err isa Exception
                # @test err isa MethodError
                # @test occursin("MethodError: no method matching $op()", sprint(showerror, err)) == true
            end
        end
.... 
    Expected: ErrorException("MethodError: no method matching params()")
      Thrown: MethodError(Moments.params, (), 0x00000000000061d2)

```

---

<div class="post-metadata">

**Author:** ![fredrikekre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fredrikekre/32/1688_2.png) [@fredrikekre](https://discourse.julialang.org/u/fredrikekre)\
**Post date:** [January 5, 2019, 10:58pm UTC](https://discourse.julialang.org/t/how-to-test-a-default-error-raising-methods-message/19224/7 "2019-01-05T22:58:29Z")

</div>

Right, it is only useful if you can construct the exception easily, as with `ErrorException` in my example.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [January 5, 2019, 11:11pm UTC](https://discourse.julialang.org/t/how-to-test-a-default-error-raising-methods-message/19224/8 "2019-01-05T23:11:03Z")

</div>

It would be nice if we can write, e.g.,

```julia
err = @test_throws SomeExceptionType f()
@test occursin("some error message", sprint(showerror, err))

```

---

<div class="post-metadata">

**Author:** ![bennedich](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bennedich/32/4894_2.png) [@bennedich](https://discourse.julialang.org/u/bennedich)\
**Post date:** [January 5, 2019, 11:32pm UTC](https://discourse.julialang.org/t/how-to-test-a-default-error-raising-methods-message/19224/9 "2019-01-05T23:32:15Z")

</div>

There’s this:

```julia
opcoll = (:coef, :coefnames, :coeftable, :islinear, :nobs, :params, :weights)

world_age = ccall(:jl_get_tls_world_age, UInt, ())

for op in opcoll
    @test_throws MethodError((@eval $op), (), world_age) @eval $op()
end

```

---

<div class="post-metadata">

**Author:** ![taqtiqa-mark](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/taqtiqa-mark/32/4383_2.png) [@taqtiqa-mark](https://discourse.julialang.org/u/taqtiqa-mark)\
**Post date:** [January 6, 2019, 1:23am UTC](https://discourse.julialang.org/t/how-to-test-a-default-error-raising-methods-message/19224/10 "2019-01-06T01:23:01Z")

</div>

> [@bennedich](#):
>
> @test\_throws MethodError((@eval $op), (), world\_age) @eval $op()

I’ll have to dig into `world_age` to fully appreciate what deeper testing this allows… on face value, it appears that approach would not be testing the code satisfied some (human) visible requirement.

For example: In one variation of the approach I outlined I discovered a typo in the source code for one of the methods/functions. This was only exposed by having the ‘correct’ error message defined in the test case.

---

<div class="post-metadata">

**Author:** ![bennedich](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bennedich/32/4894_2.png) [@bennedich](https://discourse.julialang.org/u/bennedich)\
**Post date:** [January 6, 2019, 10:39pm UTC](https://discourse.julialang.org/t/how-to-test-a-default-error-raising-methods-message/19224/11 "2019-01-06T22:39:22Z")

</div>

Keep in mind that the message (“no method matching…”) is not part of the exception itself, it’s part of the logic that prints the exception, i.e. `showerror`. If you want to test that a reasonable error message is displayed to your users, then test the error message. If you want to test that your methods are correctly defined, just test the exception.

I would like to suggest an improvement to the code you posted above. In its current form, if the test fails, it doesn’t print what operation made it fail:

```julia
Test Failed at /.../error_message.jl:13
  Expression: err isa Exception
   Evaluated: nothing isa Exception
ERROR: LoadError: There was an error during testing

```

This can be quite annoying, for example if it happens on a build server, and the test informs you that something failed but not what. I would suggest something like this:

```julia
for op in opcoll
    e = try @eval $op() catch ex; ex; end
    e isa MethodError || throw(AssertionError("$op() should not be defined"))
end

```

Which tells you:

```julia
ERROR: LoadError: AssertionError: nobs() should not be defined

```

Alternatively, if you want to test the message itself:

```julia
for op in opcoll
    e = try @eval $op() catch ex; ex; end
    occursin("no method matching $op()", sprint(showerror, e)) ||
        throw(AssertionError("$op() should not be defined"))
end

```

---

<div class="post-metadata">

**Author:** ![jw3126](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jw3126/32/3086_2.png) [@jw3126](https://discourse.julialang.org/u/jw3126)\
**Post date:** [January 7, 2019, 10:42am UTC](https://discourse.julialang.org/t/how-to-test-a-default-error-raising-methods-message/19224/12 "2019-01-07T10:42:52Z")

</div>

If you need to test error messages in many places, it may be worth defining a macro. Here is an example, where I did this:  
[https://github.com/jw3126/ArgCheck.jl/blob/b7fcbfa0763f268b883dc629ffb30df784ce48ee/test/checks.jl#L3](https://github.com/jw3126/ArgCheck.jl/blob/b7fcbfa0763f268b883dc629ffb30df784ce48ee/test/checks.jl#L3)

---

<div class="post-metadata">

**Author:** ![taqtiqa-mark](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/taqtiqa-mark/32/4383_2.png) [@taqtiqa-mark](https://discourse.julialang.org/u/taqtiqa-mark)\
**Post date:** [January 8, 2019, 8:17pm UTC](https://discourse.julialang.org/t/how-to-test-a-default-error-raising-methods-message/19224/13 "2019-01-08T20:17:22Z")

</div>

> [@bennedich](#):
>
> I would suggest something like this:

Very nice. Thank you for the tips.

Given that the code, and insights, are not trivial are these worth proposing to add to `Base.Test`?