How to test a default `error(...)` raising method's message

New to Julia
1

One way to understand the usage of a Julia package is to read the test suite…

Looking at StatsBase.coef(…) I wondered how/when I should expect to see that error message.

The test suite is silent.

What is the correct way to test that error message is observed?

Aside:
This arose in the course of a different issue, but I thought best to isolate this specific question: the correct way to test a message that is raised by falling through to a generic definition.

1 Like
2

Something like

using Test
try
    error("bla")
catch e
    buf = IOBuffer()
    showerror(buf, e)
    message = String(take!(buf))
    @test message == "bla"
end
2 Likes
3

I’d write something like

let err = nothing
    try
        error("bla")
    catch err
    end

    @test err isa Exception
    @test sprint(showerror, err) == "bla"
end

so that it fails when the code inside try block does not throw. (Also I think we can use sprint here)

5 Likes
5

There is also @test_throws:

julia> using Test

julia> f() = error("bla");

julia> @test_throws ErrorException("bla") f()
Test Passed
      Thrown: ErrorException
13 Likes
6

Just a note on why this wasn’t selected as the solution… it doesn’t seem to be the way that will generally work.

Specifically, when looping over a list of functions (such as those at the github link above):

 opcoll = (:coef, :coefnames, :coeftable, :islinear, :nobs, :params, :weights)
        for op in opcoll
            let err = nothing
                try
                    @eval $op()
                catch err
                end
                @test_throws ErrorException("MethodError: no method matching $op()") @eval($op())
                # @test err isa Exception
                # @test err isa MethodError
                # @test occursin("MethodError: no method matching $op()", sprint(showerror, err)) == true
            end
        end
.... 
    Expected: ErrorException("MethodError: no method matching params()")
      Thrown: MethodError(Moments.params, (), 0x00000000000061d2)
7

Right, it is only useful if you can construct the exception easily, as with ErrorException in my example.

8

It would be nice if we can write, e.g.,

err = @test_throws SomeExceptionType f()
@test occursin("some error message", sprint(showerror, err))
1 Like
9

There’s this:

opcoll = (:coef, :coefnames, :coeftable, :islinear, :nobs, :params, :weights)

world_age = ccall(:jl_get_tls_world_age, UInt, ())

for op in opcoll
    @test_throws MethodError((@eval $op), (), world_age) @eval $op()
end
10

I’ll have to dig into world_age to fully appreciate what deeper testing this allows… on face value, it appears that approach would not be testing the code satisfied some (human) visible requirement.

For example: In one variation of the approach I outlined I discovered a typo in the source code for one of the methods/functions. This was only exposed by having the ‘correct’ error message defined in the test case.

11

Keep in mind that the message (“no method matching…”) is not part of the exception itself, it’s part of the logic that prints the exception, i.e. showerror. If you want to test that a reasonable error message is displayed to your users, then test the error message. If you want to test that your methods are correctly defined, just test the exception.

I would like to suggest an improvement to the code you posted above. In its current form, if the test fails, it doesn’t print what operation made it fail:

Test Failed at /.../error_message.jl:13
  Expression: err isa Exception
   Evaluated: nothing isa Exception
ERROR: LoadError: There was an error during testing

This can be quite annoying, for example if it happens on a build server, and the test informs you that something failed but not what. I would suggest something like this:

for op in opcoll
    e = try @eval $op() catch ex; ex; end
    e isa MethodError || throw(AssertionError("$op() should not be defined"))
end

Which tells you:

ERROR: LoadError: AssertionError: nobs() should not be defined

Alternatively, if you want to test the message itself:

for op in opcoll
    e = try @eval $op() catch ex; ex; end
    occursin("no method matching $op()", sprint(showerror, e)) ||
        throw(AssertionError("$op() should not be defined"))
end
1 Like
12

If you need to test error messages in many places, it may be worth defining a macro. Here is an example, where I did this:
https://github.com/jw3126/ArgCheck.jl/blob/b7fcbfa0763f268b883dc629ffb30df784ce48ee/test/checks.jl#L3

13

Very nice. Thank you for the tips.

Given that the code, and insights, are not trivial are these worth proposing to add to Base.Test?

2 Likes