Human readable error messages

I find run-time error messages that are displayed in the REPL hard to parse.
Perhaps this could be improved fairly easily. But first I wanted to check whether I am the only one who feels this way.

Here is a simple example with just a single error thrown by test:

Model: Error During Test at /Users/lutz/Documents/julia/OptimizationLH/test/runtests.jl:4
  Got exception outside of a @test
  BoundsError: attempt to access 4×4 Array{Int64,2} at index [100, 200]
  Stacktrace:
   [1] setindex! at ./array.jl:768 [inlined]
   [2] compute_stats(::Model, ::Array{Int64,2}, ::Array{Float64,2}) at /Users/lutz/Documents/julia/OptimizationLH/src/model.jl:98
   [3] solve(::Model, ::OptimizationLH.ModelParams) at /Users/lutz/Documents/julia/OptimizationLH/src/model.jl:31
   [4] top-level scope at /Users/lutz/Documents/julia/OptimizationLH/test/runtests.jl:22
   [5] top-level scope at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1113
   [6] top-level scope at /Users/lutz/Documents/julia/OptimizationLH/test/runtests.jl:5
   [7] include at ./boot.jl:328 [inlined]
   [8] include_relative(::Module, ::String) at ./loading.jl:1094
   [9] include(::Module, ::String) at ./Base.jl:31
   [10] include(::String) at ./client.jl:431
   [11] top-level scope at none:5
   [12] eval(::Module, ::Any) at ./boot.jl:330
   [13] exec_options(::Base.JLOptions) at ./client.jl:271
   [14] _start() at ./client.jl:464

All that I usually need to know at this point is:

  • I have a bounds error
  • it occurs in line 98 of compute_stats

I come from Matlab where errors are far more compact.

I think it would generally be desirable to suppress references to built-in (as opposed to user written) code (especially lines 5 to 14.

A little bit of formatting would make this easier to read (note the long lines giving the entire method signature followed by “at line number in file”).

Ideally, I would like to see condensed information that can be expanded if the user needs it. Perhaps even opened in an editor instead of the REPL?

Or perhaps this is just me and I need to generate fewer errors…

8 Likes

If the error would have been thrown inside a @test there is some functionality that tries to remove some unnecessary stuff from the backtrace.

julia> @testset "test" begin
           [][1]
       end
test: Error During Test at REPL[3]:1
  Got exception outside of a @test
  BoundsError: attempt to access 0-element Array{Any,1} at index [1]
  Stacktrace:
   [1] getindex(::Array{Any,1}, ::Int64) at ./array.jl:728
   [2] top-level scope at REPL[3]:2
   [3] top-level scope at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1113
   [4] top-level scope at REPL[3]:2
   [5] eval(::Module, ::Any) at ./boot.jl:330
   [6] eval_user_input(::Any, ::REPL.REPLBackend) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/REPL/src/REPL.jl:86
   [7] macro expansion at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/REPL/src/REPL.jl:118 [inlined]
   [8] (::getfield(REPL, Symbol("##26#27")){REPL.REPLBackend})() at ./task.jl:268
  
Test Summary: | Error  Total
test          |     1      1
ERROR: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.

julia> @testset "test" begin
           @test [][1]
       end
test: Error During Test at REPL[4]:2
  Test threw exception
  Expression: ([])[1]
  BoundsError: attempt to access 0-element Array{Any,1} at index [1]
  Stacktrace:
   [1] getindex(::Array{Any,1}, ::Int64) at ./array.jl:728
   [2] top-level scope at REPL[4]:2
   [3] top-level scope at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1113
   [4] top-level scope at REPL[4]:2
  
Test Summary: | Error  Total
test          |     1      1
ERROR: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.

Also note that in the past couple days, Julia’s master branch has had a bunch of stacktrace improvements especially for include and kwargs.

1 Like

I am sure that stack traces could be improved, but generally it is not easy for Julia to figure out which part you would find “interesting”. Eg consider a BoundsError which

  1. technically shows up in a getindex,
  2. called inside a package A that accepts all AbstractArrays,
  3. to which you passed a custom object from package B implementing AbstractArray

Even if we accept the fact that Base is fairly robust, both A and B could have bugs, or it could be a user error. With multiple dispatch, multiple levels this can happen easily, so determining a good threshold for cutting off the stack printing is not always trivial.

1 Like

I could imagine a condensed stack trace that shows (1) the method that threw the error, (2) the first (topmost) method called in each module, and (3) all methods defined in Main.

4 Likes

That sounds like a good idea. It would be interesting to have a pluggable framework so packages could experiment with these things, like eg OhMyREPL.jl for the REPL.

1 Like

Hello.

If I might add a small formatting issue which has bothered me a bit lately. Consider this example which on top might get line wrapped in the REPL depending on terminal width:

julia> function demonstration(a::Float64,b::Float64,c::Float64,d::Float64,e::Float64)
       end
demonstration (generic function with 1 method)

julia> demonstration(1.0,2.0,3,4.0,5.0)
ERROR: MethodError: no method matching demonstration(::Float64, ::Float64, ::Int64, ::Float64, ::Float64)
Closest candidates are:
  demonstration(::Float64, ::Float64, ::Float64, ::Float64, ::Float64) at REPL[1]:2
Stacktrace:
 [1] top-level scope at none:0

Would it make sense to change the formatting a bit so to at least align the beginning of all function definitions in the error message, e.g.

ERROR: MethodError: no method matching 
  demonstration(::Float64, ::Float64, ::Int64, ::Float64, ::Float64)
Closest candidates are:
  demonstration(::Float64, ::Float64, ::Float64, ::Float64, ::Float64) at REPL[1]:2

This would probably make it easier to match arguments between the different lines.

As it is now the

ERROR: MethodError: no method matching demonstration(::Float64, ::Float64, ::Int64, ::Float64, ::Float64)

is completely shifted. It gets worse if the function definition contains complicated type annotations and/or spans multiple lines and/or if there are multiple method definitions which do not match.

When not running in the REPL (?) non-matches are marked with a “!” as far as I recall but the basic mis-alignment is the same and it makes it harder to parse the error message at least for me.

A problem with alignment between the arguments when printing remains, i.e.

demonstration(::Int64, ::Int64, ::Int64, ::Int64, ::Int64)
demonstration(::Any, ::Any, ::Float64, ::Any, ::Any) at REPL[1]:2

vs.

demonstration(::Int64, ::Int64, ::Int64    , ::Int64, ::Int64)
demonstration(::Any  , ::Any  , ::Float64  , ::Any  , ::Any) at REPL[1]:2

(this looks right in the preview, although it is off in the editing window)
but I believe that is impossible to get right in general for complicated multi-line output.

This is with 1.0.4.

2 Likes

Thank you for all the replies.

In the spirit of ckoe-bccms’s comment, I see some low hanging fruit here. For example:

  1. Alignment (as proposed above)
  2. Split lines at the “at” statement as in
    [2] compute_stats(::Model, ::Array{Int64,2}, ::Array{Float64,2}) at /Users/lutz/Documents/julia/OptimizationLH/src/model.jl:98
  3. Color code source lines that come from user / std libraries.
  4. Omitting all but top lines from a module (see above).

The potential issue that I can see is that sometimes the user has to see all the details. Perhaps it would make sense to generate an Error object that can be displayed with varying levels of detail? That would be somewhat less low hanging fruit.

7 Likes

In my opinion this is the most important point, the problem with error messages is that they are only printed once and that is basically all in the information you will get (or at least I know how to get). So it has to contain everything that might be important. If the printed error message would contain something like a ticket number under which one could ask for more specific information about the error that would

  1. make it clear that there is a way to get more information
  2. reduce the necessary verbosity of the printed error message
  3. make it possible to reprint the information even after the session has closed (which is important if you are not working in the REPL or if there are multiple errors you have to distinguish or compare)
  4. make it possible for people to write their own error printing methods for the error object containing the information about the ticket

Having different questions (methods) one can ask to retrieve information from a ticket would also help with another problem which at least I have: Out of my head I couldn’t say how I parse error messages or which information from them I need the most during everyday work. I probably could have said during the first months of learning Julia but now I’m so much used to reading them that I would have to start explicitly monitor myself during everyday works error handling to be able to say how they could be improved.

Different methods to display the error in different styles or to show different parts of them would make it more explicit which style one ends up to use the most or which information you keep requesting about an error the most (“I noticed that most of the time I keep reprinting errors with emphasizemodule('<ticket>', MyModule), can’t be that the default way the message is printed?” or “I keep calling less('<ticket>', stacklevels=1:3) can’t the first three stack levels always contain a snippet of the code around those lines?”).

Long text short: I think hendri54s suggestion for having some way to generate an error object containing the information about an error is great because

  1. people can implement their own display of error messages and
  2. it reduces the necessity to find one display style to rule them all.
5 Likes

Yes, having the last 10 or so stack traces saved globally for later inspection could be useful. Not sure about persistence (your 3rd point), that could be difficult to implement.

If you are working interactively, i.e. if isinteractive() returns true, it is probably pretty safe to show an abbreviated stacktrace by default (with a global setting to disable this), and maybe have a lasterror() function to give verbose information on the last thrown error.

For non-interactive usage, we could continue printing the full stacktrace.

4 Likes

Having a way of displaying the last several errors would be nice, though.
I’m thinking about calling test and getting a few errors at a time that one would like to page through using a nice interface (which would be a separate implementation from the record keeping; perhaps even in an IDE).

This is incidental to your comment about better error messages, but you can avoid a lot of those kinds of MethodErrors by writing your code like this:

function demonstration(a::T, b::T, c::T, d::T, e::T) where {T<:Real} end

function demonstration(a::Real, b::Real, c::Real, d::Real, e::Real) 
    return demonstration(promote(a, b, c, d, e)...)
end

(Assuming your function is supposed to apply only to real numbers.)

In particular, you will never get errors when you accidentally pass Ints instead of Floats to functions written this way. Here is the relevant section of the docs: https://docs.julialang.org/en/v1.0/manual/conversion-and-promotion/

That was just a throw-away example to provoke that error message with a somewhat confusing and complex function signature :slight_smile:

Now we have useful suggestions. How do we get them implemented?

I am not comfortable editing Julia’s Base code myself. Should I open an issue?

Or should someone who is more familiar with Julia internals do so?

I think that this issue covers a lot of points mentioned here:

so you may just want to contribute there.

1 Like

The SciML libraries got a ton of customized error messages to improve the situation:

While better default error messages could be helpful, I have found that in many cases higher level error throwing can add in more context and be really beneficial.

4 Likes