Human readable error messages

hendri54 · September 15, 2019, 12:09pm

I find run-time error messages that are displayed in the REPL hard to parse.
Perhaps this could be improved fairly easily. But first I wanted to check whether I am the only one who feels this way.

Here is a simple example with just a single error thrown by test:

Model: Error During Test at /Users/lutz/Documents/julia/OptimizationLH/test/runtests.jl:4
  Got exception outside of a @test
  BoundsError: attempt to access 4×4 Array{Int64,2} at index [100, 200]
  Stacktrace:
   [1] setindex! at ./array.jl:768 [inlined]
   [2] compute_stats(::Model, ::Array{Int64,2}, ::Array{Float64,2}) at /Users/lutz/Documents/julia/OptimizationLH/src/model.jl:98
   [3] solve(::Model, ::OptimizationLH.ModelParams) at /Users/lutz/Documents/julia/OptimizationLH/src/model.jl:31
   [4] top-level scope at /Users/lutz/Documents/julia/OptimizationLH/test/runtests.jl:22
   [5] top-level scope at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1113
   [6] top-level scope at /Users/lutz/Documents/julia/OptimizationLH/test/runtests.jl:5
   [7] include at ./boot.jl:328 [inlined]
   [8] include_relative(::Module, ::String) at ./loading.jl:1094
   [9] include(::Module, ::String) at ./Base.jl:31
   [10] include(::String) at ./client.jl:431
   [11] top-level scope at none:5
   [12] eval(::Module, ::Any) at ./boot.jl:330
   [13] exec_options(::Base.JLOptions) at ./client.jl:271
   [14] _start() at ./client.jl:464

All that I usually need to know at this point is:

I have a bounds error
it occurs in line 98 of compute_stats

I come from Matlab where errors are far more compact.

I think it would generally be desirable to suppress references to built-in (as opposed to user written) code (especially lines 5 to 14.

A little bit of formatting would make this easier to read (note the long lines giving the entire method signature followed by “at line number in file”).

Ideally, I would like to see condensed information that can be expanded if the user needs it. Perhaps even opened in an editor instead of the REPL?

Or perhaps this is just me and I need to generate fewer errors…

kristoffer.carlsson · September 15, 2019, 12:22pm

If the error would have been thrown inside a @test there is some functionality that tries to remove some unnecessary stuff from the backtrace.

julia> @testset "test" begin
           [][1]
       end
test: Error During Test at REPL[3]:1
  Got exception outside of a @test
  BoundsError: attempt to access 0-element Array{Any,1} at index [1]
  Stacktrace:
   [1] getindex(::Array{Any,1}, ::Int64) at ./array.jl:728
   [2] top-level scope at REPL[3]:2
   [3] top-level scope at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1113
   [4] top-level scope at REPL[3]:2
   [5] eval(::Module, ::Any) at ./boot.jl:330
   [6] eval_user_input(::Any, ::REPL.REPLBackend) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/REPL/src/REPL.jl:86
   [7] macro expansion at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/REPL/src/REPL.jl:118 [inlined]
   [8] (::getfield(REPL, Symbol("##26#27")){REPL.REPLBackend})() at ./task.jl:268
  
Test Summary: | Error  Total
test          |     1      1
ERROR: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.

julia> @testset "test" begin
           @test [][1]
       end
test: Error During Test at REPL[4]:2
  Test threw exception
  Expression: ([])[1]
  BoundsError: attempt to access 0-element Array{Any,1} at index [1]
  Stacktrace:
   [1] getindex(::Array{Any,1}, ::Int64) at ./array.jl:728
   [2] top-level scope at REPL[4]:2
   [3] top-level scope at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1113
   [4] top-level scope at REPL[4]:2
  
Test Summary: | Error  Total
test          |     1      1
ERROR: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.

Oscar_Smith · September 15, 2019, 12:32pm

Also note that in the past couple days, Julia’s master branch has had a bunch of stacktrace improvements especially for include and kwargs.

Tamas_Papp · September 15, 2019, 1:19pm

I am sure that stack traces could be improved, but generally it is not easy for Julia to figure out which part you would find “interesting”. Eg consider a BoundsError which

technically shows up in a getindex,
called inside a package A that accepts all AbstractArrays,
to which you passed a custom object from package B implementing AbstractArray

Even if we accept the fact that Base is fairly robust, both A and B could have bugs, or it could be a user error. With multiple dispatch, multiple levels this can happen easily, so determining a good threshold for cutting off the stack printing is not always trivial.

stevengj · September 15, 2019, 1:53pm

I could imagine a condensed stack trace that shows (1) the method that threw the error, (2) the first (topmost) method called in each module, and (3) all methods defined in Main.

Tamas_Papp · September 15, 2019, 2:35pm

That sounds like a good idea. It would be interesting to have a pluggable framework so packages could experiment with these things, like eg OhMyREPL.jl for the REPL.

ckoe-bccms · September 15, 2019, 4:34pm

Hello.

If I might add a small formatting issue which has bothered me a bit lately. Consider this example which on top might get line wrapped in the REPL depending on terminal width:

julia> function demonstration(a::Float64,b::Float64,c::Float64,d::Float64,e::Float64)
       end
demonstration (generic function with 1 method)

julia> demonstration(1.0,2.0,3,4.0,5.0)
ERROR: MethodError: no method matching demonstration(::Float64, ::Float64, ::Int64, ::Float64, ::Float64)
Closest candidates are:
  demonstration(::Float64, ::Float64, ::Float64, ::Float64, ::Float64) at REPL[1]:2
Stacktrace:
 [1] top-level scope at none:0

Would it make sense to change the formatting a bit so to at least align the beginning of all function definitions in the error message, e.g.

ERROR: MethodError: no method matching 
  demonstration(::Float64, ::Float64, ::Int64, ::Float64, ::Float64)
Closest candidates are:
  demonstration(::Float64, ::Float64, ::Float64, ::Float64, ::Float64) at REPL[1]:2

This would probably make it easier to match arguments between the different lines.

As it is now the

ERROR: MethodError: no method matching demonstration(::Float64, ::Float64, ::Int64, ::Float64, ::Float64)

is completely shifted. It gets worse if the function definition contains complicated type annotations and/or spans multiple lines and/or if there are multiple method definitions which do not match.

When not running in the REPL (?) non-matches are marked with a “!” as far as I recall but the basic mis-alignment is the same and it makes it harder to parse the error message at least for me.

A problem with alignment between the arguments when printing remains, i.e.

demonstration(::Int64, ::Int64, ::Int64, ::Int64, ::Int64)
demonstration(::Any, ::Any, ::Float64, ::Any, ::Any) at REPL[1]:2

vs.

demonstration(::Int64, ::Int64, ::Int64    , ::Int64, ::Int64)
demonstration(::Any  , ::Any  , ::Float64  , ::Any  , ::Any) at REPL[1]:2

(this looks right in the preview, although it is off in the editing window)
but I believe that is impossible to get right in general for complicated multi-line output.

This is with 1.0.4.

hendri54 · September 15, 2019, 4:45pm

Thank you for all the replies.

In the spirit of ckoe-bccms’s comment, I see some low hanging fruit here. For example:

Alignment (as proposed above)
Split lines at the “at” statement as in
[2] compute_stats(::Model, ::Array{Int64,2}, ::Array{Float64,2}) at /Users/lutz/Documents/julia/OptimizationLH/src/model.jl:98
Color code source lines that come from user / std libraries.
Omitting all but top lines from a module (see above).

The potential issue that I can see is that sometimes the user has to see all the details. Perhaps it would make sense to generate an Error object that can be displayed with varying levels of detail? That would be somewhat less low hanging fruit.

karlroyen · September 16, 2019, 3:52am

In my opinion this is the most important point, the problem with error messages is that they are only printed once and that is basically all in the information you will get (or at least I know how to get). So it has to contain everything that might be important. If the printed error message would contain something like a ticket number under which one could ask for more specific information about the error that would

make it clear that there is a way to get more information
reduce the necessary verbosity of the printed error message
make it possible to reprint the information even after the session has closed (which is important if you are not working in the REPL or if there are multiple errors you have to distinguish or compare)
make it possible for people to write their own error printing methods for the error object containing the information about the ticket

Having different questions (methods) one can ask to retrieve information from a ticket would also help with another problem which at least I have: Out of my head I couldn’t say how I parse error messages or which information from them I need the most during everyday work. I probably could have said during the first months of learning Julia but now I’m so much used to reading them that I would have to start explicitly monitor myself during everyday works error handling to be able to say how they could be improved.

Different methods to display the error in different styles or to show different parts of them would make it more explicit which style one ends up to use the most or which information you keep requesting about an error the most (“I noticed that most of the time I keep reprinting errors with emphasizemodule('<ticket>', MyModule), can’t be that the default way the message is printed?” or “I keep calling less('<ticket>', stacklevels=1:3) can’t the first three stack levels always contain a snippet of the code around those lines?”).

Long text short: I think hendri54s suggestion for having some way to generate an error object containing the information about an error is great because

people can implement their own display of error messages and
it reduces the necessity to find one display style to rule them all.

Tamas_Papp · September 16, 2019, 6:03am

Yes, having the last 10 or so stack traces saved globally for later inspection could be useful. Not sure about persistence (your 3rd point), that could be difficult to implement.

stevengj · September 16, 2019, 11:44am

If you are working interactively, i.e. if isinteractive() returns true, it is probably pretty safe to show an abbreviated stacktrace by default (with a global setting to disable this), and maybe have a lasterror() function to give verbose information on the last thrown error.

For non-interactive usage, we could continue printing the full stacktrace.

hendri54 · September 16, 2019, 3:34pm

Having a way of displaying the last several errors would be nice, though.
I’m thinking about calling test and getting a few errors at a time that one would like to page through using a nice interface (which would be a separate implementation from the record keeping; perhaps even in an IDE).

doomphoenix-qxz · September 16, 2019, 4:32pm

This is incidental to your comment about better error messages, but you can avoid a lot of those kinds of MethodErrors by writing your code like this:

function demonstration(a::T, b::T, c::T, d::T, e::T) where {T<:Real} end

function demonstration(a::Real, b::Real, c::Real, d::Real, e::Real) 
    return demonstration(promote(a, b, c, d, e)...)
end

(Assuming your function is supposed to apply only to real numbers.)

In particular, you will never get errors when you accidentally pass Ints instead of Floats to functions written this way. Here is the relevant section of the docs: https://docs.julialang.org/en/v1.0/manual/conversion-and-promotion/

ckoe-bccms · September 16, 2019, 4:44pm

That was just a throw-away example to provoke that error message with a somewhat confusing and complex function signature

hendri54 · September 18, 2019, 12:10pm

Now we have useful suggestions. How do we get them implemented?

I am not comfortable editing Julia’s Base code myself. Should I open an issue?

Or should someone who is more familiar with Julia internals do so?

Tamas_Papp · September 18, 2019, 12:26pm

I think that this issue covers a lot of points mentioned here:

github.com/JuliaLang/julia

Stacktrace gripes

opened 06:31PM - 24 Aug 19 UTC

antoine-levitt

error handling display and printing

Stacktraces are in general pretty hard to read. Here's a list of gripes. I'm not… posting this as separate issues because I don't know what is bug and what is feature here. All tests are on 1.3alpha from an official release. This is related but not the same as https://github.com/JuliaLang/julia/issues/25375, which is about formatting. Let's start small: ``` julia> sqrt(-2) ERROR: DomainError with -2.0: sqrt will only return a complex result if called with a complex argument. Try sqrt(Complex(x)). Stacktrace: [1] throw_complex_domainerror(::Symbol, ::Float64) at ./math.jl:32 [2] sqrt at ./math.jl:492 [inlined] [3] sqrt(::Int64) at ./math.jl:518 [4] top-level scope at REPL[4]:1 ``` * Where is `./math.jl`? * I don't care that [2] is inlined * The last frame is unnecessary (the :1 definitely is, and I don't know what the [4] is) Now a stdlib: ``` julia> lu([0 0;0 0]) ERROR: SingularException(1) Stacktrace: [1] checknonsingular at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/LinearAlgebra/src/factorization.jl:19 [inlined] [2] #lu!#128(::Bool, ::typeof(lu!), ::Array{Float64,2}, ::Val{true}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/LinearAlgebra/src/lu.jl:85 [3] #lu! at ./none:0 [inlined] [4] #lu#134(::Bool, ::typeof(lu), ::Array{Int64,2}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/LinearAlgebra/src/lu.jl:295 [5] lu(::Array{Int64,2}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/LinearAlgebra/src/lu.jl:286 [6] top-level scope at REPL[2]:1 ``` * The paths are bad; that means that nice things like ctrl-q will not work, and neither will the functionality in some editors that go to the path. * The keyword arguments lowering, which should really be an implementation detail, is exposed here, and confusing (wtf is [3]?) `include`d user files: ``` julia> include("scratch.jl") ERROR: LoadError: UndefVarError: qsd not defined Stacktrace: [1] f() at /home/antoine/scratch.jl:2 [2] top-level scope at /home/antoine/scratch.jl:4 [3] include at ./boot.jl:328 [inlined] [4] include_relative(::Module, ::String) at ./loading.jl:1094 [5] include(::Module, ::String) at ./Base.jl:31 [6] include(::String) at ./client.jl:432 [7] top-level scope at REPL[3]:1 in expression starting at /home/antoine/scratch.jl:4 ``` * Five frames for the include are hard to read * Having the information at the bottom is confusing: where am I supposed to find the source of the error? In the stacktrace or at the bottom? * The LoadError: UndefVarError is annoying (it gets worse for more complex cases, ie multiple includes) Failing tests: ``` ERROR: LoadError: LoadError: LoadError: DomainError with -2.0: sqrt will only return a complex result if called with a complex argument. Try sqrt(Complex(x)). Stacktrace: [1] throw_complex_domainerror(::Symbol, ::Float64) at ./math.jl:32 [2] sqrt at ./math.jl:492 [inlined] [3] sqrt(::Int64) at ./math.jl:518 [4] @SArray(::LineNumberNode, ::Module, ::Any) at /home/antoine/.julia/packages/StaticArrays/3KEjZ/src/SArray.jl:83 [5] include at ./boot.jl:328 [inlined] [6] include_relative(::Module, ::String) at ./loading.jl:1094 [7] include(::Module, ::String) at ./Base.jl:31 [8] include(::String) at ./client.jl:432 [9] top-level scope at /home/antoine/.julia/packages/StaticArrays/3KEjZ/test/runtests.jl:19 [10] include at ./boot.jl:328 [inlined] [11] include_relative(::Module, ::String) at ./loading.jl:1094 [12] include(::Module, ::String) at ./Base.jl:31 [13] include(::String) at ./client.jl:432 [14] top-level scope at none:5 in expression starting at /home/antoine/.julia/packages/StaticArrays/3KEjZ/test/SArray.jl:30 in expression starting at /home/antoine/.julia/packages/StaticArrays/3KEjZ/test/SArray.jl:1 in expression starting at /home/antoine/.julia/packages/StaticArrays/3KEjZ/test/runtests.jl:19 ERROR: Package StaticArrays errored during testing ``` I'm a bit confused about what's going on here to be honest... I did not try with the other ways of including code, macros, etc. So, to sum up, in subjective increasing order of seriousness * There are minor printing things that could be nicer: REPL frames, LoadError nesting, inline, `./` path for base * The keyword argument lowering is a bit nasty * The paths on official releases should be corrected * The stacktraces with `include` are very hard to parse. This is possibly the most controversial issue because enhancing this would mean "lying" to the user, ie treating `include` as a special function

so you may just want to contribute there.

ChrisRackauckas · October 8, 2022, 6:15pm

The SciML libraries got a ton of customized error messages to improve the situation:

While better default error messages could be helpful, I have found that in many cases higher level error throwing can add in more context and be really beneficial.