Long stack trace, identify precise error location

erlebach · May 3, 2023, 11:09pm

I am trying to solve a UODE where the neural network uses a tensor-basis neural net. Here are the first few lines of my stack trace (the details are not important since I am not asking you to debug my code ):

ERROR: MethodError: no method matching init(::Nothing, ::Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}; sensealg::InterpolatingAdjoint{0, true, Val{:central}, ReverseDiffVJP{true}}, saveat::Float64)

Closest candidates are:
  init(::OptimizationProblem, ::Any, ::Any...; kwargs...)
   @ SciMLBase ~/.julia/packages/SciMLBase/VdcHg/src/solve.jl:146
  init(::PDEProblem, ::SciMLBase.AbstractDEAlgorithm, ::Any...; kwargs...)
   @ DiffEqBase ~/.julia/packages/DiffEqBase/ihYDa/src/solve.jl:1011
  init(::SciMLBase.AbstractJumpProblem, ::Any...; kwargs...)
   @ DiffEqBase ~/.julia/packages/DiffEqBase/ihYDa/src/solve.jl:441
  ...

Stacktrace:
  [1] macro expansion
    @ ~/.julia/packages/Zygote/SuKWp/src/compiler/interface2.jl:0 [inlined]
  [2] _pullback(::Zygote.Context{false}, ::typeof(Core.kwcall), ::NamedTuple{(:sensealg, :saveat), Tuple{InterpolatingAdjoint{0, true, Val{:central}, ReverseDiffVJP{true}}, Float64}}, ::typeof(init), ::Nothing, ::Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False})
    @ Zygote ~/.julia/packages/Zygote/SuKWp/src/compiler/interface2.jl:9
  [3] _apply(::Function, ::Vararg{Any})
    @ Core ./boot.jl:838

The first line is clear enough:

ERROR: MethodError: no method matching init(::Nothing, ::Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}; sensealg::InterpolatingAdjoint{0, true, Val{:central}, ReverseDiffVJP{true}}, saveat::Float64)

I followed the very long trace, and I did not find the specific line where this function was executed, and therefore, the error is hard to fix. Given the very long trace, it is hard to believe that this line is not available to the stack trace.

Perhaps there is too much inlining going on. Is it possible to run Julia from the command line with options such that inlining is disabled? That might help. Thanks.

ChrisRackauckas · May 3, 2023, 11:50pm

Hard to tell without any code, but it sounds like you did solve(nothing, Tsit5()) somewhere instead of passing an ODEProblem?

erlebach · May 3, 2023, 11:52pm

Of course, what you say is logical, but if it is the case, it is not obvious at all. That is the problem with using high-level interface. The error is happening somewhere within my Ensemble functions. To solve the problem, you’d have to see the entire code, but I should create a MWE, which again, will take time. I have decided to learn how to use Debugging.jl.

hendri54 · May 4, 2023, 12:41pm

Perhaps relevant: There are several packages, such as RelevanceStrackTraces.jl that are designed to make it easier to find the place in the user code that caused the exception to be thrown.

erlebach · May 4, 2023, 2:39pm

Thank you. I had not heard of this. Of course, I’ll try it out.
So far, I have been unsuccessful in my attempts to use Debugger.jl.

hendri54 · May 4, 2023, 3:49pm

What I generally do in cases of deeply nested calls is to find the first item in the stacktrace where one of my functions occurs. RelevanceStrackTraces is designed to point to that place clearly.
Then I insert an @assert !isnothing(x) type of statement there (which I just keep unless the code is really performance critical). Then I work my way backwards through the stacktrace.
I have not had much luck with the debugger, but I rarely miss it.

erlebach · May 4, 2023, 6:12pm

I have a question. When getting a stack trace that starts of as:

ERROR: MethodError: no method matching (::Chain{Tuple{Dense{typeof(tanh), Matrix{Float32}, …}, Dense{typeof(identity), Matrix{Float32}, …}}})(::Vector{Float64}, ::Vector{Float32})
Closest candidates are:
  (::Chain)(::Any)
   @ Flux ~/.julia/packages/Flux/FWgS0/src/layers/basic.jl:51

Stacktrace:
   [1] tbnn_opt(σ::Matrix{Float32}, D::SMatrix{3, 3, Float64, 9}, model_weights::Vector{Float32}, model_univ::Chain{Tuple{Dense{typeof(tanh), Matrix{Float32}, …}, Dense{typeof(identity), Matrix{Float32}, …}}}, t::Float32)
     @ Main ~/src/2022/rude/giesekus/GE_rude.jl/alex_report_code_2023-03-24/julia19/rude_giesekus_optimized_reduced_scalar.jl:67
   [2] dudt_univ_opt!(

would it be hard to provide the line number and file that generated the error starting with

ERROR: MethodError: no method matching (::Chain{Tuple{Dense{typeof(tanh), Matrix{Float32}, …}, Dense{typeof(identity), Matrix{Float32}, …}}})(::Vector{Float64}, ::Vector{Float32})

Why can’t the information be provided on the very same line together with a link. Isn’t this information available? I still have not figured out which function generated my original message let alone this one. Thanks.

joa-quim · May 4, 2023, 6:22pm

I fee your pain with the inscrutable stack traces.
The debugger is simple to use (not saying it’s very good) from Visual Studio Code. Just do
@run your_command and it will stop somewhere, than you can follow the stacktrace on the left side till you find the point where the error originated.

erlebach · May 4, 2023, 6:31pm

@run does not work in REPL.
I love Julia, but it is not written with ease of use in mind. I absolutely do not understand why Julia does not have built in options to manipulate the stack traces in useful ways. Even the long traces shouldn’t be the default. The original error should clearly state what file and what line it occurred (or a range of lines). Sorry for the rant.

ToucheSir · May 4, 2023, 6:39pm

Maybe I’m misunderstanding your question, but isn’t this what

erlebach:

   [1] tbnn_opt(σ::Matrix{Float32}, D::SMatrix{3, 3, Float64, 9}, model_weights::Vector{Float32}, model_univ::Chain{Tuple{Dense{typeof(tanh), Matrix{Float32}, …}, Dense{typeof(identity), Matrix{Float32}, …}}}, t::Float32)
     @ Main ~/src/2022/rude/giesekus/GE_rude.jl/alex_report_code_2023-03-24/julia19/rude_giesekus_optimized_reduced_scalar.jl:67

is showing? The second line starting with @ there shows (in order) the module, file name and line number on which the error occurred.

joa-quim · May 4, 2023, 6:46pm

I said from Visual Studio Code (it’s own REPL). But it does also from the normal REPL if you install Debugger package, though I’ve no experience with it.

adienes · May 4, 2023, 6:47pm

the debugger is indeed simple to attempt to use, but as far as my experience attempting goes, it doesn’t work particularly well (to the extent where I end up just returning to print debugging)

erlebach · May 4, 2023, 7:05pm

@TouchSir,

No. The line that says tbnn is one of my function calls. There is no reference to Chain on that line.

ToucheSir · May 4, 2023, 7:17pm

Oh I see. So that line is actually part of the MethodError error message and not the stacktrace, though I can see why it might be confusing if you’re not familiar with the error format.

How to interpret this is that at rude_giesekus_optimized_reduced_scalar.jl:67, tbnn_opt calls a Chain struct. You should be able to confirm that by looking at the line in question. no method matching (::Chain{Tuple{Dense{typeof(tanh), Matrix{Float32}, …}, Dense{typeof(identity), Matrix{Float32}, …}}})(::Vector{Float64}, ::Vector{Float32}) looks intimidating, but that’s mostly because the type parameters take up a lot of space. If we remove them, that line becomes no method matching (::Chain)(::Vector{Float64}, ::Vector{Float32}).

As you can tell, Flux.Chain only accepts one argument when called, so something is erroneously passing two. The MethodError message tries to help here by referencing (::Chain)(::Any) (note one arg only) and linking to its definition here so you can investigate further.

erlebach · May 4, 2023, 8:03pm

Thanks for the feedback. I do know how to interpret a trace, or thought I did. Concerning Chain and the fact that it has only one argument, I beg to differ. Running methods Flux.Chain, I get the result:

methods(Flux.Chain)
# 3 methods for type constructor:
 [1] Chain(; kw...)
     @ ~/.julia/packages/Flux/FWgS0/src/layers/basic.jl:40
 [2] Chain(layers::T) where T<:Union{Tuple, NamedTuple, AbstractVector}
     @ ~/.julia/packages/Flux/FWgS0/src/layers/basic.jl:36
 [3] Chain(xs...)
     @ ~/.julia/packages/Flux/FWgS0/src/layers/basic.jl:39

Doesn’t the third choice (and even the second) imply that multiple arguments are accepted, contrary to what the error message implies?

erlebach · May 4, 2023, 8:18pm

@ToucheSir : Here is some of my code:

function tbnn_opt(σ, D, model_weights, model_univ, t)
    # Compute elements of the tensor basis
    I  = SA[1. 0. 0.; 0. 1. 0.; 0. 0. 1.]

    # Compute the integrity basis from scalar invariants. Traces. 
    λ = zeros(1) # must be mutale
    λ[1] = tr(σ)
    println("input to NN, λ: ", λ)
    g = model_univ(λ, model_weights)    ### Error
    F = g[1] .* I 
end

# NN model for the nonlinear function F(σ,γ̇)
model_univ = Flux.Chain(Flux.Dense(1, 8, tanh),
                       Flux.Dense(8, 1))
p_model, re = Flux.destructure(model_univ)

# Parameters of the linear response (η0,τ)
p_system = Float32[1, 1]

Yes, I am using Julia 1.9, and so should be using Lux. But surely Flux has not been deprecated, right?

Look at the page: Built-in Layers · Flux, the stable version of Flux.jl . Tons of examples where Flux.Chain take multiple arguments. So I still do not undertand the error.

ToucheSir · May 4, 2023, 8:35pm

The constructor mymodel = Chain(layer1, layer2, ...) takes multiple arguments. Calling that model mymodel(x) takes just one. I’m not sure what model_weights and model_univ are meant to be in tbnn_opt because you generally wouldn’t separate model structure from weights in Flux, so this seems like a better question for one of the SciML help channels.

This also seems like something specific to only SciML and should likewise be asked there

erlebach · May 4, 2023, 8:37pm

Ok, will think some more and do that. I have never posted to specific channels yet.