# Very long stack trace

**URL:** <https://discourse.julialang.org/t/very-long-stack-trace/92025>\
**Category:** Performance\
**Created:** [December 22, 2022, 11:09pm UTC](https://discourse.julialang.org/t/very-long-stack-trace/92025 "2022-12-22T23:09:51Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![erlebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erlebach/32/12973_2.png) [@erlebach](https://discourse.julialang.org/u/erlebach)\
**Post date:** [December 22, 2022, 11:09pm UTC](https://discourse.julialang.org/t/very-long-stack-trace/92025/1 "2022-12-22T23:09:52Z")

</div>

I am running a SciML type program called RUDE. The code generates an error, with a stack trace of almost 1000 lines long. I attach the trace herein. Perhaps somebody could give me sense of what the error is about? Perhaps an “out of memory error”? If so, perhaps the software responsible for the trace could be appropriately shorted? I have no suggestions on how to do it though. Thanks.[Stack-Trace (1000 lines long)](https://drive.google.com/file/d/1VPszHfhnj7CIMaUH1MJG8DF2HQYWMuvc/view?usp=sharing)

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [December 22, 2022, 11:19pm UTC](https://discourse.julialang.org/t/very-long-stack-trace/92025/2 "2022-12-22T23:19:54Z")

</div>

This is an uncaught Enzyme error at the LLVM level. Could you share the program that generates this error as an issue for Enzyme.jl? That is what would be required to turn the LLVM assertion into a Julia-side error.

---

<div class="post-metadata">

**Author:** ![erlebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erlebach/32/12973_2.png) [@erlebach](https://discourse.julialang.org/u/erlebach)\
**Post date:** [December 22, 2022, 11:28pm UTC](https://discourse.julialang.org/t/very-long-stack-trace/92025/3 "2022-12-22T23:28:01Z")

</div>

Ok. On the GitHub repository I assume? Or here?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [December 22, 2022, 11:29pm UTC](https://discourse.julialang.org/t/very-long-stack-trace/92025/4 "2022-12-22T23:29:24Z")

</div>

> **[Issues · EnzymeAD/Enzyme.jl](https://github.com/EnzymeAD/Enzyme.jl/issues)**
>
> Julia bindings for the Enzyme automatic differentiator - Issues · EnzymeAD/Enzyme.jl

---

<div class="post-metadata">

**Author:** ![erlebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erlebach/32/12973_2.png) [@erlebach](https://discourse.julialang.org/u/erlebach)\
**Post date:** [December 22, 2022, 11:44pm UTC](https://discourse.julialang.org/t/very-long-stack-trace/92025/5 "2022-12-22T23:44:07Z")

</div>

> [@erlebach](#):
>
> I am

Done.

---

<div class="post-metadata">

**Author:** ![erlebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erlebach/32/12973_2.png) [@erlebach](https://discourse.julialang.org/u/erlebach)\
**Post date:** [December 23, 2022, 2:56pm UTC](https://discourse.julialang.org/t/very-long-stack-trace/92025/6 "2022-12-23T14:56:27Z")

</div>

I created a new project, and added the Zygote.jl package. Checking the toml files, I find that version 0.6.51 is loaded. Yet, the version is at 0.10.x . Why would this be? Thanks.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [December 23, 2022, 3:32pm UTC](https://discourse.julialang.org/t/very-long-stack-trace/92025/7 "2022-12-23T15:32:37Z")

</div>

If you tell it to add the latest, what does it say?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [December 23, 2022, 3:33pm UTC](https://discourse.julialang.org/t/very-long-stack-trace/92025/8 "2022-12-23T15:33:59Z")

</div>

BTW, forgot to post that the fix for your case is to just not let it test Enzyme at all and just add `sensealg = ReverseDiffVJP()` (or `sensealg = ReverseDiffVJP(true)`) to the solve call. But we should get this assertion fixed in Enzyme anyways.

---

<div class="post-metadata">

**Author:** ![erlebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erlebach/32/12973_2.png) [@erlebach](https://discourse.julialang.org/u/erlebach)\
**Post date:** [December 23, 2022, 4:20pm UTC](https://discourse.julialang.org/t/very-long-stack-trace/92025/11 "2022-12-23T16:20:51Z")

</div>

I removed the `callback` argument, and ran the `solve` for five iterations. No errors.  
My issues appear to be fixed at this time. Thanks @ChrisRackauckas for the help!  
Have a nice Christmas!

Gordon

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [December 23, 2022, 4:36pm UTC](https://discourse.julialang.org/t/very-long-stack-trace/92025/12 "2022-12-23T16:36:17Z")

</div>

What did you run?

---

<div class="post-metadata">

**Author:** ![erlebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erlebach/32/12973_2.png) [@erlebach](https://discourse.julialang.org/u/erlebach)\
**Post date:** [December 23, 2022, 4:45pm UTC](https://discourse.julialang.org/t/very-long-stack-trace/92025/13 "2022-12-23T16:45:46Z")

</div>

I ran the following (my loss now increases):

```julia
k = 1
loss_fn(θ) = loss_univ([θ; p_system], protocols[1:k], tspans[1:k], σ0, σ12_all, k)
cb_fun(θ, l) = callback(θ, l, protocols[1:k], tspans[1:k], σ0, σ12_all, k)
adtype = Optimization.AutoZygote()
optf = Optimization.OptimizationFunction((x,p)->loss_fn(x), adtype)
optprob = Optimization.OptimizationProblem(optf, θi) # get_parameter_values(nn_eqs)) # nn_eqs???
parameter_res = Optimization.solve(optprob, Optimisers.AMSGrad(), callback=cb_fun, sensealg = ReverseDiffVJP(true), maxiters=50)
global θi = result_univ.minimizer

```

All works now. I tried with and without the `sensealg` you suggested. All works now. But of course, I have the new `Enzyme.jl` package.  
I need to figure out how to best use the output of the solve.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [December 23, 2022, 5:06pm UTC](https://discourse.julialang.org/t/very-long-stack-trace/92025/14 "2022-12-23T17:06:57Z")

</div>

Cool, and Enzyme will release its patch and that should improve the errors.

---

<div class="post-metadata">

**Author:** ![erlebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erlebach/32/12973_2.png) [@erlebach](https://discourse.julialang.org/u/erlebach)\
**Post date:** [December 23, 2022, 5:22pm UTC](https://discourse.julialang.org/t/very-long-stack-trace/92025/15 "2022-12-23T17:22:47Z")

</div>

Excellent! I am glad you were able to help.

Gordon

---

<div class="post-metadata">

**Author:** ![erlebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erlebach/32/12973_2.png) [@erlebach](https://discourse.julialang.org/u/erlebach)\
**Post date:** [December 23, 2022, 9:10pm UTC](https://discourse.julialang.org/t/very-long-stack-trace/92025/16 "2022-12-23T21:10:58Z")

</div>

@ChrisRackauckas ,

I’ll continue the thread here. I was able to run `Optimization.solve` several hundred iterations before. But now, I get a strange error:

```julia
┌ Warning: dt(9.536743e-7) <= dtmin(9.536743e-7) at t=7.84356. Aborting. There is either an error in your model specification or the true solution is unstable.
└ @ SciMLBase ~/.julia/packages/SciMLBase/VKnrY/src/integrator_interface.jl:518
┌ Warning: Endpoints do not match. Return code: DtLessThanMin. Likely your time range is not a multiple of `saveat`. sol.t[end]: 7.84356, ts[end]: 12.0
└ @ SciMLSensitivity ~/.julia/packages/SciMLSensitivity/DInxI/src/concrete_solve.jl:1401
ERROR: DimensionMismatch: dimensions must match: a has dims (Base.OneTo(62),), b has dims (Base.OneTo(61),), mismatch at 1
Stacktrace:

```

If there were an error in my model specification, I could not have gotten results I have, so it must be an unstable solution. Given that I am using Tsit5, which is a variable time step (since it is in the RK family), what is happening?

What does `Endpoints do not match` mean? Could an instability make that happen although this mismatch has not happened before? And then there is a dimension mismatch. Again, how is that possible given that the code has been running quite a few iterations before this occurrence.

To be clear, I am referring to the line:

```julia
    parameter_res = Optimization.solve(optprob, Optimisers.AdamW(), callback=cb_fun, sensealg = ReverseDiffVJP(true), allow_f_increases=false, maxiters=100)

```

In any case, perhaps I just have to sleep on it, and experiment with other solvers, and options.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [December 23, 2022, 9:54pm UTC](https://discourse.julialang.org/t/very-long-stack-trace/92025/17 "2022-12-23T21:54:13Z")

</div>

> [@erlebach](#):
>
> `Warning: dt(9.536743e-7) <= dtmin(9.536743e-7) at t=7.84356. Aborting.`

This means that at the given parameters it was unable to solve all of the way. Given the dtmin, this looks to be because it’s using Float32 numbers. Float32 is not going to be as capable of handling numerically difficult cases as Float64, so I’d recommend trying Float64 and seeing that’s fine.

> [@erlebach](#):
>
> Again, how is that possible given that the code has been running quite a few iterations before this occurrence.

Any simulation can go unstable depending on the parameters. `Optimisers.AdamW` is a stochastic optimizer, so you’ll get different parameters each time.

---

<div class="post-metadata">

**Author:** ![erlebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erlebach/32/12973_2.png) [@erlebach](https://discourse.julialang.org/u/erlebach)\
**Post date:** [December 23, 2022, 11:01pm UTC](https://discourse.julialang.org/t/very-long-stack-trace/92025/18 "2022-12-23T23:01:11Z")

</div>

I figured out the problem. I have some function closures that refer to global variables. There is a section below in the code that overwrites these global variables and that created the problem. Thanks!