# DifferentialEquations Package Kills Performance Everywhere when TruncatedStacktraces is used

**URL:** <https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417>\
**Category:** Performance\
**Tags:** package, differentialequation\
**Created:** [March 21, 2023, 7:01pm UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417 "2023-03-21T19:01:34Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![leespen1](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/leespen1/32/221928_2.png) [@leespen1](https://discourse.julialang.org/u/leespen1)\
**Post date:** [March 21, 2023, 7:01pm UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/1 "2023-03-21T19:01:34Z")

</div>

After loading the DifferentialEquations.jl package in the REPL, Julia seems to start performing very slowly, possibly only when text is output to the REPL, but in a way that makes the REPL extremely frustrating to use. Starting from a fresh Julia REPL session, I perform the following operations:

```julia
julia> @time a = [1,2,3,4]
  0.000009 seconds (1 allocation: 96 bytes)
4-element Vector{Int64}:
 1
 2
 3
 4

julia> using DifferentialEquations

julia> @time a = [1,2,3,4]
  0.000003 seconds (1 allocation: 96 bytes)
4-element Vector{Int64}:
 1
 2
 3
 4

julia> 

```

The first `@time a = [1,2,3,4]` is nearly instantaneous, as I would expect. But after `using DifferentialEquations` (which itself takes 30-60 seconds), nearly 10 seconds elapse between hitting enter after `@time a = [1,2,3,4]` and when I am able to type in the REPL again.

This is especially bad when an exception occurs. The stacktrace output can take over a minute to finish.

Why is the inclusion of the DifferentialEquations package so dramatically affecting the performance of seemingly unrelated operations, including those in Base? I have observed this behavior on two computers.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [March 21, 2023, 7:02pm UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/2 "2023-03-21T19:02:51Z")

</div>

This is because DifferentialEquations is overloading type printing to make stacktraces better. It forces any code that might show a type to recompile (and you really aren’t supposed to do things like this because it also breaks things like help mode in the repl for types that have the overloaded show methods. @ChrisRackauckas

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [March 21, 2023, 8:17pm UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/3 "2023-03-21T20:17:18Z")

</div>

A PR to Base fixes this with an experimental feature that we’re already setup to use:

> <https://github.com/JuliaLang/julia/pull/48444>
>
> It is fairly common to define types with parameters whose only use is specializa…tion (of code and memory layout), and printing those parameters everywhere can cause excessive noise. One motivating example is https://github.com/SciML/OrdinaryDiffEq.jl/blob/2eb73bed76c31248113fa8f3f05ff2f84448651a/src/integrators/type.jl#L80. The proposed feature here is to put \`Experimental.@quietparams\` in a struct declaration, and then printing instances and types within stacktraces will omit parameters, e.g.
> \`\`\`
> julia\> struct Foo{A,B,C}
> a::A
> b::B
> c::C
> Base.Experimental.@quietparams
> end
> 
> julia\> map(error, \[Foo((1,2),4,5)\])
> ERROR: Foo((1, 2), 4, 5)
> Stacktrace:
> \[1\] error(s::Foo{...})
> @ Base ./error.jl:44
> \`\`\`
> 
> In the case of \`ODEIntegrator\`, each type can be multiple screens full of text and it is causing quite a problem.

Sadly, it’s not being merged, even as a marked experimental feature. I mentioned this here:

> **[Making stacktraces shorter (reducing the number of frames) · JuliaLang/julia...](https://github.com/JuliaLang/julia/discussions/49044#discussioncomment-5356944)**
>
> Particularly with #40537 and the arrival of AbbreviatedStackTraces, there's been growing interest/pressure in getting Julia to simplify its error-reporting. From my own experience teaching, I'd agr...

I don’t understand why we cannot have this as an experimental feature in v1.9, even if it’s removed later we have everything safely defined:

> <https://github.com/SciML/OrdinaryDiffEq.jl/blob/f0195185209bb51e79316968ef5fc171ed389a78/src/integrators/type.jl#L179-L181>

So if this was merged for v1.9.0 and removed shortly after then we would be okay. But without getting some buy-in with Base, there’s literally no tool to solve what we need to solve.

That said, the TruncatedStacktraces.jl README is very clear that you can disable all of the stacktrace truncation with just a line of code:

> **[GitHub - SciML/TruncatedStacktraces.jl: Simpler stacktraces for the Julia...](https://github.com/SciML/TruncatedStacktraces.jl#disabling-truncatedstacktracesjl)**
>
> Simpler stacktraces for the Julia Programming Language - GitHub - SciML/TruncatedStacktraces.jl: Simpler stacktraces for the Julia Programming Language

```julia
using Preferences, UUIDs

using TruncatedStacktraces
Preferences.set_preferences!(TruncatedStacktraces, "disable" => true)

# OR if you don't want to load TruncatedStacktraces.jl

Preferences.set_preferences!(UUID("781d530d-4396-4725-bb49-402e4bee1e77"), "disable" => true)

```

So take your pick: either stack traces are super long and no longer solved, or you get invalidations. The answer of getting short stack traces and no invalidations just requires a custom Julia build (using the branch I linked) or begging someone to just merge something into Base Julia.

---

<div class="post-metadata">

**Author:** ![leespen1](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/leespen1/32/221928_2.png) [@leespen1](https://discourse.julialang.org/u/leespen1)\
**Post date:** [March 21, 2023, 8:19pm UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/4 "2023-03-21T20:19:05Z")

</div>

Thanks! Is this new behavior? I’m surprised such a popular package has such an impactful issue. If possible I would like to rollback to a previous version where this behavior does not occur.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [March 21, 2023, 8:19pm UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/5 "2023-03-21T20:19:51Z")

</div>

> [@leespen1](#):
>
> I would like to rollback to a previous version where this behavior does not occur.

You don’t have to. Literally just set the preference.

```julia-auto
using Preferences, UUIDs
Preferences.set_preferences!(UUID("781d530d-4396-4725-bb49-402e4bee1e77"), "disable" => true)

```

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [March 21, 2023, 8:24pm UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/6 "2023-03-21T20:24:30Z")

</div>

That won’t fix it. The problem isn’t the abreviated stack trace, but the redefinition of type showing. For example [SciMLBase.jl/callbacks.jl at 587cfcc9cf4277b35d3a0a2b08a21d5c9467bf8f · SciML/SciMLBase.jl · GitHub](https://github.com/SciML/SciMLBase.jl/blob/587cfcc9cf4277b35d3a0a2b08a21d5c9467bf8f/src/callbacks.jl#LL313C14-L313C15) means that anyone who loads SciMLBase will have this issue.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [March 21, 2023, 8:29pm UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/7 "2023-03-21T20:29:01Z")

</div>

That makes no sense. The redefinition of type showing only exists because the stack trace silencing hasn’t merged in Base like we were expected it to. So yes, if it merges then we just change `TruncatedStacktraces.@truncate_stacktrace VectorContinuousCallback` to match the Base stack trace silencing and it all goes away. We’re waiting and hoping this merges for v1.9 or something soon.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [March 21, 2023, 8:30pm UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/8 "2023-03-21T20:30:47Z")

</div>

The problem is that by overloading type showing, you invalidate any code that shows a type which includes package loading, a bunch of the repl, and docstrings.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [March 21, 2023, 8:32pm UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/9 "2023-03-21T20:32:05Z")

</div>

Yes, and so then it all goes away when you set the preference because they are statically not defined.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [March 21, 2023, 8:44pm UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/10 "2023-03-21T20:44:37Z")

</div>

The method I’m complaining about is in SciMLBase. It gets defined even if TruncatedStackTraces.jl isn’t loaded.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [March 21, 2023, 8:46pm UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/11 "2023-03-21T20:46:38Z")

</div>

That was an oversight fixed in [Missed a show -\> truncatedstacktraces macro by ChrisRackauckas · Pull Request #423 · SciML/SciMLBase.jl · GitHub](https://github.com/SciML/SciMLBase.jl/pull/423)

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [March 22, 2023, 4:20pm UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/12 "2023-03-22T16:20:34Z")

</div>

> [@leespen1](#):
>
> Thanks! Is this new behavior? I’m surprised such a popular package has such an impactful issue. If possible I would like to rollback to a previous version where this behavior does not occur.

Totally! Why isn’t this disruptive behavior opt-in?

> [@ChrisRackauckas](#):
>
> Literally just set the preference.

It should be the other way around: default behavior is “don’t change anything”, with the option to “enable” stacktraces rewriting.

And the “disable” setting doesn’t seem to work for me: even after `Preferences.set_preferences!(UUID("781d530d-4396-4725-bb49-402e4bee1e77"), "disable" => true)` stracktraces mention that package:

```julia-auto
Some of the types have been truncated in the stacktrace for improved reading. To emit complete information
in the stack trace, evaluate `TruncatedStacktraces.VERBOSE[] = true` and re-run the code.

```

This message isn’t even correct: nothing is actually truncated.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [March 23, 2023, 1:56am UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/13 "2023-03-23T01:56:41Z")

</div>

Dare I suggest another solution to the SciML stacktrace problem? Don’t define types where 10 out the 13 fields are parameterized, like [this one](https://github.com/SciML/SciMLBase.jl/blob/0faa9d51cbca5a28581a20539df77632d6d12dc3/src/solutions/ode_solutions.jl#L29):

```julia
struct ODESolution{T, N, uType, uType2, DType, tType, rateType, P, A, IType, S,
                   AC <: Union{Nothing, Vector{Int}}} <:
       AbstractODESolution{T, N, uType}
    u::uType
    u_analytic::uType2
    errors::DType
    t::tType
    k::rateType
    prob::P
    alg::A
    interp::IType
    dense::Bool
    tslocation::Int
    stats::S
    alg_choice::AC
    retcode::ReturnCode.T
end

```

In my opinion, this is the real reason why SciML stacktraces are hard to read.

---

<div class="post-metadata">

**Author:** ![DanielVandH](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielvandh/32/31134_2.png) [@DanielVandH](https://discourse.julialang.org/u/DanielVandH)\
**Post date:** [March 23, 2023, 2:03am UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/14 "2023-03-23T02:03:15Z")

</div>

What is the alternative here? The types likely do need to be quite generic since there are many options to choose from, but just leaving the types as `::Any` wouldn’t be so performant which is what the point of much of these types is for.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [March 23, 2023, 2:18am UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/15 "2023-03-23T02:18:36Z")

</div>

I’m no expert on SciML internals, but the alternative to parameterizing every field is to choose the types that you want to use for your internal code and work a little harder in your constructors to convert user input into the concrete data types that your code will actually use. This is made even easier by the implicit conversions done by default constructors.

Another alternative is to leave the fields as `Any` (or an appropriate abstract type) and use function barriers. I would guess that most of the performance critical parts of SciML are behind function barriers anyways.

---

<div class="post-metadata">

**Author:** ![SteffenPL](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/steffenpl/32/206270_2.png) [@SteffenPL](https://discourse.julialang.org/u/SteffenPL)\
**Post date:** [March 23, 2023, 2:48am UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/16 "2023-03-23T02:48:37Z")

</div>

I am quite certain that less parameterization is not a feasible option.

To give some examples, even if only `u` would be parameterized, it can be a pretty long expression:  
Types which could appear here for example `Unitful` quantities, `Duals` for automatic differentiation, some kind of `ComponentArray` for better access to the solution, `CuArrays` for GPU computation, `Intervals` for uncertainty measurements or `Nums` for symbolic nightmares ;). This is just a list of types I have used already, I’m sure there are many more. Note that many of these cases are actually really different types with different arithmetic, so, you cannot just convert it to an `Array{Float, N}`.

The problem why `Any` as a `uType` is not good, is that you don’t want a runtime dispatch in each time step. Moreover, you might want to forward `ODESolution` (or other SciML internal types like integrator) to many functions, so, it is not only one function barrier but many more. The alternative would be to then forward only `u` with its concrete type, but then again stack traces get long again.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [March 23, 2023, 2:57am UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/17 "2023-03-23T02:57:32Z")

</div>

Ok, that explains one of the parameters. What about the other 9? I could certainly be wrong, but my intuition says there’s a way to redesign the code base that doesn’t rely so heavily on type parameters. That wasn’t even the most egregious example. Consider these:

```julia
mutable struct DEOptions{absType, relType, QT, tType, Controller, F1, F2, F3, F4, F5, F6,
                         F7, tstopsType, discType, ECType, SType, MI, tcache, savecache,
                         disccache}
# ...
end

```

```julia
mutable struct ODEIntegrator{algType <: Union{OrdinaryDiffEqAlgorithm, DAEAlgorithm}, IIP,
                             uType, duType, tType, pType, eigenType, EEstT, QT, tdirType,
                             ksEltype, SolType, F, CacheType, O, FSALType, EventErrorType,
                             CallbackCacheType, IA} <:
               DiffEqBase.AbstractODEIntegrator{algType, IIP, uType, tType}

```

I can’t even count how many type parameters that is.

Furthermore, many of those type parameters are set to `Nothing`, which you can see in SciML stacktraces. Why have 20 parameters, 15 of which are `Nothing`?

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [March 23, 2023, 3:00am UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/18 "2023-03-23T03:00:31Z")

</div>

`absType` and `relType` are the type of the absolute and relative errors respectively. These can be either Floats if you want an L2 norm or arrays if you want component-wise norms. `prob` is the problem type which can be a wide variety of things. `alg` is the type of the algorithm that was used (there are about 200 options here). `stats` could possibly be made to be a concrete type, etc.

---

<div class="post-metadata">

**Author:** ![SteffenPL](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/steffenpl/32/206270_2.png) [@SteffenPL](https://discourse.julialang.org/u/SteffenPL)\
**Post date:** [March 23, 2023, 3:02am UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/19 "2023-03-23T03:02:51Z")

</div>

One good example is maybe [Parallel Ensemble Simulations · DifferentialEquations.jl](https://docs.sciml.ai/DiffEqDocs/stable/features/ensemble/) where you essentially run the whole thing millions of times… Even is a single solve is fast, you cannot afford much runtime dispatch in such a setup.

But just to be sure, I get the point that the design could be maybe be less greedy. But that’s not really in the nature of that project.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [March 23, 2023, 3:15am UTC](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417/20 "2023-03-23T03:15:00Z")

</div>

Look at this chunk from a SciML stacktrace:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/5/5/554b83ac454ed6978c787dd3507aca8ccd3be2c8.png)

Notice that ~90% of the parameters are `Nothing`. If 90% of your fields are `nothing`, then that indicates that you could develop types that better capture your domain logic and are not 90% empty.

Again, I’m not a SciML developer, but to me this all points to an opportunity to rethink the design of SciML types. But take it with a grain of salt. I’ll get off my soap box now.

[Next page](https://discourse.julialang.org/t/differentialequations-package-kills-performance-everywhere-when-truncatedstacktraces-is-used/96417.md?page=2)