Catalyst and BifurcationKit

General Usage
1

Hi,

Is there any reason why BifurcationKit is blocked at 0.4.4 in Catalyst?

2

I think the reason is that Catlayst currently have capped MTK t0 <v9.60. The reason for that is that MTK made some changes in that release which broke a bounce of Catalyst features, and we don’t want to mark Catalyst as compatible with with higher versions until we have sorted that out.

This PR: https://github.com/SciML/Catalyst.jl/pull/1186 (hopefully merged in the next few days, although we might need to wait for another MTK fix as something more just popped up) will make Catalyst master compatible with the latest MTK version. After that, we will hopefully will make a release vaguely soon (but hard to say, we have hoped that for a while now, and things are sadly a bit outside of our control).

2 Likes
3

The last Catalyst release only supported BifurkationKit .3. Master currently caps MTK as mentioned. We are trying to fix that and get a new release out, but things in Catalyst keep breaking due to changes in dependencies faster than we can get them fixed.

1 Like
4

Would you happen to which version specifically is compatible? BifurcationKit@0.3.6 and 0.3.0 both downgrade Catalyst and MTK. The Catalyst Project.toml seems to suggest 0.4.4 but that doesn’t work either.

5

The last Catalyst release only declares compatibility with BifurcationKit 0.3X releases. You should probably only use that Catalyst release with ModelingToolkit 9.59 and earlier, as ModelingToolkit 9.60 and up have made changes that break lots of Catalyst stuff. (So I’d suggest downgrading ModelingToolkit to 9.59.)

On master, Catalyst declares support for BifurcationKit v0.4, but we currently cap ModelingToolkit due to all the issues we are having with Catalyst features that are broken when using ModelingToolkit 9.60 and up. As soon as we can get Catalyst working with newer ModelingToolkit releases we will release Catalyst 15, but so far we have been unable to implement fixes at a pace that exceeds the pace at which things have become broken from new ModelingToolkit releases.

Unfortunately, with the way compat is typically managed in Julia packages, if a package we depend on makes changes that break Catalyst, but which that package does not declare are breaking, then it will be installed by default when you install the latest Catalyst release. Going forward we plan to cap our compatibility with certain packages to try to minimize such issues, but right now we are stuck with this problem for Catalyst V14.

6

Thanks, yes I did hear about that. Looking forward to the next release.

Right now I haven’t installed MTK explicitly so I just have the version the package manager installed along with the latest Catalyst release (14.4.1). It installed MTK 9.47.0. If I look at the Project.toml for MTK@9.47.0, it says it depends on BifurcationKit@0.3. However, neither BifurcationKit@0.3.0 nor BifurcationKit@0.3.6 install correctly.

Is there any way to check the Manifest.toml of the environment used for this example?: Bifurcation Diagrams · Catalyst.jl

7

Check out the bottom of this page:

8 
Status `~/work/Catalyst.jl/Catalyst.jl/docs/Project.toml`
  [6e4b80f9] BenchmarkTools v1.5.0
⌅ [0f109fa4] BifurcationKit v0.3.6
  [13f3f980] CairoMakie v0.12.9
  [479239e8] Catalyst v14.4.1 `~/work/Catalyst.jl/Catalyst.jl`
  [a93c6f00] DataFrames v1.6.1
  [1130ab10] DiffEqParamEstim v2.2.0
  [31c24e10] Distributions v0.25.111
  [e30172f5] Documenter v1.7.0
  [61744808] DynamicalSystems v3.3.20
  [af5da776] GlobalSensitivity v2.7.0
  [1ecd5474] GraphMakie v0.5.12
  [86223c79] Graphs v1.11.2
  [f213a82b] HomotopyContinuation v2.11.0
  [40713840] IncompleteLU v0.2.1
  [ccbc3e58] JumpProcesses v9.13.7
  [23fbe1c1] Latexify v0.16.5
  [7ed4a6bd] LinearSolve v2.34.0
  [961ee093] ModelingToolkit v9.38.0
  [8913a72c] NonlinearSolve v3.14.0
  [429524aa] Optim v1.9.4
  [7f7a1694] Optimization v3.28.0
  [3e6eede4] OptimizationBBO v0.3.0
  [4e6fcdb7] OptimizationNLopt v0.2.2
  [36348300] OptimizationOptimJL v0.3.2
  [42dfb2eb] OptimizationOptimisers v0.2.1
  [1dea7af3] OrdinaryDiffEq v6.89.0
  [91a5bcdd] Plots v1.40.8
  [8a4e6c94] QuasiMonteCarlo v0.3.3
  [0bca4576] SciMLBase v2.52.2
  [1ed8b502] SciMLSensitivity v7.66.2
  [276daf66] SpecialFunctions v2.4.0
  [90137ffa] StaticArrays v1.9.7
  [9672c7b4] SteadyStateDiffEq v2.4.0
  [789caeaf] StochasticDiffEq v6.69.0
  [220ca800] StructuralIdentifiability v0.5.9
  [0c5d862f] Symbolics v6.11.0
  [56ddb016] Logging
9

Just reiterating what Sam said about the update. We have put in a lot of work to have the new Catalyst release ready as soon as possible now. If we are lucky it will happen quite soon, however, there are still 1 or 2 issues in our dependant packages that needs to be resolved before we are comfortable with a new release. Unfortunatley this is mostly outside of our control, but I hope it should be resolved soon.

10

Thank you both for your work on this! I’ll try using that environment and see if it works.

1 Like
11

The key is that previous versions of Catalyst made some assumptions that are incorrect w.r.t. model transformations. This kind of worked because Catalyst generally made explicit ODE models, whereas the older ModelingToolkit and SciML interfaces generally opted those models out of doing anything special before. More specifically, there was a growing list of requirements that needed to be hit in order to be in the “simple” case, which Catalyst just happened to hit for about 90% of its models (10% though were already starting to show the cracks, where each time this list was updated an issue was hit). Specifically that list was:

With v9.60 the final non-simplifying special cases were finally made not weirdly non-special because of improvements to the initialization system. Specifically, in order to ensure parameter dependencies could be simplified and ensured, even the simplest of explicit ODEs required hitting the standard path, and so it moved.

What happened then is that all of the invalid assumptions inside of Catalyst were finally tested. Things like “don’t assume you have the same number of equations prior to simplifying the system”, “don’t assume the ordering of the variables” etc., suddenly mattered to Catalyst. To MTK users, this is of course nothing special as these have been written in big bold warnings in the docs and these error messages have existed for every DAE generated since 2021, but the key thing here was that it now actually mattered for Catalyst, and Catalyst had some old ModelingToolkit v5 code that directly modified structs and directly indexed values instead of using the documented interfaces. And again, even though those interfaces were deprecated almost a half decade ago, that was fine as long as the ODEs happened to hit the explicit path. But now they didn’t, so the Catalyst technical debt reared its head and it needs a bit of an update to make sure it’s not making incorrect assumptions about final systems.

So yeah, it’s a bit of an odd one because 99% of users will have noticed nothing change, because any DAE case is exactly the same and anyone using the explicit ODE interfaces with the documented interfaces would be fine. However, this specialized codepath was pretty killer because not only was it just a completely different compiler branch, but that compiler branch didn’t include any warnings or error messages for going out of the standard interfaces, since those warnings and error messages were all created and added for the 99% path. So… it was a tough bandaid to rip off because the moment it gets moved over it just gets smacked with a message of everything that changed since 2021. :person_shrugging:

Going forward, a few things to then note from this:

  1. There is now no specialized codepath. Everything uses the standard path now, everything has an initialization system, everything is the same. The slow growth of that system is thus completed.
  2. Everything can now have symbolic optimizations. Before we used to always say it’s possible, but in the edge code of only having explicit ODEs there were no specializations. But now there are some (for parameter inits) and growing. For example, if D(x)~0 then move x to be a parameter since it’s constant, or if you have a time dependent rhs replace it with a callback that computes the integral, or if you have a linear ODE for part of your system eliminate it and replace it with the matrix exponential analytical solution. These are all things on the table that would apply to these explicit ODEs, so yes do not assume all of your ODEs are naively being solved. This means making sure the symbolic interfaces are respected and used.
  3. We will make a MTK v10 hopefully soon, which should add more errors to the non symbolic interfaces that make too many indexing assumptions so we can make sure other programs are robust and not using internal behaviors.
1 Like
12

Thanks for this detailed explanation! Really interesting to see the behind the scenes of the development of these packages.