NeuralPDE fails to solve system when parameters change

I am hitting a weird situation that I can not figure out why – perhaps it’s a bug within NeuralPDE or maybe just need to tweak my settings.

Given the Lorenzs system in this example (i.e., eqs) I am able to do forward simulation using this code with the parameter values specified

strategy = NeuralPDE.QuadratureTraining(; abstol = 1e-6, reltol = 1e-6, batch = 200)
discretization = NeuralPDE.PhysicsInformedNN([chain1, chain2, chain3], strategy, param_estim=false)
@named pdesystem = PDESystem(eqs, bcs, domains, [t], [x(t), y(t), z(t)],  
       [σ_, ρ, β], defaults = Dict([σ_ => 10.0, ρ => 9.0, β => 8/3]))
prob = discretize(pdesystem, discretization)
opt = BFGS(linesearch = BackTracking())
res = Optimization.solve(prob, opt; maxiters = 2000)

The neural network solution matches the numerical solution (see below) using traditional solvers in OrdinaryDiffEq. (Side note: the retcode from solve is actually a ‘Failure’)

Now, if I change the parameter to ρ => 10.0, it seems to fail all of a sudden? i.e., running

@named pdesystem = PDESystem(eqs, bcs, domains, [t], [x(t), y(t), z(t)],  
       [σ_, ρ, β], defaults = Dict([σ_ => 10.0, ρ => 10.0, β => 8/3]))

produces the wrong solution (but what’s weird is that now the retcode is a 'Success`).

so the question is whydoes changing this parameter mess up the results? For values of ρ between 1 to 9, it works fine but as soon as you change to ρ >= 10, it messes up.

Even more weirdness is that I can use the PINN solver to estimate these parameters (like in the example linked above). So if I start with all parameters Dict([σ_ => 1.0, ρ => 1.0, β => 1.0])) and set param_estim = true, it can recover the parameters!!!

(Aside: @ChrisRackauckas I know you said not to use the PINN/PDEsystem API for ODE systems but I was just testing and learning the API to get a deeper understanding!)

Happy to provide full reproducible code

Okay it turns out this was issue with the training and optimization. I switched over to GridTraining(0.001) and bumped the max iterations in the optimization to 10,000 which seemed to have worked.

The problem is then if I bump ρ => 28.0 it fails again. I decreased dt again to 0.0001 and increased maxiters = 50000 but this still does not make the NN converge to the solution.

Is this just the behaviour of the Lorenzs system? Maybe I need a deeper neural network.