How to Approach Parameter Estimation with Non-Linear Optimizers

Specific Domains Optimization (Mathematical)
1

Hello,

I am trying to estimate the parameters of an ODE given discrete data x(t) that is the solution to that ODE. The ODE is: m\ddot{x}=f(x,\theta) where \theta is some vector of parameters. I can estimate the parameters in Python using SciPy; however, the choice of algorithms was rather limited. I am using a Newton’s Method with trust regions at the moment. As it stands my code does not give good results. I think this is because my data, x(t), does not capture a full period of oscillation. I am hoping Julia has some more things for me to try. If anyone has recommendations for this problem I’d love to hear them!

I am looking for an unconstrained non-linear solver to find the minimum of the loss function. I see plenty of options (Optim.jl, NLopt.jl ParamEstimDiffEq.jl etc.) but its a little unclear to me what the right choice is or if all of these packages are up-to-date. In the Optim.jl package I can get a few test cases working; however, based on their documentation I should only be using the “zeroth” order methods as taking the derivative/hessian of my loss function is not strictly feasible. I know Julia and (Optim.jl) has automatic differentiation baked in but this idea is new to me. Will automatic differentiation be able to handle a loss function that contains the solution to an ODE in it? My loss function currently, is the MSE between the data and the solution to the above ODE for the current parameter set.

Is GalaticOptim/Optimization.jl or DiffEqFlux the preferred packages for this now? Its hard to tell what is the most recent and most maintained package currently.

2

Why? If it’s an ODE, that’s all covered by SciMLSensitivity.jl. I’m currently improving the docs a bit, but this example is probably a good one to take a look at:

https://docs.sciml.ai/SciMLSensitivity/v7.2/ode_fitting/optimization_ode/

Anything documented in the SciML ecosystem is maintained by the SciML contributors. Thus use the SciML docs as a source of truth for the choices:

https://docs.sciml.ai/Overview/stable/

Extra detail can be found in Parameter Estimation, Bayesian Analysis, and Inverse Problems · Overview of Julia's SciML .

The tl;dr for you is:

  • DiffEqParamEstim.jl is a utility that makes it easy to write cost functions for Optimization.jl
  • SciMLSensitivity.jl is a utility that makes it easy to calculate gradients of cost functions which have differential equation (and other equation) solvers in them
  • Optimization.jl is the optimization library to use for these tasks, and the NLopt.jl wrappers from Optimization.jl tend to be the most robust.

And BTW, in about a week improved docs on some of this stuff should pop up, we’ve been overhauling our docs over the last few months and simplifying this part of parameter estimation has been one of the big pushes.

2 Likes
3

Thanks! I’ll try those out.

4

(if you wait a few days for the new DiffEqParamEstim.jl tutorial to go up, which should be the easiest way to do this, though SciMLSensitivity’s tutorials are the same thing but with manually defining the simplest l2-loss functions)

1 Like