There are plenty of methods to solve a minimization problem. Many of them iteratively solves for the parameters by decrementing them by a fraction (determined by a learning rate) of the partial derivatives of the objective function. Why not just solve an ODE instead of inventing optimizers? What can…

[image] tomaklutfu: However, there is also nonlinear problems that you traditionally follow Newton flow locally but we usually do not see further and do not take large steps. It may help these cases. […] We still can incorporate ideas from the sate-of-the-art existing optimizers here for what th…

Well absolutely. You start with your ODE solver which you tell to follow gradient descent. Then you change the descent direction to a more appropriate one. Maybe you use the Hessian to follow a suitably regularized Newton flow, maybe you incorporate some curvature information with a quasi-Newton alg…

[image] antoine-levitt: supposedly better than standard algorithms at certain type of optimization problems I suppose I am somewhat cynical, but arxiv is full of papers describing optimization algorithms that happen to be beat existing methods for a few specific test problems. I have implemen…

in addition to the good points mentioned already, following the surface in high dimensions is going to lead you to a local optimum. even things like LBFGS from the Optim package was blown out of the water by BlackBoxOptim using differential evolution strategies on a problem I was working on yesterda…

Dismissing the idea upfront as if repurposing is a bad idea and then saying sufficiently different ideas will get you somewhere is not constructive. I even got a book advice :clap: :clap:. What a great day to start a discussion. I was not expecting such cynicism here. With this type of cynicism, Jul…

Well I mean you asked for feedback, I’ve given you my feedback respectfully, as have other people here. I won’t claim to be an optimization or diff eq expert, but I did implement a few of these methods, taught them and even have a few papers in related areas. Unbridled optimism is good for blue sky …

[image] tomaklutfu: However, I expected some improvements on the idea at least. In addition to what @antoine-levitt said: if you want people to buy into a proposal to the extent that they invest time on improving it, you may want to consider rigging up at least demonstration. You can find a c…

I should say that turning a problem into a differential equation, and then looking at the property of the paths that the differential equation takes to discover facts about a mathematical object is a super effective strategy for sampling from many Bayesian posterior distributions. This is known as H…

[image] antoine-levitt: You start with your ODE solver which you tell to follow gradient descent. Then you change the descent direction to a more appropriate one. Maybe you use the Hessian to follow a suitably regularized Newton flow, maybe you incorporate some curvature information with a quasi…

There are papers showing that standard adaptive schemes will not reproduce correct long-time dynamics and in particular not converge to an equilibrium. I can’t remember the names :(. If somebody happens to know them I’d be grateful for the references.

An Idea for Optimization as if Solving ODE

Specific Domains Optimization (Mathematical)

tomaklutfu June 28, 2020, 9:23pm 19

Thanks a ton. I’ll look into them.

Topic		Replies	Views
Parameter estimation of an ODE in Julia is slower than in R Performance optim , r , differentialequation , parameters	25	1881	August 7, 2022
Neural ODE fitting really slow Machine Learning diffeq , flux , sciml	13	2112	September 11, 2020
Optim and differential equations? Specific Domains	45	3689	July 5, 2019
Differences between NLsolve and Optim in solving system of equations Optimization (Mathematical)	24	3178	November 16, 2022
Parameter Estimate Problem Optimization (Mathematical) diffeq	6	1186	February 14, 2019

An Idea for Optimization as if Solving ODE

Related topics