I am a very frequent user of the Nelder-Mead optimisation routine (of the excellent Optim.jl package) which I find very effective for problems with a handful of free parameters to tune. I have very little knowledge of how the algorithm works, but it seems to also do well also with problems that may be discontinuous or slightly noisy too.
In some cases, I have noticed that the algorithm gets “stuck” and I have to wait for the maximum number of iterations to be reached before I get my solution (the solution can be good or bad independently of Nelder-Mead getting stuck). Now, by “stuck” I mean that the function value does not seem to be improving (at least judging by the output trace in the terminal) and the value for
√(Σ(yᵢ-ȳ)²)/n also seems to be stuck at a certain value.
In such a case, when Nelder-Mead gets “stuck”, it would be nice if it would stop the optimisation and return the result. I have tried playing around with
f_tol but Nelder-Mead still displays the same behaviour of getting “stuck”.
In order to be absolutely clear, here is an example of the output trace when getting “stuck”:
Iter Function value √(Σ(yᵢ-ȳ)²)/n ------ -------------- -------------- ... 4200 -5.535884e+01 2.449928e-02 4300 -5.535884e+01 2.449928e-02 4400 -5.535884e+01 2.449928e-02 ...
Does somebody have some advice perhaps on how I could tell Nelder-Mead to promptly stop the optimisation when it enters this behaviour?
Thanks very much for taking the time to read this.