What package allowed setting the learning rate for a minimization problem?
Thank you
P.S. it looks like Opitm.jl isn’t allowed.
Many of the optimizers available through Flux allow for setting the learning rate, such as the ones here. Do these suit your needs?
Can I use Flux to just find a minimum of any function?
I thought that Flux was some kind of DeepLearning models only
I believe that you can use it with arbitrary code. See this example, which does linear regression and sets a learning rate for gradient descent.
For arbitrary non-stochastic optimization (unlike deep learning), people normally use gradient-based algorithms that do not use a fixed learning rate, but which set step sizes adaptively. There are lots of these algorithms for nonlinear optimization available in Julia packages like Optim, Nonconvex, NLopt, JuMP, …
Fixed-rate algorithms like Adam can be implemented in only a few lines of code. There are various Julia implementations lying around but they mostly seem to be embedded in other packages.
1 Like
Adam does not fix the learned rate; it is adaptive.
I didn’t find a way how the Optim can be set the learning rate.
P.S. Yes, it’s quite easy to write gradient descend code.
Sort of. It’s true that it scales the gradient inversely with a variance of the gradient. However, it then unconditionally takes a step with a fixed formula based on a stepsize parameter α and the moment estimates — it’s quite different from algorithms that do adaptive backtracking or trust regions based on e.g. sufficient-decrease conditions.
Yes, for simple algorithms like Adam, gradient-descent with a fixed learning rate, accelerated gradient descent, etcetera. More sophisticated algorithms like L-BFGS are not so easy to implement.
3 Likes