# What package allowed setting the learning rate for a minimization problem?

**URL:** <https://discourse.julialang.org/t/what-package-allowed-setting-the-learning-rate-for-a-minimization-problem/97523>\
**Category:** General Usage\
**Tags:** optimization\
**Created:** [April 15, 2023, 10:48pm UTC](https://discourse.julialang.org/t/what-package-allowed-setting-the-learning-rate-for-a-minimization-problem/97523 "2023-04-15T22:48:33Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![BMval](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bmval/32/7647_2.png) [@BMval](https://discourse.julialang.org/u/BMval)\
**Post date:** [April 15, 2023, 10:48pm UTC](https://discourse.julialang.org/t/what-package-allowed-setting-the-learning-rate-for-a-minimization-problem/97523/1 "2023-04-15T22:48:33Z")

</div>

What package allowed setting the learning rate for a minimization problem?

Thank you

P.S. it looks like Opitm.jl isn’t allowed.

---

<div class="post-metadata">

**Author:** ![liamfdoherty](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/liamfdoherty/32/24516_2.png) [@liamfdoherty](https://discourse.julialang.org/u/liamfdoherty)\
**Post date:** [April 15, 2023, 11:26pm UTC](https://discourse.julialang.org/t/what-package-allowed-setting-the-learning-rate-for-a-minimization-problem/97523/2 "2023-04-15T23:26:03Z")

</div>

Many of the optimizers available through Flux allow for setting the learning rate, such as the ones [here](http://fluxml.ai/Optimisers.jl/dev/api/#Optimisation-Rules). Do these suit your needs?

---

<div class="post-metadata">

**Author:** ![BMval](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bmval/32/7647_2.png) [@BMval](https://discourse.julialang.org/u/BMval)\
**Post date:** [April 15, 2023, 11:29pm UTC](https://discourse.julialang.org/t/what-package-allowed-setting-the-learning-rate-for-a-minimization-problem/97523/3 "2023-04-15T23:29:15Z")

</div>

Can I use Flux to just find a minimum of any function?  
I thought that Flux was some kind of DeepLearning models only

---

<div class="post-metadata">

**Author:** ![liamfdoherty](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/liamfdoherty/32/24516_2.png) [@liamfdoherty](https://discourse.julialang.org/u/liamfdoherty)\
**Post date:** [April 15, 2023, 11:33pm UTC](https://discourse.julialang.org/t/what-package-allowed-setting-the-learning-rate-for-a-minimization-problem/97523/4 "2023-04-15T23:33:51Z")

</div>

I believe that you can use it with arbitrary code. See [this example](http://fluxml.ai/Optimisers.jl/dev/api/#Optimisation-Rules), which does linear regression and sets a learning rate for gradient descent.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [April 16, 2023, 12:30am UTC](https://discourse.julialang.org/t/what-package-allowed-setting-the-learning-rate-for-a-minimization-problem/97523/5 "2023-04-16T00:30:29Z")

</div>

> [@BMval](#):
>
> Can I use Flux to just find a minimum of any function?

For arbitrary non-stochastic optimization (unlike deep learning), people normally use gradient-based algorithms that do not use a fixed learning rate, but which set step sizes adaptively. There are lots of these algorithms for nonlinear optimization available in Julia packages like Optim, Nonconvex, NLopt, JuMP, …

Fixed-rate algorithms like Adam can be implemented in only a few lines of code. There are various Julia implementations lying around but they mostly seem to be embedded in other packages.

---

<div class="post-metadata">

**Author:** ![BMval](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bmval/32/7647_2.png) [@BMval](https://discourse.julialang.org/u/BMval)\
**Post date:** [April 16, 2023, 12:36am UTC](https://discourse.julialang.org/t/what-package-allowed-setting-the-learning-rate-for-a-minimization-problem/97523/6 "2023-04-16T00:36:21Z")

</div>

Adam does not fix the learned rate; it is adaptive.

I didn’t find a way how the Optim can be set the learning rate.

P.S. Yes, it’s quite easy to write gradient descend code.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [April 16, 2023, 12:50am UTC](https://discourse.julialang.org/t/what-package-allowed-setting-the-learning-rate-for-a-minimization-problem/97523/7 "2023-04-16T00:50:05Z")

</div>

> [@BMval](#):
>
> Adam does not fix the learned rate; it is adaptive.

Sort of. It’s true that it scales the gradient inversely with a variance of the gradient. However, it then unconditionally takes a step with a fixed formula based on a stepsize parameter α and the moment estimates — it’s quite different from algorithms that do adaptive backtracking or trust regions based on e.g. sufficient-decrease conditions.

> [@BMval](#):
>
> Yes, it’s quite easy to write gradient descent code.

Yes, for simple algorithms like Adam, gradient-descent with a fixed learning rate, accelerated gradient descent, etcetera. More sophisticated algorithms like L-BFGS are not so easy to implement.