# Function in Flux to estimate learning rate

**URL:** <https://discourse.julialang.org/t/function-in-flux-to-estimate-learning-rate/21734>\
**Category:** Machine Learning\
**Created:** [March 11, 2019, 12:33pm UTC](https://discourse.julialang.org/t/function-in-flux-to-estimate-learning-rate/21734 "2019-03-11T12:33:21Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![samq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samq/32/47935_2.png) [@samq](https://discourse.julialang.org/u/samq)\
**Post date:** [March 11, 2019, 12:33pm UTC](https://discourse.julialang.org/t/function-in-flux-to-estimate-learning-rate/21734/1 "2019-03-11T12:33:21Z")

</div>

Is there a function in Flux to estimate the best learning rate for a good gradient descent before training a neural network?

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [March 11, 2019, 7:09pm UTC](https://discourse.julialang.org/t/function-in-flux-to-estimate-learning-rate/21734/2 "2019-03-11T19:09:01Z")

</div>

No.  
Or yes, if you are willing to accept the defaults of the optimizers.  
(Which TBH I normally am)  
In general this is not possible, sorting this out is the job of hyperparameter optimization.

---

<div class="post-metadata">

**Author:** ![samq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samq/32/47935_2.png) [@samq](https://discourse.julialang.org/u/samq)\
**Post date:** [March 11, 2019, 10:09pm UTC](https://discourse.julialang.org/t/function-in-flux-to-estimate-learning-rate/21734/3 "2019-03-11T22:09:19Z")

</div>

@oxinabox It’s inaccurate to state “_in general this is not possible_”; of course it is!

There is a fast.ai package ([Understanding Learning Rates and How It Improves Performance in Deep Learning | by Hafidz Zulkifli | Towards Data Science](https://towardsdatascience.com/understanding-learning-rates-and-how-it-improves-performance-in-deep-learning-d0d4059c1c10)), developed by Jeremy Howard that does exactly this, so I believe it would be in the best interest of Flux users to have such a functionality in the Flux library.

I hoping Flux developers are listening!

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [March 11, 2019, 10:26pm UTC](https://discourse.julialang.org/t/function-in-flux-to-estimate-learning-rate/21734/4 "2019-03-11T22:26:39Z")

</div>

That is not what a I would call estimating the learning rate **before** training the network.  
That is changing the learning rate **during** training.  
i.e., learning rate schedualling (and smarter varients there of, maybe)

Which is a different thing.  
I assumed you were asking about determining the optimal (initial) learning rate.

Anyway, I am pretty sure flux doesn’t have that yet,  
but you can implement it by hand without too much trouble.  
Like in this example

> <https://github.com/FluxML/model-zoo/blob/4ca0c89604ef841b1513174d161196ca5ddce95d/vision/mnist/conv.jl#L114-L120>

Anyway, I agree Flux should have convienence helpers for this.  
Particularly for more complicated varients.

---

<div class="post-metadata">

**Author:** ![samq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samq/32/47935_2.png) [@samq](https://discourse.julialang.org/u/samq)\
**Post date:** [March 11, 2019, 10:37pm UTC](https://discourse.julialang.org/t/function-in-flux-to-estimate-learning-rate/21734/5 "2019-03-11T22:37:54Z")

</div>

@ oxinabox I humbly stand corrected!  
Thank you for the example!

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [March 12, 2019, 4:46am UTC](https://discourse.julialang.org/t/function-in-flux-to-estimate-learning-rate/21734/6 "2019-03-12T04:46:24Z")

</div>

Doesn’t ADAM automatically work out the learning rate adaptively? So you don’t need to specify the learning rate like in SGD.

---

<div class="post-metadata">

**Author:** ![samq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samq/32/47935_2.png) [@samq](https://discourse.julialang.org/u/samq)\
**Post date:** [March 12, 2019, 6:54pm UTC](https://discourse.julialang.org/t/function-in-flux-to-estimate-learning-rate/21734/7 "2019-03-12T18:54:46Z")

</div>

@xiaodai You are right about Adam modifying the learning rate adaptively.

Yet, the code at this URL: [Function in Flux to estimate learning rate - #5 by samq](https://discourse.julialang.org/t/function-in-flux-to-estimate-learning-rate/21734/5)  
uses Adam as the optimizer, but changes the learning rate anyway here:

```julia
# If we haven't seen improvement in 5 epochs, drop our learning rate:
    if epoch_idx - last_improvement >= 5 && opt.eta > 1e-6
        opt.eta /= 10.0
        @warn(" -> Haven't improved in a while, dropping learning rate to $(opt.eta)!")

        # After dropping learning rate, give it a few epochs to improve
        last_improvement = epoch_idx
    end

```

The reasoning behind doing this can be found at this URL: [neural network - Should we do learning rate decay for adam optimizer - Stack Overflow](https://stackoverflow.com/questions/39517431/should-we-do-learningrate-decay-for-adam-optimizer)

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [March 12, 2019, 9:36pm UTC](https://discourse.julialang.org/t/function-in-flux-to-estimate-learning-rate/21734/8 "2019-03-12T21:36:39Z")

</div>

> Doesn’t ADAM automatically work out the learning rate adaptively? So you don’t need to specify the learning rate like in SGD.

My understanding is that this is less important for ADAM, but some papers have shown that it does still help.