# How to apply a decay for learning rate?

**URL:** <https://discourse.julialang.org/t/how-to-apply-a-decay-for-learning-rate/81108>\
**Category:** New to Julia\
**Tags:** question, package, error, error-message\
**Created:** [May 15, 2022, 5:54pm UTC](https://discourse.julialang.org/t/how-to-apply-a-decay-for-learning-rate/81108 "2022-05-15T17:54:36Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![vamp](https://avatars.discourse-cdn.com/v4/letter/v/f4b2a3/32.png) [@vamp](https://discourse.julialang.org/u/vamp)\
**Post date:** [May 15, 2022, 5:54pm UTC](https://discourse.julialang.org/t/how-to-apply-a-decay-for-learning-rate/81108/1 "2022-05-15T17:54:36Z")

</div>

Hello,

I am very new to Julia, so I apologize if I do not explain the problem correctly (feel free to ask me).

I am trying to use [this solvers.](https://github.com/JuliaPOMDP/TabularTDLearning.jl/tree/master/src) For the `exploration_policy` I can use a decay with `exploration_policy= EpsGreedyPolicy( MDP,LinearDecaySchedule(start=1.0, stop=0.01, steps=10000))` but when I use it for ` learning_rate::Float64` it says that it can`t convert to `Float64` .

I saw [ParameterSchedulers.jl](https://github.com/darsnack/ParameterSchedulers.jl) but I do not know if I can use it and how.

Thank you 🙂

---

<div class="post-metadata">

**Author:** ![khorrami1](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/khorrami1/32/36257_2.png) [@khorrami1](https://discourse.julialang.org/u/khorrami1)\
**Post date:** [May 15, 2022, 9:39pm UTC](https://discourse.julialang.org/t/how-to-apply-a-decay-for-learning-rate/81108/2 "2022-05-15T21:39:46Z")

</div>

Generally speaking, Flux works with Flaot32. Try learning\_rate::Float32

---

<div class="post-metadata">

**Author:** ![skleinbo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skleinbo/32/36080_2.png) [@skleinbo](https://discourse.julialang.org/u/skleinbo)\
**Post date:** [May 16, 2022, 6:00am UTC](https://discourse.julialang.org/t/how-to-apply-a-decay-for-learning-rate/81108/3 "2022-05-16T06:00:49Z")

</div>

Please provide the full command and error message. Ideally a minimal working example too.

> **This works for example**
>
> ```julia
> julia> exppolicy = EpsGreedyPolicy(mdp,LinearDecaySchedule(start=1.0, stop=0.01, steps=10000))
> EpsGreedyPolicy{LinearDecaySchedule{Float64}, Random._GLOBAL_RNG, NTuple{4, Symbol}}(LinearDecaySchedule{Float64}(1.0, 0.01, 10000) (function of type LinearDecaySchedule{Float64})
> start: Float64 1.0
> stop: Float64 0.01
> steps: Int64 10000
> , Random._GLOBAL_RNG(), (:up, :down, :left, :right))
> 
> julia> solver = QLearningSolver(exploration_policy=exppolicy, learning_rate=0.1, n_episodes=5000, max_episode_length=50, eval_every=50, n_eval_traj=100)
> 
> ```

---

<div class="post-metadata">

**Author:** ![vamp](https://avatars.discourse-cdn.com/v4/letter/v/f4b2a3/32.png) [@vamp](https://discourse.julialang.org/u/vamp)\
**Post date:** [May 16, 2022, 6:42am UTC](https://discourse.julialang.org/t/how-to-apply-a-decay-for-learning-rate/81108/4 "2022-05-16T06:42:34Z")

</div>

Hello,

The decay is for the learning rate also

```julia
#Q-Learning solver
q_learning_solver = QLearningSolver(n_episodes=1000, 
                                max_episode_length = 1000,
                                learning_rate= LinearDecaySchedule(start=1.0, stop=0.0, steps=1000),
                                exploration_policy= EpsGreedyPolicy(mdp,LinearDecaySchedule(start=1.0, stop=0.0, steps=1000)), 
                                eval_every = 10000, 
                                n_eval_traj = 20, 
                                verbose=true) 

```

Error

```julia

ERROR: MethodError: Cannot `convert` an object of type LinearDecaySchedule{Float64} to an object of type Float64
Closest candidates are:
  convert(::Type{T}, ::ColorTypes.Gray24) where T<:Real at C:\Users\X\.julia\packages\ColorTypes\6m8P7\src\conversions.jl:114
  convert(::Type{T}, ::ColorTypes.Gray) where T<:Real at C:\Users\X\.julia\packages\ColorTypes\6m8P7\src\conversions.jl:113
  convert(::Type{T}, ::Unitful.Gain) where T<:Real at C:\Users\X\.julia\packages\Unitful\SUQzL\src\logarithm.jl:62    
  ...
Stacktrace:
 [1] QLearningSolver{EpsGreedyPolicy{LinearDecaySchedule{Float64}, Random._GLOBAL_RNG, Vector{Action}}}(n_episodes::Int64, max_episode_length::Int64, learning_rate::Function, exploration_policy::EpsGreedyPolicy{LinearDecaySchedule{Float64}, Random._GLOBAL_RNG, Vector{Action}}, Q_vals::Nothing, eval_every::Int64, n_eval_traj::Int64, rng::Random._GLOBAL_RNG, verbose::Bool)
   @ TabularTDLearning C:\Users\X\.julia\packages\Parameters\MK0O4\src\Parameters.jl:503
 [2] QLearningSolver(n_episodes::Int64, max_episode_length::Int64, learning_rate::Function, exploration_policy::EpsGreedyPolicy{LinearDecaySchedule{Float64}, Random._GLOBAL_RNG, Vector{Action}}, Q_vals::Nothing, eval_every::Int64, n_eval_traj::Int64, rng::Random._GLOBAL_RNG, verbose::Bool)
   @ TabularTDLearning C:\Users\X\.julia\packages\Parameters\MK0O4\src\Parameters.jl:526
 [3] QLearningSolver(; n_episodes::Int64, max_episode_length::Int64, learning_rate::Function, exploration_policy::EpsGreedyPolicy{LinearDecaySchedule{Float64}, Random._GLOBAL_RNG, Vector{Action}}, Q_vals::Nothing, eval_every::Int64, n_eval_traj::Int64, rng::Random._GLOBAL_RNG, verbose::Bool)
   @ TabularTDLearning C:\Users\X\.julia\packages\Parameters\MK0O4\src\Parameters.jl:545
 [4] top-level scope
   @ c:\Users\X\Desktop\X\X\VS Code Projects\Algortihms Test\X\X\X_v2.jl:103

```

---

<div class="post-metadata">

**Author:** ![skleinbo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skleinbo/32/36080_2.png) [@skleinbo](https://discourse.julialang.org/u/skleinbo)\
**Post date:** [May 16, 2022, 7:13am UTC](https://discourse.julialang.org/t/how-to-apply-a-decay-for-learning-rate/81108/5 "2022-05-16T07:13:26Z")

</div>

It appears this is currently not possible. Look at

> <https://github.com/JuliaPOMDP/TabularTDLearning.jl/blob/3630ddfdb1bac2e95912fa9648e14d6fb6831563/src/q_learn.jl#L62>

The solver assumes the `learning_rate` parameter to be a number, not a function which you are trying to pass with.

I guess it not too difficult to make it work. Might be worth opening an issue with `TabularTDLearning.jl`.

Maybe you could run the solver for fewer episodes, adjust the learning rate, restart the solve, and so on.

Is it sensible? I don’t have enough practical knowledge of POMDPs to answer that, but here is a SE question in that direction

[https://ai.stackexchange.com/questions/12268/in-q-learning-shouldnt-the-learning-rate-change-dynamically-during-the-learnin](https://ai.stackexchange.com/questions/12268/in-q-learning-shouldnt-the-learning-rate-change-dynamically-during-the-learnin)

---

<div class="post-metadata">

**Author:** ![vamp](https://avatars.discourse-cdn.com/v4/letter/v/f4b2a3/32.png) [@vamp](https://discourse.julialang.org/u/vamp)\
**Post date:** [May 16, 2022, 7:18am UTC](https://discourse.julialang.org/t/how-to-apply-a-decay-for-learning-rate/81108/6 "2022-05-16T07:18:13Z")

</div>

Hello,

Thank you so much for your help, I will take a look to it!

Thanks 🙂