# Flux.jl: Different set of optimisation parameters per layer

**URL:** https://discourse.julialang.org/t/flux-jl-different-set-of-optimisation-parameters-per-layer/22518
**Category:** Machine Learning
**Created:** [March 29, 2019, 9:56pm UTC](https://discourse.julialang.org/t/flux-jl-different-set-of-optimisation-parameters-per-layer/22518 "2019-03-29T21:56:43Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![rederekt](https://avatars.discourse-cdn.com/v4/letter/r/d07c76/32.png) [@rederekt](https://discourse.julialang.org/u/rederekt)
#### Post date: [March 29, 2019, 9:56pm UTC](https://discourse.julialang.org/t/flux-jl-different-set-of-optimisation-parameters-per-layer/22518/1 "2019-03-29T21:56:43Z")

</div>

Hi,

Just copying over [my question from the GitHub issues](https://github.com/FluxML/Flux.jl/issues/715) to hopefully get more visibility here.

In Flux.jl, I’d like to use a separate set of optimisation parameters for each hidden layer in a single Chain. For example, if I had a pre-trained network I wanted to append layers to, I’d want to make the learning rate for those layers much less than the learning rate for my new layers. Maybe I’d want the momentum, dropout, etc. to be different as well.

Looking at [the source code](https://github.com/FluxML/Flux.jl/blob/3a4c6274fadf1b94468da455a06f070b2cda6a64/src/optimise/train.jl#L64), using multiple Optimisers during training may have once been possible, but nothing elsewhere in the documentation (from what I can tell) indicates that it is now. Am I missing something obvious? Thanks in advance!

_Edit: Using Julia 1.0.3 and Flux 0.7.3_

---

<div class="post-metadata">

### Author: ![rederekt](https://avatars.discourse-cdn.com/v4/letter/r/d07c76/32.png) [@rederekt](https://discourse.julialang.org/u/rederekt)
#### Post date: [March 30, 2019, 8:59pm UTC](https://discourse.julialang.org/t/flux-jl-different-set-of-optimisation-parameters-per-layer/22518/2 "2019-03-30T20:59:49Z")

</div>

Update: this should work just fine for my purposes.

```julia
distribute(opts, m) = collect(Iterators.flatten([repeat([opt], length(params(l))) for (l, opt) in zip(m, opts)]))

function train!(loss, ps, data, opts::Array; cb = () -> ())
  cb = runall(cb)
	ps = Params(ps)
  for d in data
    try
      gs = gradient(ps) do
        loss(d...)
      end
      for (p, opt) in zip(ps, opts) 
        update!(opt, p, gs[p])
      end
      if cb() == :stop
        depwarn("Use of `:stop` is deprecated; use `Flux.stop()` instead", :stop)
        break
      end
    catch ex
      if ex isa StopException
        break
      else
        rethrow(ex)
      end
    end
  end
end

m = Chain(Dense(2,4,tanh), Dense(2,4,tanh))

opts = distribute([Descent(0.3), Descent(0.1)], m) # [Descent(0.3), Descent(0.3), Descent(0.1), Descent(0.1)]

@epochs 500 train!(loss, params(m), data, opts)

```