# Create a layer in Flux

**URL:** <https://discourse.julialang.org/t/create-a-layer-in-flux/89468>\
**Category:** Machine Learning\
**Tags:** flux\
**Created:** [October 28, 2022, 9:08pm UTC](https://discourse.julialang.org/t/create-a-layer-in-flux/89468 "2022-10-28T21:08:03Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![qwerty](https://avatars.discourse-cdn.com/v4/letter/q/4491bb/32.png) [@qwerty](https://discourse.julialang.org/u/qwerty)\
**Post date:** [October 28, 2022, 9:08pm UTC](https://discourse.julialang.org/t/create-a-layer-in-flux/89468/1 "2022-10-28T21:08:03Z")

</div>

What are the best practices for creating a custom layer in Flux?  
I noticed that, for example, creating a layer with internal state is not trivial.  
What should I do in general if I want to create a layer?

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [October 29, 2022, 12:58am UTC](https://discourse.julialang.org/t/create-a-layer-in-flux/89468/2 "2022-10-29T00:58:20Z")

</div>

Not sure what “internal state” means here, but if it means what I think it means doing this should be pretty trivial. Have you seen [Advanced Model Building · Flux](https://fluxml.ai/Flux.jl/stable/models/advanced/)?

---

<div class="post-metadata">

**Author:** ![qwerty](https://avatars.discourse-cdn.com/v4/letter/q/4491bb/32.png) [@qwerty](https://discourse.julialang.org/u/qwerty)\
**Post date:** [October 31, 2022, 9:14pm UTC](https://discourse.julialang.org/t/create-a-layer-in-flux/89468/3 "2022-10-31T21:14:12Z")

</div>

For example, a recurrent layer has a state, how should I build a recurrent layer?  
I know that Zygote does not allow to change variables, the state here, and it is preferable not to use global variables in julia language. Any recommendations?

---

<div class="post-metadata">

**Author:** ![bertschi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bertschi/32/33462_2.png) [@bertschi](https://discourse.julialang.org/u/bertschi)\
**Post date:** [October 31, 2022, 9:49pm UTC](https://discourse.julialang.org/t/create-a-layer-in-flux/89468/4 "2022-10-31T21:49:49Z")

</div>

According to the documentation [recurrence](https://fluxml.ai/Flux.jl/stable/models/recurrence/) is straight-forward in Flux. Basically, you define a (stateless) layer taking two inputs `h, x` and producing two outputs `h', y`, i.e., the state `h` is explicitly passed in and returned. Then, [Flux.Recur](https://fluxml.ai/Flux.jl/stable/models/layers/#Flux.Recur) can be used to implicitly pass the state, i.e., turns your layer into a stateful layer.

---

<div class="post-metadata">

**Author:** ![qwerty](https://avatars.discourse-cdn.com/v4/letter/q/4491bb/32.png) [@qwerty](https://discourse.julialang.org/u/qwerty)\
**Post date:** [November 4, 2022, 7:00pm UTC](https://discourse.julialang.org/t/create-a-layer-in-flux/89468/5 "2022-11-04T19:00:07Z")

</div>

What do you think of this code?

```julia
# custom rnn layer
using Flux, Functors

struct RecModel
    W_h::Matrix{Float32}
    W_i::Matrix{Float32}
    b_h::Vector{Float32}
    b_i::Vector{Float32}
    state0::Vector{Float32}
end
(m::RecModel)(h, x) = (relu(m.W_h*h .+ m.b_h), relu(m.W_i*x .+ m.W_h*h .+ m.b_i))
@functor RecModel (W_h, W_i, b_h, b_i)

m = RecModel(Flux.glorot_normal(10, 10), Flux.glorot_normal(10, 10), randn(Float32, 10), randn(Float32, 10), zeros(Float32, 10))
model = Flux.Recur(m, zeros(Float32, 10))
p = Flux.params(model)

loss(x, y) = Flux.mse(model(x), y)
loss(rand(Float32, 10), rand(Float32, 10))

g = Flux.gradient(()->loss(rand(Float32, 10), rand(Float32, 10)), p)

```

and without `Flux.Recur`:

```julia
mutable struct MyRecur{T, S}
    cell::T
    state::S
end
@functor MyRecur
Flux.trainable(m::MyRecur) = (m.cell,)
function (m::MyRecur)(x)
    m.state, y = m.cell(m.state, x)
    y
end
my_reset!(m::MyRecur) = 
    m.state = m.cell.state0 

m = RecModel(Flux.glorot_normal(10, 10), Flux.glorot_normal(10, 10), randn(Float32, 10), randn(Float32, 10), zeros(Float32, 10))
model = MyRecur(m, zeros(Float32, 10))
p = Flux.params(model)

loss(x, y) = Flux.mse(model(x), y)
loss(rand(Float32, 10), rand(Float32, 10))
    
g = Flux.gradient(()->loss(rand(Float32, 10), rand(Float32, 10)), p)

```