# RNN is not trained

**URL:** <https://discourse.julialang.org/t/rnn-is-not-trained/59553>\
**Category:** Machine Learning\
**Created:** [April 18, 2021, 4:51pm UTC](https://discourse.julialang.org/t/rnn-is-not-trained/59553 "2021-04-18T16:51:10Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sunny](https://avatars.discourse-cdn.com/v4/letter/s/7c8e57/32.png) [@Sunny](https://discourse.julialang.org/u/Sunny)\
**Post date:** [April 18, 2021, 4:51pm UTC](https://discourse.julialang.org/t/rnn-is-not-trained/59553/1 "2021-04-18T16:51:10Z")

</div>

I’m trying to reproduce this [example](https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/) but my model won’t train.  
I use only train [dataset](https://raw.githubusercontent.com/jbrownlee/Datasets/master/airline-passengers.csv) to achieve overfitting and get just a line as prediction:

```julia
using Pandas
using Plots
using Flux

df = read_csv("airline-passengers.csv")

# Get the two datasets of one point time-series
# The train data and target 
train = values(df[1:length(df)-2].Passengers)
target_train = values(df[2:length(df)].Passengers)

Plots.plot(train)
plot!(target_train)

train = [[convert(Float32, train[i])] for i = 1:length(train)]
train = reshape(train, (length(train), 1))
target_train = reshape(target_train, (length(target_train), 1))
target_train = [convert(Float32, target_train[i]) for i = 1:length(target_train)]

# Model and Training 

model = Chain(RNN(1,4), Dense(4, 1, x->x))
loss(x, y) = sum(abs2.(eval_model(x) .- y))

function eval_model(x)
    #@show x
    out = model.(x)
    @show out
    #Flux.reset!(model)
    out = [out[i][1] for i=1:length(out)]
end

ps = Flux.params(model);
opt = Flux.ADAM(0.01)
epochs = 100

for epoch in 1:epochs
        @show epoch
        #loss(train, target_train)
        gs = Flux.gradient(ps) do
             @show loss(train, target_train)
        end
        Flux.Optimise.update!(opt, ps, gs)
end

out = model.(train)
out = [out[i][1] for i=1:length(out)]
Plots.plot(out)

```

Does anybody see any problem?

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [April 18, 2021, 5:35pm UTC](https://discourse.julialang.org/t/rnn-is-not-trained/59553/2 "2021-04-18T17:35:48Z")

</div>

`reset!` needs to be called after every minibatch, else the network will keep on accumulating with residual state from the last set of sequences.

---

<div class="post-metadata">

**Author:** ![Sunny](https://avatars.discourse-cdn.com/v4/letter/s/7c8e57/32.png) [@Sunny](https://discourse.julialang.org/u/Sunny)\
**Post date:** [April 18, 2021, 6:35pm UTC](https://discourse.julialang.org/t/rnn-is-not-trained/59553/3 "2021-04-18T18:35:33Z")

</div>

I uncommented line with reset. But only `Dense` weights have non-zero grad the rest (all for RNN) are zeros

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [April 18, 2021, 6:38pm UTC](https://discourse.julialang.org/t/rnn-is-not-trained/59553/4 "2021-04-18T18:38:24Z")

</div>

The line with reset! isn’t called at all during training. You’d need to move it to after `gs = ` or `update!`.

---

<div class="post-metadata">

**Author:** ![Sunny](https://avatars.discourse-cdn.com/v4/letter/s/7c8e57/32.png) [@Sunny](https://discourse.julialang.org/u/Sunny)\
**Post date:** [April 18, 2021, 6:40pm UTC](https://discourse.julialang.org/t/rnn-is-not-trained/59553/5 "2021-04-18T18:40:47Z")

</div>

```julia
for epoch in 1:epochs
        @show epoch
        #loss(train, target_train)
        gs = Flux.gradient(ps) do
             @show loss(train, target_train)
        end
        Flux.Optimise.update!(opt, ps, gs)
        Flux.reset!(model)
end

```

I moved now.  
If I use `Dense(1,1)` it trains, if add `RNN`, it is not updated

---

<div class="post-metadata">

**Author:** ![mkschleg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkschleg/32/8035_2.png) [@mkschleg](https://discourse.julialang.org/u/mkschleg)\
**Post date:** [April 20, 2021, 10:46pm UTC](https://discourse.julialang.org/t/rnn-is-not-trained/59553/6 "2021-04-20T22:46:12Z")

</div>

> [@Sunny](#):
>
> `out = model.(x)`

Maybe try

```julia
out = map(model, x)

```

I know there was a bug sometime back where RNNs wouldn’t work as expected w/ broadcasting in Zygote. I’m not sure if it has been fixed. If that doesn’t work I have an [example](https://github.com/mkschleg/FluxBooks.jl/blob/main/RNNs.ipynb) using 0.10.x/0.11.x LSTMs (although glancing at the code it should work for the new 0.12.x revamp), which should work w/ RNNs as well.

---

<div class="post-metadata">

**Author:** ![jeremiedb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeremiedb/32/29150_2.png) [@jeremiedb](https://discourse.julialang.org/u/jeremiedb)\
**Post date:** [April 22, 2021, 5:31pm UTC](https://discourse.julialang.org/t/rnn-is-not-trained/59553/7 "2021-04-22T17:31:10Z")

</div>

Here’s an example that has been a little bit reworked that looks to train properly:

```julia
using CSV
using DataFrames
using Flux
using Plots

df = DataFrame(CSV.File("airline-passengers.txt"))

# Get the two datasets of one point time-series
# The train data and target 
X_raw = Float32.(df[!,:Passengers][1:nrow(df)-1]) ./ 100
Y_raw = Float32.(df[!,:Passengers][2:nrow(df)]) ./ 100

Plots.plot(X_raw)
plot!(Y_raw)

X = [X_raw[i:i] for i in 1:length(X_raw)]
Y = Y_raw

# Model and Training 
model = Chain(RNN(1,4), Dense(4, 1), x -> reshape(x, :))
# model.(X)
loss(x, y) = sum(abs2.(Flux.stack(model.(x), 1) .- y))

ps = Flux.params(model);
opt = Flux.ADAM(0.01)
epochs = 100

for epoch in 1:epochs
        @show epoch
        gs = Flux.gradient(ps) do
             loss(X, Y)
        end
        Flux.Optimise.update!(opt, ps, gs)
end

out = vec(Flux.stack(model.(X), 1))
Plots.plot(out)
plot!(Y)

```

Something that may have contributed to the issue is the scaling the of the features/target. Here I roughly divided each by 100.