# RNN (LSTM) Training issues with loss function

**URL:** <https://discourse.julialang.org/t/rnn-lstm-training-issues-with-loss-function/53483>\
**Category:** Machine Learning\
**Tags:** question, lstm\
**Created:** [January 17, 2021, 12:11pm UTC](https://discourse.julialang.org/t/rnn-lstm-training-issues-with-loss-function/53483 "2021-01-17T12:11:36Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jparcinski](https://avatars.discourse-cdn.com/v4/letter/j/9fc348/32.png) [@jparcinski](https://discourse.julialang.org/u/jparcinski)\
**Post date:** [January 17, 2021, 12:11pm UTC](https://discourse.julialang.org/t/rnn-lstm-training-issues-with-loss-function/53483/1 "2021-01-17T12:11:36Z")

</div>

Hi, I have problem with understanding 1 issue, maybe someone will help me.

I’m trying to build LSTM or simple RNN for predicting price values. To start understand how to do it in Julia I used the example from here: [A Basic RNN](http://learningjulia.com/2019/10/11/a-basic-rnn.html#generate-data). I copied the code and got the same results, as in the example.

Later I tried to switch the example data to my own dataset with price values (using same format of arrays etc!). Unfortunately whatever hypermarameters I set, the same problem kept happening. NN seemed to work fine and succesfully could predict values (in some cases - mostly RNN instead of LSTM), however loss function during training was returning multiple values (2-83) instead of 1 like in the example.

┌ Info: Epoch 1  
└ @ Main C:\Users\japa.julia\packages\Flux\Fj3bt\src\optimise\train.jl:121

sum(loss.(hist\_test, target\_test)) = 5195.030318554835  
sum(loss.(hist\_test, target\_test)) = 14109.728906001292

┌ Info: Epoch 2  
└ @ Main C:\Users\japa.julia\packages\Flux\Fj3bt\src\optimise\train.jl:121

sum(loss.(hist\_test, target\_test)) = 4988.243513415623  
sum(loss.(hist\_test, target\_test)) = 9636.505577573585.

_Each learning iteration was returning multiple loss value. Depending on the model and other parameters (I think), it was either 2 (like here) or even 80 sometimes._ Can someone please explain to me why it is happening? When I just execute loss function, before training, I get 1 result always:

println("Training loss before = ", sum(loss.(hist\_train, target\_train)))  
Training loss before = 293785.2022681254

My code:  
**#FUNCTIONS**

#Model 1  
simple\_rnn = Flux.RNN(1, 1, (x → x))

#Model 2 - same results. Here I executed code with Model 1.  
simple\_rnn = Chain(LSTM(memory,8), Dense(8,1))

function eval\_model(x)  
out = simple\_rnn.(x)[end]  
Flux.reset!(simple\_rnn)  
out  
end

**#PARAMETERS**  
loss(x, y) = abs(sum((eval\_model(x) .- y)))  
ps = Flux.params(simple\_rnn)  
opt = Flux.ADAM()

**#TRAINING**  
evalcb() = @show(sum(loss.(hist\_test, target\_test)))  
@epochs num\_epochs Flux.train!(loss, ps, zip(hist\_train, target\_train), opt, cb = Flux.throttle(evalcb, 1))

**Input data format:**

**Hist\_train:**  
1000-element Array{Array{Float64,1},1}:  
[228.925, 229.072, 229.367, 229.203, 229.28, 229.399, 229.438]  
[229.072, 229.367, 229.203, 229.28, 229.399, 229.438, 229.732]  
[229.367, 229.203, 229.28, 229.399, 229.438, 229.732, 230.634]  
[229.203, 229.28, 229.399, 229.438, 229.732, 230.634, 231.369]  
[229.28, 229.399, 229.438, 229.732, 230.634, 231.369, 231.761]  
[229.399, 229.438, 229.732, 230.634, 231.369, 231.761, 232.06]  
[229.438, 229.732, 230.634, 231.369, 231.761, 232.06, 231.605]  
[229.732, 230.634, 231.369, 231.761, 232.06, 231.605, 231.998]

**target\_train:**  
1000-element Array{Float64,1}:  
227.233  
226.61  
226.373  
226.538  
226.688  
226.37  
226.276  
226.405  
226.42  
226.891

_So to sum up: I have the same model, the exact same code. I’m using the same format of input data. I even tried to set exactly 7 memory values (like in the example). And yet when I execute my code it behaves differently. Please help me, because I don’t know what is happening, maybe i missed something. Thanks!_