# Batch training for LSTMs in Flux or Knet
**URL:** https://discourse.julialang.org/t/batch-training-for-lstms-in-flux-or-knet/47762
**Category:** New to Julia
**Tags:** question, knet, flux
**Created:** [October 5, 2020, 6:06am UTC](https://discourse.julialang.org/t/batch-training-for-lstms-in-flux-or-knet/47762 "2020-10-05T06:06:25Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@danielw2904](https://discourse.julialang.org/u/danielw2904)
#### Post date: [October 5, 2020, 6:06am UTC](https://discourse.julialang.org/t/batch-training-for-lstms-in-flux-or-knet/47762/1 "2020-10-05T06:06:25Z")
I’m new to Flux so apologies if this is a dumb question/ has already been answered (I cannot figure it out based on a search here and the model zoo)
I am trying to train an LSTM on a sequence of choices where more than one thing can be chosen per time period. Let’s say we have 3 periods and 5 possible choices. The “multi-hot” representation would look like this:
```julia
X₁ = [[0;1;0;0;1], [1;0;0;0;0], [0;0;1;1;0]]
```
where the choices at time t are
```julia
X₁[t]
```
I then create my labels by using the choices of the next period:
```julia
y₁ = X₁[2:end]
X₁ = X₁[1:end-1]
```
And run the following model:
```julia
data = Flux.Data.DataLoader((X₁, y₁))
m = Chain(LSTM(5,5), softmax)
loss(x, y) = sum(logitcrossentropy.(m.(x), y))
loss(X₁, y₁)
opt = ADAM()
ps = Flux.params(m)
for _ in 1:10000
Flux.train!(loss, ps, data, opt)
end
```
Which seems to work:
```julia
julia> reset!(m)
julia> m(X₁[2])
5-element Array{Float32,1}:
0.07553877
0.19324568
0.31979644
0.3028726
0.10854646
```
My questions are:
a) Does this seem correct?
b) When does the hidden state have to be reset?
c) How can I extend this example to batch training?
If this is more easily done in Knet I will try that.
Thanks! Any help is appreciated!
---
### Author:  [@dhairyagandhi96](https://discourse.julialang.org/u/dhairyagandhi96)
#### Post date: [October 5, 2020, 5:54pm UTC](https://discourse.julialang.org/t/batch-training-for-lstms-in-flux-or-knet/47762/2 "2020-10-05T17:54:48Z")
You typically want to reset after calculating the loss since you don’t want to accumulate the gradients beyond one optimisation step.
I would be confused about how the labels are encoded, but I guess it’s fine to do it this way.
I guess the question is what you’re looking to do with it?
Also [here is a simpler lstm example](https://github.com/FluxML/model-zoo/blob/dg/zygote/text/char-rnn/char-rnn.jl)
---
### Author:  [@danielw2904](https://discourse.julialang.org/u/danielw2904)
#### Post date: [October 5, 2020, 6:07pm UTC](https://discourse.julialang.org/t/batch-training-for-lstms-in-flux-or-knet/47762/3 "2020-10-05T18:07:47Z")
Thank you that helps a lot! So basically the loss in the example runs the model on one batch of sequences and then resets the hidden state. Am I getting this right?
---
### Author:  [@danielw2904](https://discourse.julialang.org/u/danielw2904)
#### Post date: [October 5, 2020, 6:57pm UTC](https://discourse.julialang.org/t/batch-training-for-lstms-in-flux-or-knet/47762/4 "2020-10-05T18:57:28Z")
> [@dhairyagandhi96](#):
>
> would be confused about how the labels are encoded
Oh I’m confused alright. 😃 but what would be the alternative?
---
### Author:  [@denizyuret](https://discourse.julialang.org/u/denizyuret)
#### Post date: [October 18, 2020, 7:02pm UTC](https://discourse.julialang.org/t/batch-training-for-lstms-in-flux-or-knet/47762/5 "2020-10-18T19:02:00Z")
Knet has some RNN examples in the tutorial: [https://github.com/denizyuret/Knet.jl/tree/master/tutorial](https://github.com/denizyuret/Knet.jl/tree/master/tutorial)
---
### Author:  [@danielw2904](https://discourse.julialang.org/u/danielw2904)
#### Post date: [October 18, 2020, 7:57pm UTC](https://discourse.julialang.org/t/batch-training-for-lstms-in-flux-or-knet/47762/6 "2020-10-18T19:57:40Z")
Thank you. I will take a look!