# LSTM with batch vs with individual values

**URL:** <https://discourse.julialang.org/t/lstm-with-batch-vs-with-individual-values/73722>\
**Category:** Machine Learning\
**Tags:** flux, time-series\
**Created:** [December 28, 2021, 7:28pm UTC](https://discourse.julialang.org/t/lstm-with-batch-vs-with-individual-values/73722 "2021-12-28T19:28:26Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ForceBru](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/forcebru/32/21389_2.png) [@ForceBru](https://discourse.julialang.org/u/ForceBru)\
**Post date:** [December 28, 2021, 7:28pm UTC](https://discourse.julialang.org/t/lstm-with-batch-vs-with-individual-values/73722/1 "2021-12-28T19:28:26Z")

</div>

Looks like there are two ways of calling models (running forecasting/regression/etc) in Flux: “batch mode” (`model(data_matrix)`) and “vector comprehension mode” (`[model(elem)[1] for elem in data_vector]`), where `elem` is one input vector.

These two ways of calling the model produce different results:

```julia
let model = Chain(LSTM(1, 3))
    data = Float32[1,2,3,4,5,6]
    
    model(data') |> display
    
    Flux.reset!(model)
    [model([d]) for d in data]
end

```

## Output

```julia-auto
3×6 Matrix{Float32}:
 -0.0297324 -0.0248715 -0.0134866 -0.00586918 -0.00224418 -0.0007936
  0.115042 0.23459 0.325777 0.387996 0.431214 0.463294
  0.0504465 0.0913864 0.111791 0.112188 0.0996351 0.0816775

6-element Vector{Vector{Float32}}:
 [-0.029732374, 0.11504154, 0.050446533]
 [-0.03805526, 0.32181326, 0.15570784]
 [-0.03267687, 0.50102127, 0.286734]
 [-0.02288559, 0.5983699, 0.4003932]
 [-0.0133526595, 0.64711636, 0.48314008]
 [-0.006731839, 0.6773652, 0.5399319]

```

The results are different, even though I called `Flux.reset!(model)` before computing the second output.

## Question

Why does this happen? Is this because of broadcasting issues discussed [in the docs](https://fluxml.ai/Flux.jl/stable/models/recurrence/#Working-with-sequences)?

> Mapping and broadcasting operations with stateful layers such are discouraged, since the julia language doesn’t guarantee a specific execution order.

However, `model(data')` executes much faster and uses all CPU cores, while the vector comprehension version is really slow (1 epoch per second vs 10 epochs per second) and uses only one core. Which one should I use?

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [December 28, 2021, 8:06pm UTC](https://discourse.julialang.org/t/lstm-with-batch-vs-with-individual-values/73722/2 "2021-12-28T20:06:16Z")

</div>

`model(data')` is feeding in a _single_ timestep with 1 feature and batch size 6. `[model([d]) for d in data] `. Is feeding in _6_ timesteps with 1 feature and batch size 1. If you want to want to feed multiple timesteps to an RNN simultaneously, pass in a 3D array of shape `features x batch x timesteps`. This is covered in [Recurrence · Flux](https://fluxml.ai/Flux.jl/dev/models/recurrence/), but since we haven’t had a stable docs build in a while that may well be falling through the cracks.

---

<div class="post-metadata">

**Author:** ![visr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/visr/32/17204_2.png) [@visr](https://discourse.julialang.org/u/visr)\
**Post date:** [December 29, 2021, 5:06pm UTC](https://discourse.julialang.org/t/lstm-with-batch-vs-with-individual-values/73722/3 "2021-12-29T17:06:26Z")

</div>

Ah I didn’t know 3D `features x batch x timesteps` was also supported. It doesn’t seem to be covered on the recurrence page you linked, also on master. I believe you are referring to this addition: [#1686](https://github.com/FluxML/Flux.jl/pull/1686)? I just tried it, however one downside of this shape versus a vector of matrices `batch x (feature x timesteps)` seems to be that the DataLoaders expect the batch (observation) dimension to be the last one.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [January 3, 2022, 12:49am UTC](https://discourse.julialang.org/t/lstm-with-batch-vs-with-individual-values/73722/4 "2022-01-03T00:49:36Z")

</div>

It’s at the bottom of the page:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/5/8/587c7c5e76d1121c5dd757a5afcd5b7dcd31ebfd.png)

Suggestions/(preferably) PRs for a better place to put it very much welcome 🙂

---

<div class="post-metadata">

**Author:** ![visr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/visr/32/17204_2.png) [@visr](https://discourse.julialang.org/u/visr)\
**Post date:** [January 3, 2022, 10:37am UTC](https://discourse.julialang.org/t/lstm-with-batch-vs-with-individual-values/73722/5 "2022-01-03T10:37:37Z")

</div>

Yeah I think the recurrence docs are quite clear about the vector of `(features, samples)`, and pretty good overall. I meant specifically that 3D array support is only briefly noted in the `Recur` docstring, which I had missed before. Though perhaps it’s too early to advertise this more given the work in [https://github.com/FluxML/Flux.jl/issues/1678](https://github.com/FluxML/Flux.jl/issues/1678).