# Parallel feedforward computation with RNN

**URL:** https://discourse.julialang.org/t/parallel-feedforward-computation-with-rnn/26378
**Category:** Machine Learning
**Tags:** flux
**Created:** [July 15, 2019, 8:51am UTC](https://discourse.julialang.org/t/parallel-feedforward-computation-with-rnn/26378 "2019-07-15T08:51:04Z")
**Posts on this page:** 1
**Page:** 1

<div class="post-metadata">

### Author: ![HenriDeh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrideh/32/8316_2.png) [@HenriDeh](https://discourse.julialang.org/u/HenriDeh)
#### Post date: [July 15, 2019, 8:51am UTC](https://discourse.julialang.org/t/parallel-feedforward-computation-with-rnn/26378/1 "2019-07-15T08:51:05Z")

</div>

Say you have a sample of 3 observations-label as a training set for a standard fully connected NN. To parallelize the computation of the loss (and the gradient) of each label, you can feed the three observations to the NN as a matrix instead of sequentially feeding the three observation vectors. Like so:

```julia
net = Chain(Dense(10,5,relu),Dense(5,1,relu)) |> gpu
x = cu(rand(10,3)) #each column of that matrix is one observation
y = cu(rand(1,3)) #each element is one label
output = net(x)
loss = Flux.mse(output,y)

```

and that’s it, the gpu matrix multiplication took care of the parallelization. However as I understand it, a RNN changes the inner state of the neural network depending on the previous inputs of the network. That means that if I input 3 different observations (of potentially different sizes), to parallelize the computation of the gradients, the state of three different networks will have to be kept in memory. Is that not a significant drawback of RNN ? Does Flux implements parallel feedforwards for RNN or do I have to create a gpu kernel or something ?