I’m not sure there’s anything wrong with your theory, but it’s hard to tell if there might be any bugs in the implementation. Have a look at https://github.com/FluxML/model-zoo/blob/master/text/treebank/recursive.jl and see if that helps. I’ve not used RNNs in Flux 0.9 either, so it would be great if you could create a MWE per Please read: make it easier to help you as well.