# Convolutional GRU/LSTM in Flux?

**URL:** <https://discourse.julialang.org/t/convolutional-gru-lstm-in-flux/74770>\
**Category:** Machine Learning\
**Created:** [January 17, 2022, 8:32pm UTC](https://discourse.julialang.org/t/convolutional-gru-lstm-in-flux/74770 "2022-01-17T20:32:28Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![johnbb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnbb/32/34233_2.png) [@johnbb](https://discourse.julialang.org/u/johnbb)\
**Post date:** [January 17, 2022, 8:32pm UTC](https://discourse.julialang.org/t/convolutional-gru-lstm-in-flux/74770/1 "2022-01-17T20:32:28Z")

</div>

Is there any implementation of (2D) convolutional GRU, LSTM or similar based on Flux? A similar question was raised in [Sept. 2020](https://discourse.julialang.org/t/using-lstm-with-conv-layers/46007).

---

<div class="post-metadata">

**Author:** ![darsnack](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/darsnack/32/10144_2.png) [@darsnack](https://discourse.julialang.org/u/darsnack)\
**Post date:** [January 19, 2022, 3:39pm UTC](https://discourse.julialang.org/t/convolutional-gru-lstm-in-flux/74770/2 "2022-01-19T15:39:46Z")

</div>

Not a complete answer, but one major change since that post is that the recurrent layers in Flux will [now accept 3D arrays](https://github.com/FluxML/Flux.jl/pull/1686) as an input format. Seems like that was the issue in the previous OP’s code? Maybe try that to see if it helps?

---

<div class="post-metadata">

**Author:** ![johnbb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnbb/32/34233_2.png) [@johnbb](https://discourse.julialang.org/u/johnbb)\
**Post date:** [January 20, 2022, 9:16am UTC](https://discourse.julialang.org/t/convolutional-gru-lstm-in-flux/74770/3 "2022-01-20T09:16:22Z")

</div>

Will that somehow be able to handle data where a single input sample is 4D (2 spatial dimensions, time, variable/channel)?

---

<div class="post-metadata">

**Author:** ![mkschleg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkschleg/32/8035_2.png) [@mkschleg](https://discourse.julialang.org/u/mkschleg)\
**Post date:** [January 20, 2022, 8:42pm UTC](https://discourse.julialang.org/t/convolutional-gru-lstm-in-flux/74770/4 "2022-01-20T20:42:06Z")

</div>

Not yet. We are still iterating on some of the changes that need to happen to Recur to make this possible. What would be a viable strategy (I think) is to overwrite the behaviour of `Recur{ConvGRU}` by specializing:

```nohighlight
function (m::Flux.Recur{ConvGRU})(x)
    # special behaviour here
end

```

Now this isn’t really scalable. I’ve been iterating on something that looks like a trait based system to unify some of the architectures. You can see some of this in [this repo](https://github.com/mkschleg/FluxRNNs.jl). I’ve been unable to work on this as much as I want due to various constraints w/ my degree/other projects/other work, so would be interested to have a basic implementation of a ConvRNN/ConvGRU/ConvLSTM that we can use to iterate.

---

<div class="post-metadata">

**Author:** ![mkschleg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkschleg/32/8035_2.png) [@mkschleg](https://discourse.julialang.org/u/mkschleg)\
**Post date:** [January 20, 2022, 8:46pm UTC](https://discourse.julialang.org/t/convolutional-gru-lstm-in-flux/74770/5 "2022-01-20T20:46:21Z")

</div>

I would also recommend the time dimension being the last, as I think we have unofficially agreed this to be standard (see [https://github.com/FluxML/Flux.jl/issues/1678](https://github.com/FluxML/Flux.jl/issues/1678)).