Simple Flux LSTM for Time Series

Specific Domains Machine Learning
, , ,
61

I’m not sure where you get this model reference from. The official reference uses the code I mentioned above.

https://fluxml.ai/Flux.jl/stable/models/layers/#Flux.LSTM
https://fluxml.ai/Flux.jl/stable/models/layers/#Flux.GRU

Even an outdated version of the docs should not have some activation function for LSTM and GRU due to the way these networks work. There is no activation function in the same sense as there is for RNN or Dense layers.

In my opinion, this resource, which I found to be quite useful, makes it clearer why a user-defined activation function for LSTM does not make sense: Understanding LSTM Networks -- colah's blog. I hope this helps!

62

https://fluxml.ai/Flux.jl/v0.5/models/layers.html

[The top google search for Julia Flux LSTM]

63

I stand corrected. So an older version does indeed have a parameter for an activation function, I’m not sure why that would be useful, perhaps someone more knowledgeable can chime in. It is unfortunate that Google returns an old version of the docs. Do use the one I linked above, it is the stable version.

Note that you could also add any custom function to the output of the LSTM using Chain.

Chain(LSTM(in, out), x -> σ.(x))