# Is there an implementation of the attention mechanism in Flux.jl?

**URL:** <https://discourse.julialang.org/t/is-there-an-implementation-of-the-attention-mechanism-in-flux-jl/26959>\
**Category:** Machine Learning\
**Tags:** flux\
**Created:** [July 29, 2019, 9:39pm UTC](https://discourse.julialang.org/t/is-there-an-implementation-of-the-attention-mechanism-in-flux-jl/26959 "2019-07-29T21:39:33Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Azamat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/azamat/32/6892_2.png) [@Azamat](https://discourse.julialang.org/u/Azamat)\
**Post date:** [July 29, 2019, 9:39pm UTC](https://discourse.julialang.org/t/is-there-an-implementation-of-the-attention-mechanism-in-flux-jl/26959/1 "2019-07-29T21:39:33Z")

</div>

Specifically, I’m interested in the attention mechanism described in [Listen, Attend and Spell](https://arxiv.org/abs/1508.01211) work (where it’s referred to as AttentionContext), but more generally any other kind of attention’s implementation would be quite useful to take a look at.

---

<div class="post-metadata">

**Author:** ![fborda](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fborda/32/8654_2.png) [@fborda](https://discourse.julialang.org/u/fborda)\
**Post date:** [July 29, 2019, 10:03pm UTC](https://discourse.julialang.org/t/is-there-an-implementation-of-the-attention-mechanism-in-flux-jl/26959/2 "2019-07-29T22:03:24Z")

</div>

In terms of attention, the Transformer is probably interesting to check, though there are probably simpler attention models in Flux around.

[https://github.com/chengchingwen/Transformers.jl](https://github.com/chengchingwen/Transformers.jl)

---

<div class="post-metadata">

**Author:** ![merckxiaan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/merckxiaan/32/4697_2.png) [@merckxiaan](https://discourse.julialang.org/u/merckxiaan)\
**Post date:** [July 30, 2019, 9:55am UTC](https://discourse.julialang.org/t/is-there-an-implementation-of-the-attention-mechanism-in-flux-jl/26959/3 "2019-07-30T09:55:39Z")

</div>

I’ve got a regular old seq2seq implementation [here](https://github.com/merckxiaan/flux-seq2seq).  
The problem is that it doesn’t work very well… So it might be risky to use it as a reference.  
If you do happen to spot a mistake I’d be happy to hear about it!

---

<div class="post-metadata">

**Author:** ![MacKenzieHnC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mackenziehnc/32/16429_2.png) [@MacKenzieHnC](https://discourse.julialang.org/u/MacKenzieHnC)\
**Post date:** [September 22, 2020, 9:32pm UTC](https://discourse.julialang.org/t/is-there-an-implementation-of-the-attention-mechanism-in-flux-jl/26959/4 "2020-09-22T21:32:36Z")

</div>

I’m digging through your code right now. Thank you for the work you put into this.

The only problem I’ve found so far is a typo on the ipynb: “Esentially, the encoder outputs and the hidden state of the decoder are used to a context vector”

Some word is missing between “used to” and “a” and I don’t know what it is.

---

<div class="post-metadata">

**Author:** ![merckxiaan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/merckxiaan/32/4697_2.png) [@merckxiaan](https://discourse.julialang.org/u/merckxiaan)\
**Post date:** [September 23, 2020, 6:29am UTC](https://discourse.julialang.org/t/is-there-an-implementation-of-the-attention-mechanism-in-flux-jl/26959/5 "2020-09-23T06:29:22Z")

</div>

Hey @MacKenzieHnC  
Thanks for taking a look!  
The sentence should have been this, I think:

> Esentially, the encoder outputs and the hidden state of the decoder are used to **create** a context vector

It’s been some time since I’ve worked on this, the Flux api has changed quite a lot (most notably the AD engine changed to Zygote).  
It definitely would be interesting to see this model implemented in the up-to-date version of Flux.  
I might do this when I get the time… but I’m notoriously slow.

---

<div class="post-metadata">

**Author:** ![Volker](https://avatars.discourse-cdn.com/v4/letter/v/77aa72/32.png) [@Volker](https://discourse.julialang.org/u/Volker)\
**Post date:** [September 23, 2020, 6:49am UTC](https://discourse.julialang.org/t/is-there-an-implementation-of-the-attention-mechanism-in-flux-jl/26959/6 "2020-09-23T06:49:40Z")

</div>

Maybe this links are interesting for you:

> **[GitHub - sdobber/FluxArchitectures.jl: Complex neural network examples for...](https://github.com/sdobber/FluxArchitectures.jl)**
>
> Complex neural network examples for Flux.jl. Contribute to sdobber/FluxArchitectures.jl development by creating an account on GitHub.

> **[FluxArchitectures: TPA-LSTM](https://sdobber.github.io/FA_TPALSTM/)**
>
> The next model in the FluxArchitectures repository is the Temporal Pattern Attention LSTM network based on the paper “Temporal Pattern Attention for Multivariate Time Series Forecasting” by Shih et. al.. It claims to have a better performance than...