Specifically, I’m interested in the attention mechanism described in Listen, Attend and Spell work (where it’s referred to as AttentionContext), but more generally any other kind of attention’s implementation would be quite useful to take a look at.

Is there an implementation of the attention mechanism in Flux.jl?

Volker September 23, 2020, 6:49am 6

Maybe this links are interesting for you:

2 Likes

Community Interest Check: LLMs from Scratch in Pure Julia

Topic		Replies	Views
Flux seq2seq Machine Learning question	12	3751	January 5, 2019
Dense Layers in Attention Don't Train Machine Learning	1	91	July 20, 2024
Flux Transformer Out of Memory Machine Learning	25	1507	March 13, 2023
Julia Implementation of Transformer Neural Network Model Machine Learning flux	3	1636	April 19, 2019
[ANN] Transformers.jl Package Announcements announcement	6	1947	February 18, 2020