Specifically, I’m interested in the attention mechanism described in Listen, Attend and Spell work (where it’s referred to as AttentionContext), but more generally any other kind of attention’s implementation would be quite useful to take a look at.

Is there an implementation of the attention mechanism in Flux.jl?

Volker September 23, 2020, 6:49am 6

Maybe this links are interesting for you:

2 Likes

Community Interest Check: LLMs from Scratch in Pure Julia

Topic		Replies	Views
Flux seq2seq Machine Learning question	12	3755	January 5, 2019
Dense Layers in Attention Don't Train Machine Learning	1	93	July 20, 2024
Flux Transformer Out of Memory Machine Learning	25	1514	March 13, 2023
Julia Implementation of Transformer Neural Network Model Machine Learning flux	3	1636	April 19, 2019
[ANN] Transformers.jl Package Announcements announcement	6	1948	February 18, 2020