Specifically, I’m interested in the attention mechanism described in Listen, Attend and Spell work (where it’s referred to as AttentionContext), but more generally any other kind of attention’s implementation would be quite useful to take a look at.

Is there an implementation of the attention mechanism in Flux.jl?

Volker September 23, 2020, 6:49am 6

Maybe this links are interesting for you:

2 Likes

Community Interest Check: LLMs from Scratch in Pure Julia

Topic		Replies	Views
Flux seq2seq Machine Learning question	12	3757	January 5, 2019
Julia Implementation of Transformer Neural Network Model Machine Learning flux	3	1642	April 19, 2019
[ANN] Transformers.jl Package Announcements announcement	6	1953	February 18, 2020
Flux: Machine Learning with Julia Machine Learning package , announcement	8	7876	March 3, 2017
Flux.jl RNN performance Machine Learning	11	3129	October 28, 2018