Specifically, I’m interested in the attention mechanism described in Listen, Attend and Spell work (where it’s referred to as AttentionContext), but more generally any other kind of attention’s implementation would be quite useful to take a look at.

Is there an implementation of the attention mechanism in Flux.jl?

Volker September 23, 2020, 6:49am 6

Maybe this links are interesting for you:

2 Likes

Community Interest Check: LLMs from Scratch in Pure Julia

Topic		Replies	Views
Dense Layers in Attention Don't Train Machine Learning	1	83	July 20, 2024
Bug when training a custom model using Flux Machine Learning flux , training	2	342	February 18, 2023
Implementation of self-attention in Transformers.jl? Machine Learning	9	1111	February 21, 2023
Julia Implementation of Transformer Neural Network Model Machine Learning flux	3	1616	April 19, 2019
Problems regarding an implementation of a word2word Autoencoder Machine Learning question , flux	7	618	February 14, 2022