Making Word Embedding Layer in Julia

General Usage
,
1

I am trying to make a word embedding layer in a flux model in Julia and I am using this tutorial as a reference https://spcman.github.io/getting-to-know-julia/deep-learning/nlp/flux-embeddings-tutorial-1/.

I believe it is a bit outdated but I was unable to find any other tutorials for making a word embedding layer in Flux. The line that was giving me trouble is (m::EmbeddingLayer)(x) = m.W * Flux.onehotbatch(reshape(x, pad_size*N), 0:vocab_size-1). Every time it runs I get the error ERROR: MethodError: no method matching *(::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}}, ::Flux.OneHotArray{UInt32, 20, 1, 2, Vector{UInt32}}).

Please let me know how I can fix this error or if there is a better tutorial or method that I should follow to create a word embedding layer in Julia.

Thanks,

Jack

2

I would try to mark as non-differentiable. It used to be Zygote.@ignore, but now this should be done with ChainRules and I am not super profficient in their syntax (despite writing them).

Using Zygote.@ignore, I would do

using Zygote
(m::EmbeddingLayer)(x) = m.W * Zygote.@ignore Flux.onehotbatch(x[:], 0:vocab_size-1)

And this would work.

Looking at 0:vocab_size-1, do not forget that Julia’s indexes starts with 1.

3

There’s a built-in Embedding layer in Flux and an updated implementation at https://github.com/FluxML/Flux.jl/pull/1656. You can use directly or as a reference for tutorial purposes.