# Making Word Embedding Layer in Julia

**URL:** <https://discourse.julialang.org/t/making-word-embedding-layer-in-julia/82829>\
**Category:** General Usage\
**Tags:** flux, nlp\
**Created:** [June 15, 2022, 6:31pm UTC](https://discourse.julialang.org/t/making-word-embedding-layer-in-julia/82829 "2022-06-15T18:31:52Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jack\_N](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jack_n/32/37265_2.png) [@Jack\_N](https://discourse.julialang.org/u/Jack_N)\
**Post date:** [June 15, 2022, 6:31pm UTC](https://discourse.julialang.org/t/making-word-embedding-layer-in-julia/82829/1 "2022-06-15T18:31:52Z")

</div>

I am trying to make a word embedding layer in a flux model in Julia and I am using this tutorial as a reference [https://spcman.github.io/getting-to-know-julia/deep-learning/nlp/flux-embeddings-tutorial-1/](https://spcman.github.io/getting-to-know-julia/deep-learning/nlp/flux-embeddings-tutorial-1).

I believe it is a bit outdated but I was unable to find any other tutorials for making a word embedding layer in Flux. The line that was giving me trouble is ` (m::EmbeddingLayer)(x) = m.W * Flux.onehotbatch(reshape(x, pad_size*N), 0:vocab_size-1)`. Every time it runs I get the error `ERROR: MethodError: no method matching *(::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}}, ::Flux.OneHotArray{UInt32, 20, 1, 2, Vector{UInt32}})`.

Please let me know how I can fix this error or if there is a better tutorial or method that I should follow to create a word embedding layer in Julia.

Thanks,

Jack

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [June 15, 2022, 7:31pm UTC](https://discourse.julialang.org/t/making-word-embedding-layer-in-julia/82829/2 "2022-06-15T19:31:56Z")

</div>

I would try to mark as non-differentiable. It used to be `Zygote.@ignore`, but now this should be done with ChainRules and I am not super profficient in their syntax (despite writing them).

Using `Zygote.@ignore`, I would do

```julia
using Zygote
(m::EmbeddingLayer)(x) = m.W * Zygote.@ignore Flux.onehotbatch(x[:], 0:vocab_size-1)

```

And this would work.

Looking at ` 0:vocab_size-1`, do not forget that Julia’s indexes starts with 1.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [June 15, 2022, 8:17pm UTC](https://discourse.julialang.org/t/making-word-embedding-layer-in-julia/82829/3 "2022-06-15T20:17:23Z")

</div>

There’s a built-in `Embedding` layer in Flux and an updated implementation at [https://github.com/FluxML/Flux.jl/pull/1656](https://github.com/FluxML/Flux.jl/pull/1656). You can use directly or as a reference for tutorial purposes.