# Simple NLP in Flux with Embedding Layer

**URL:** <https://discourse.julialang.org/t/simple-nlp-in-flux-with-embedding-layer/27870>\
**Category:** New to Julia\
**Tags:** flux\
**Created:** [August 23, 2019, 12:32am UTC](https://discourse.julialang.org/t/simple-nlp-in-flux-with-embedding-layer/27870 "2019-08-23T00:32:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nigel\_Adams](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nigel_adams/32/9273_2.png) [@Nigel\_Adams](https://discourse.julialang.org/u/Nigel_Adams)\
**Post date:** [August 23, 2019, 12:32am UTC](https://discourse.julialang.org/t/simple-nlp-in-flux-with-embedding-layer/27870/1 "2019-08-23T00:32:40Z")

</div>

I’m trying to learn the use of embedding layers by the way of 10 string documents labelled as positive or negative.

```julia
StringDocs = ["well done", "good work", "great effort", "nice work", "excellent", "weak", "poor effort", "not good", "poor work", "could have done better"]
y = [1,1,1,1,1,0,0,0,0,0]
pad_size=4
N = 10

```

Each word in the document (vocab) is represented by an integer and after some code to prepare the training data x becomes.

(Note it is transposed as an ‘input’ x N matrix for Flux).

```julia
x = [ 0 0 0 0 0 0 0 0 0 2 ;
       0 0 0 0 0 0 0 0 0 8 ;
      13 6 7 9 0 0 11 10 11 3 ;
       3 14 4 14 5 12 4 6 14 1 ]

```

The rest of the code is below. I was hoping the embedding layer (W) would change and learn after every epoch but it’s not changing? I’ve been stuck on this for a while and need a few pointers. Would be grateful for any pointers or examples that may help.

```julia
data = [(x, y)]

W = param(Flux.glorot_normal(8, 51))
max_features, vocab_size = size(W)

one_hot_matrix=Flux.onehotbatch(reshape(x, pad_size*N), 0:vocab_size-1)

m = Chain(x -> W * one_hot_matrix,
          x -> reshape(x, max_features, pad_size, N),
          x -> mean(x, dims=2),
          x -> reshape(x, 8, 10),
          Dense(8, 1),
)

# if I add softmax above the loss doesn't change

loss(x, y) = Flux.mse(m(x), y)
optimizer = Flux.Descent(0.001)

for epoch in 1:10
    Flux.train!(loss, Flux.params(m), data, optimizer)
    println("loss=",loss(x, y).data)
end
show(Flux.params(m))

```

The output is : -

```julia
loss=3.7767549
loss=3.7263222
loss=3.6780336
loss=3.6317985
loss=3.5875306
loss=3.5451438
loss=3.5045598
loss=3.4657013
loss=3.4284942
loss=3.39287
Params([Float32[0.419828 -0.139223 -0.225595 0.0708142 0.232704 0.0907047 0.707192 -0.167613] (tracked), Float32[0.144213] (tracked)])

```

I was expecting to also see 8x51 params for the embedding layer.  
I could be close or I could be way off but I can’t find examples close enough to help me progress.

---

<div class="post-metadata">

**Author:** ![tanhevg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tanhevg/32/12025_2.png) [@tanhevg](https://discourse.julialang.org/u/tanhevg)\
**Post date:** [August 23, 2019, 9:58am UTC](https://discourse.julialang.org/t/simple-nlp-in-flux-with-embedding-layer/27870/2 "2019-08-23T09:58:46Z")

</div>

`params` in Flux works only with structures that have been made `@treelike`; this is why it is not catching the embedding layer params. To make it work you need to change it to something like this:

```julia
struct EmbeddingLayer
   W
   EmbeddingLayer(mf, vs) = new(param(Flux.glorot_normal(mf, vs)))
end
@Flux.treelike EmbeddingLayer
(m::EmbeddingLayer)(x) = m.W * Flux.onehotbatch(reshape(x, pad_size*N), 0:vocab_size-1)

m = Chain(EmbeddingLayer(max_features, vocab_size),
          x -> reshape(x, max_features, pad_size, N),
          x -> mean(x, dims=2),
          x -> reshape(x, 8, 10),
          Dense(8, 1),
)

```

The rest of your snippet should stay the same.

Hope this helps.

---

<div class="post-metadata">

**Author:** ![Nigel\_Adams](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nigel_adams/32/9273_2.png) [@Nigel\_Adams](https://discourse.julialang.org/u/Nigel_Adams)\
**Post date:** [August 25, 2019, 3:28am UTC](https://discourse.julialang.org/t/simple-nlp-in-flux-with-embedding-layer/27870/3 "2019-08-25T03:28:21Z")

</div>

Thanks so much @tanhevg I got it going with your help 😀!!

I had made a couple of other rookie errors which I corrected in this blog post.

> **[Julia Word Embedding Layer in Flux - Self Trained](https://spcman.github.io/getting-to-know-julia/deep-learning/nlp/flux-embeddings-tutorial-1/)**
>
> A simple example of a word embedding layer with Flux (not pre-trained)

If you stumble on this page grappling with something similar have a read.