Julia/Flux creating a model correctly - using Chain Embedding layer reshaping & Dense layers

Specific Domains Machine Learning
, ,
1

I am trying to follow an old article about word embedding in Julia language with Flux: Julia Word Embedding Layer in Flux - Self Trained - Getting to Know Julia

As it is an old outdated article I managed to handle the embedding layer correctly using the new embedding layer of Flux (0.13.11). The Chain looks like this:

embedding = Flux.Embedding(vocab_size => max_features, init=Flux.glorot_normal)
model = Chain(embedding(Flux.onehotbatch(reshape(x, pad_size*N), 0:vocab_size-1)),
           x -> reshape(x, max_features, pad_size, N),
           x -> mean(x, dims=2),
           x -> reshape(x, 8, 10),
           Dense(8, 1),
)

When I try to check the model my calling model(x) I get: ERROR: MethodError: objects of type Matrix{Float32} are not callable Use square brackets for indexing an Array.

My guess that the problem somehow connected to the reshaping. I am new to both Julia & Flux so sorry if the question is basic.

Any clue what should I do?

2

This is one of those cases where looking at the full stacktrace and not just the error message helps. embedding is a callable struct which represents an embedding layer. Instead of passing this layer object to Chain like the tutorial does, you’re passing the output from calling it on some arbitrary input. Said output is a Matrix, so when Chain goes to call what it thinks is an Embedding layer it ends up invoking output_matrix_from_embedding(x) and predictably fails.

If you’re familiar with how layers work in other ML frameworks, Flux layers work much the same.

3

Ok - As I am new to both Julia, Flux & Deep Learning I understand that something basic was wrong.
Should I just input the transposed document term index as input to the embedding layer? - how will it be hot encoded? I am clearly missing something…

4

Very little has to change. Just make sure what you’re passing to Chain are actual layers:

model = Chain(
  embedding,
  x -> reshape(x, max_features, pad_size, N),
  x -> mean(x, dims=2),
  x -> reshape(x, 8, 10),
  Dense(8, 1),
)

And then you can call your model with whatever you want:

x = Flux.onehotbatch(reshape(x, pad_size*N), 0:vocab_size-1)
model(x)

This I believe should match the behaviour in the linked article.