How are called "transversal" layers in NN?

Domains Machine Learning
#1

Hello, how is it known the concept in a neural network of a “transversal” or “cross section” layer ? I can’t find anything relevant with these two words…
Its characteristic is that its activation function takes as input all the output from the previous layer and its output dimension size is given by the dimension in output of its activation function.

I implemented a generic one in my own NN Library, to use it for classification (with the softmax activation function) or for pooling the neurons of the previous layer (max, avg,…).

However I can’t find how such kind of layer is known in the literature on in standard NN libraries as Flux or KNET…

#2

What is the name of your layer in your library?

I’ll try to guess.

  1. Inception or one derived from it?
  2. The layer used in DenseNet?

Maybe one of its elements is called in other libraries, like keras, Concatenate layer

#3

I did call it “VectorFuncionLayer”
Not sure… for what I understood that kind of layer type merges different layers, like two separate nn merging to one, while mine remains on the simple one chain model.
Other diff is that that layer works on n-dimensional tensors, while I remain on 1d.
Finally, in kera each different merging operation is a different kind of layers, while here in VectorFunctionLayer the layer can be associated with any R^N → R^M activation function, like softmax or pool1d.

1 Like
#4

You don’t want to act on multiple layers, but just want to apply a vector function?
Or am I making a mistake and misunderstanding?

In Flux you can simpliy do this (and those layers do not have a specific name):

model = Chain(
    Dense(28^2, 200), my_vector_function, 
    Dense(200, 100), my_vector_function,    # change 200 to the output size of my_vector_function
    Dense(100, 10), softmax
)

It seems simple, so I can’t tell if I’m missing something in the translation from English. In case I am making a mistake I apologize.

1 Like
#5

Oh, I see… basically in Flux you can chain “Layer” objects (like Dense(200,100) with directly functions “alone” (and perhaps other kind of stuff).
In BetaML I can chain only “Layers” and hence I introduced a “weightless” layer to “support” the function, at the end it is the same…

Thank you
/Antonello

#6

Yes, supporting plain old julia functions as layers is an explicit design goal of Flux. Layer structs should only be required when one wants to keep track of internal parameters/state that should have a gradient.

1 Like