Hi there
I have seen the examples but I am still unclear as to how to apply per-layer regularisation, the examples seem to show a single layer network or applying regularisation to all layers.
Coming from TensorFlow and Python, Julia and Flux are an incredible combination, allowing for me to translate ideas to code with absolute clarity without superfluous heavy syntax. Very impressed.
Thanks!
The examples use params to acquire the implicit parameters of a function. For example, the multilayer example (Regularisation · Flux) uses
sum(norm, params(m))
where m is the full model. One of the cool parts about Flux is that you can index a chain of layers. So m[1] corresponds to the first layer, m[2] the second, and m[end] the last. See here: Model Reference · Flux
This means that if I want the implicit parameters of for just layer 1, then I can use params(m[1]). So a norm penalty for just the first layer’s parameters could be
sum(norm, params(m[1]))
I am not sure what your exact regularizer looks like, but I think you can extend above to get what you want. Remember you can write a function for the regularizer that uses a for-loop or map to apply a per-layer penalty function. And you can use multiple dispatch to apply penalties to only certain types of layers (which may be necessary depending on the regularizer you want). Just reply if you have any more questions!