How to find gradients of weights and bias of multilayer neural network?

New to Julia
1

Hi,

I am trying to display gradients of weights and biases in each layer of a multi layer network as below:

f = rand(2,10)
y = rand(10)
dims = [2, 4, 2, 1]
layers = [Dense(dims[i], dims[i+1], Flux.relu) for i in 1:length(dims)-1];
m = Chain(layers...)
L(x, y) = Flux.Losses.mse(m(x), y) 
gs = gradient(() -> L(f, y), ps)

I got like below:
Grads(...)
But how to see the gradient of each layer?

For a single layer, I could see gradients as gs[W]… But this is not working multilayer… How to solve this?

Manu

2

You can just look at gs.grads, but I agree that the current default printing is a bit useless:

julia> gs.grads
IdDict{Any,Any} with 8 entries:
  Float32[0.751766 0.864498]            => [-0.0341256 0.0]
  :(Main.y)                             => [0.167831, 0.0865121, 0.165344, 0.0217242, 0.132331, 0.0278387, 0.020141…
  Float32[0.0, 0.0, 0.0, 0.0]           => AbstractFloat[0.0, 0.0, -0.0343127, -0.0879389]
  Float32[0.0]                          => [-0.94404]
  Float32[0.0, 0.0]                     => AbstractFloat[-0.709697, 0.0]
  Float32[-0.19732 -0.970261; -0.57558… => Any[0.0 0.0; 0.0 0.0; -0.00559779 -0.0127858; -0.0364102 -0.0305574]
  Float32[-0.227719 -0.539766 0.158784… => Any[0.0 0.0 -0.00993986 -0.194303; 0.0 0.0 0.0 0.0]
  :(Main.f)                             => [-0.00434088 -0.00453109 … -0.000261 -0.00431912; -0.00181489 -0.0018944…
1 Like
3

@simeonschaub : is it possible to extract gradients of each layer as an array??

4

You can index the gradients using the parameters. If you have a layer, Flux.params should give you all parameters of that layer.

1 Like
5

Thank you @simeonschaub