# Gradient of NN not changing with different inputs

While playing around with some neural networks (NN) I found out this

``````using Flux
using Random
using Zygote
using ForwardDiff

Random.seed!(9120)

n = 1
m = 5
hidden = 10

x, y = rand(m), rand(n) # some data
model = Flux.Chain(Flux.Dense(m, hidden), Flux.Dense(hidden, n))

g = z -> ForwardDiff.gradient(w -> model(w), z)
# Getting the weights of the model as an array
ps, re = Flux.destructure(model)

display(g(x)) # Checking with original data
display(g(ps[1:m])) # Checking with weights

gs = Zygote.gradient(w -> model(w), rand(m)) # Notice a different random vector
display(gs)
gs = Zygote.gradient(w -> model(w), zeros(m)) # Now with zeros
display(gs)
``````

Now, the result is always the same

``````5-element Array{Float64,1}:
-0.7097914769304869
-0.14294694831147323
-0.04312831631528913
0.2866390831645096
-0.4046597463981584
5-element Array{Float32,1}:
-0.7097915
-0.14294694
-0.043128345
0.2866391
-0.40465972
(Float32[-0.7097915, -0.14294693, -0.04312831, 0.28663906, -0.40465975],)
(Float32[-0.7097915, -0.14294693, -0.04312831, 0.28663906, -0.40465975],)
``````

Is there a reason why this is the case? I should be inclined to believe that this is because
the input is not being evaluated at all.
Does this mean that the gradient taken is with respect to the weights of the model?

1 Like

To make a nonlinear model you need to pass an activation function into `Dense` as the 3rd arg.
Otherwise it defaults to `identity` and thus you get a linear model.