# Gradient of NN not changing with different inputs

**URL:** <https://discourse.julialang.org/t/gradient-of-nn-not-changing-with-different-inputs/56435>\
**Category:** Numerics\
**Tags:** differentiation, machine-learning\
**Created:** [March 3, 2021, 7:27pm UTC](https://discourse.julialang.org/t/gradient-of-nn-not-changing-with-different-inputs/56435 "2021-03-03T19:27:34Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![edwinb-ai](https://avatars.discourse-cdn.com/v4/letter/e/bb73d2/32.png) [@edwinb-ai](https://discourse.julialang.org/u/edwinb-ai)\
**Post date:** [March 3, 2021, 7:27pm UTC](https://discourse.julialang.org/t/gradient-of-nn-not-changing-with-different-inputs/56435/1 "2021-03-03T19:27:34Z")

</div>

While playing around with some neural networks (NN) I found out this

```julia
using Flux
using Random
using Zygote
using ForwardDiff

Random.seed!(9120)

n = 1
m = 5
hidden = 10

x, y = rand(m), rand(n) # some data
model = Flux.Chain(Flux.Dense(m, hidden), Flux.Dense(hidden, n))

# Gradient with ForwardDiff
g = z -> ForwardDiff.gradient(w -> model(w)[1], z)
# Getting the weights of the model as an array
ps, re = Flux.destructure(model)

display(g(x)) # Checking with original data
display(g(ps[1:m])) # Checking with weights

# Gradient with Zygote
gs = Zygote.gradient(w -> model(w)[1], rand(m)) # Notice a different random vector
display(gs)
gs = Zygote.gradient(w -> model(w)[1], zeros(m)) # Now with zeros
display(gs)

```

Now, the result is always the same

```julia
5-element Array{Float64,1}:
 -0.7097914769304869
 -0.14294694831147323
 -0.04312831631528913
  0.2866390831645096
 -0.4046597463981584
5-element Array{Float32,1}:
 -0.7097915
 -0.14294694
 -0.043128345
  0.2866391
 -0.40465972
(Float32[-0.7097915, -0.14294693, -0.04312831, 0.28663906, -0.40465975],)
(Float32[-0.7097915, -0.14294693, -0.04312831, 0.28663906, -0.40465975],)

```

Is there a reason why this is the case? I should be inclined to believe that this is because  
the input is not being evaluated at all.  
Does this mean that the gradient taken is with respect to the _weights_ of the model?

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [March 3, 2021, 8:08pm UTC](https://discourse.julialang.org/t/gradient-of-nn-not-changing-with-different-inputs/56435/2 "2021-03-03T20:08:48Z")

</div>

Your model is linear.  
A linear model has the same gradient everywhere.

To make a nonlinear model you need to pass an activation function into `Dense` as the 3rd arg.  
Otherwise it defaults to `identity` and thus you get a linear model.

[https://fluxml.ai/Flux.jl/stable/models/layers/#Flux.Dense](https://fluxml.ai/Flux.jl/stable/models/layers/#Flux.Dense)

---

<div class="post-metadata">

**Author:** ![edwinb-ai](https://avatars.discourse-cdn.com/v4/letter/e/bb73d2/32.png) [@edwinb-ai](https://discourse.julialang.org/u/edwinb-ai)\
**Post date:** [March 3, 2021, 8:12pm UTC](https://discourse.julialang.org/t/gradient-of-nn-not-changing-with-different-inputs/56435/3 "2021-03-03T20:12:28Z")

</div>

That’s it. How could I have missed that? Thanks a lot!