# Slope at a point of RNN using Flux
**URL:** https://discourse.julialang.org/t/slope-at-a-point-of-rnn-using-flux/84207
**Category:** Machine Learning
**Tags:** flux, zygote
**Created:** [July 14, 2022, 6:14am UTC](https://discourse.julialang.org/t/slope-at-a-point-of-rnn-using-flux/84207 "2022-07-14T06:14:30Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@FrootLoops](https://discourse.julialang.org/u/FrootLoops)
#### Post date: [July 14, 2022, 6:14am UTC](https://discourse.julialang.org/t/slope-at-a-point-of-rnn-using-flux/84207/1 "2022-07-14T06:14:30Z")
I am using a recurrent neural network for data of the form (x\_t, y\_t)\_{t=1}^T. My RNN has therefore the form
y\_t^{nn} = NN(x\_{t-1}, x\_t, y\_{t-1}),
with 1 hidden layer, 20 neurons, a sigmoid activation function and a linear output layer i.e.
`model = Chain(RNN(3 => 10, sigmoid), Dense(10 => 1, identity)) `
After 100 epochs the RNN shows good convergence and I receive good results (see Fig).
[](https://i.stack.imgur.com/EY24C.png)
Now I am interested in calculating the slope at a specific t for example (x\_{5}, y\_{5}). My idea was to calculate the Jacobian leading to \frac{\partial NN}{\partial x\_{t-1}}, \frac{\partial NN}{\partial x\_t} and \frac{\partial NN}{\partial y\_{t-1}}. The searched slope should then be equal to \frac{\partial NN}{\partial x\_t}.
```
idx = 5
tangent_nn = Flux.jacobian(model, X[idx]) #= (Float32[0.04896392 -0.046510044 0.9080559])
tangent_exact = (Y[idx] - X[idx][3])/(X[idx][2]-X[idx][1]) #= 7.666664985015
```
One can see that none of the three entries of the jacobian is equal to the “exact” tangent. Can someone explain me what I am missing?
---
### Author:  [@skleinbo](https://discourse.julialang.org/u/skleinbo)
#### Post date: [July 18, 2022, 8:47am UTC](https://discourse.julialang.org/t/slope-at-a-point-of-rnn-using-flux/84207/2 "2022-07-18T08:47:04Z")
Let’s focus on the recurrent unit first. Call it f. It is a function of two variables, the current element of the input sequence x and its internal/hidden state h: y = f(x, h).
Taking the derivative yields
dy = \partial\_x f\,dx + \partial\_h f\,dh
Your reasoning focused on the first term only and neglected the change in hidden state.
Because the output becomes the hidden state in the next iteration, going through the sequence gives the recursive relation y\_k = f(x\_k, y\_{k-1}) where y\_0 is a learned parameter.
Point is, the slope of the graph plotted above is not the change of y\_k with x\_k but with k. Imagine a continuous version of the map where x(t) and y(t) are parameterized by a real number instead of an integer. Then one could write for some \lambda
y(t) = f(x(t), y(t-\lambda)) You are after dy/dt which involves varying both arguments.
By the recursion relation
\frac{dy\_k}{dt} = \partial\_x f(x\_k, y\_{k-1})\,\underbrace{\dot{x}(t)}\_{=1} + \partial\_h f(x\_k, y\_{k-1})\,\frac{dy\_{k-1}}{dt}
This is again a recursion and thus one needs to accumulate derivatives along the sequence of inputs and hidden states to calculate the sought-after slope.
Lastly, the result is to be multiplied by the Jacobian of the dense layer.
I’ve tried it on a similar network to yours but with one input dimension instead of three ([Derivative of RNN approximating piecewise linear function · GitHub](https://gist.github.com/skleinbo/ab822198e0ae0898447932fbba7d4a2e)), learning a piecewise linear function with slopes 7 and 2.
Going through the motions, I end up with the following comparison between the derivative and finite difference approach.
One could certainly try to train the model better, but I guess the ballpark is alright.