### Author:  [@Mason](https://discourse.julialang.org/u/Mason)
#### Post date: [December 13, 2023, 1:58pm UTC](https://discourse.julialang.org/t/accurate-derivative-for-tanh/107568/4 "2023-12-13T13:58:00Z")
It’s not relevant for most applications of DiffRules, but it definitely is relevant to how it’s computed in ChainRules.jl since ChainRules does explicitly re-use the answer computed for the forwards pass:
>
Perhaps what we should do here is add a branch like
```julia
abs(real(x)) <= 14 ? (1 - Ω^2) : sech(x)^2
```
And then maybe remove that branch only for the `@fastmath` version?
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [December 13, 2023, 2:01pm UTC](https://discourse.julialang.org/t/accurate-derivative-for-tanh/107568/5 "2023-12-13T14:01:55Z")
> [@Tamas\_Papp](#):
>
> but it was changed for CSE.
With your new formulation you’ll lose any CSE too, so you might as well go back to using `sech`.
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [December 13, 2023, 2:09pm UTC](https://discourse.julialang.org/t/accurate-derivative-for-tanh/107568/6 "2023-12-13T14:09:25Z")
> [@Mason](#):
>
> `abs(real(x)) <= 14 ? (1 - Ω^2) : sech(x)^2`
That’s way too aggressive. By `x = 4` the `1 - tanh(x)^2` formula has already lost 2 digits, by `x = 6` it has lost 4 digits, and by `x = 14` it has lost 10 digits.
Doing a few quick experiments, the crossover point where `sech(x)^2` becomes more accurate seems to be around `x = 1`, so you could do `abs(real(x)) <= 1 ? (1 - Ω^2) : sech(x)^2` (or `real(x)^2 <= 1` … not sure which is faster).
---
### Author:  [@Mason](https://discourse.julialang.org/u/Mason)
#### Post date: [December 13, 2023, 2:14pm UTC](https://discourse.julialang.org/t/accurate-derivative-for-tanh/107568/7 "2023-12-13T14:14:53Z")
> [@stevengj](#):
>
> That’s way too aggressive.
Yeah, it was just an example value taken from Tamas’ table.
> [@stevengj](#):
>
> Doing a few quick experiments, the crossover point where `sech(x)^2` becomes more accurate seems to be around `x = 1`,
I don’t think accuracy is the only concern here though, since AD people are typically also very performance sensitive. My assumption is that so long as values can be re-used from the primal calculation without truly catastrophic losses of performance, that’s typically going to be preferred.
---
### Author:  [@devmotion](https://discourse.julialang.org/u/devmotion)
#### Post date: [December 13, 2023, 2:16pm UTC](https://discourse.julialang.org/t/accurate-derivative-for-tanh/107568/8 "2023-12-13T14:16:28Z")
> [@Mason](#):
>
> It’s not relevant for most applications of DiffRules
ForwardDiff applies CSE to the rules generated from the DiffRules expressions.
> [@Mason](#):
>
> it definitely is relevant to how it’s computed in [ChainRules.jl](https://juliahub.com/ui/Packages/ChainRules).
Yes, the issue also affects ChainRules: [rule for tanh has catastrophic cancellation for |x| \> 20 · Issue #102 · JuliaDiff/DiffRules.jl · GitHub](https://github.com/JuliaDiff/DiffRules.jl/issues/102#issuecomment-1853967962)
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [December 13, 2023, 2:18pm UTC](https://discourse.julialang.org/t/accurate-derivative-for-tanh/107568/9 "2023-12-13T14:18:49Z")
> [@Mason](#):
>
> I don’t think accuracy is the only concern here though, since AD people are typically also very performance sensitive. My assumption is that so long as values can be re-used from the primal calculation without truly catastrophic losses of performance, that’s typically going to be preferred.
I disagree. _Some_ applications of AD might need only low accuracy, but in general when a standard library computes a math function the programmer should generically expect it to be computed to close to machine precision.
If they programmer wants lower accuracy they should use lower precision, or a different function name `tanh_approx` (or some macro `@approx` that rewrites `tanh` to `tanh_approx` etc.).
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [December 13, 2023, 2:36pm UTC](https://discourse.julialang.org/t/accurate-derivative-for-tanh/107568/10 "2023-12-13T14:36:17Z")
Yes, debugging speed issues is much easier than debugging numerical issues. Whenever there is a trade-off, inaccurate but fast should be opt-in.
I should have explanded, but I thought we could keep CSE by also calculating `tanh` from `exp(2*x)` etc. But maybe I misunderstand how that works, or how relevant CSE is.
---
### Author:  [@devmotion](https://discourse.julialang.org/u/devmotion)
#### Post date: [December 13, 2023, 2:37pm UTC](https://discourse.julialang.org/t/accurate-derivative-for-tanh/107568/11 "2023-12-13T14:37:12Z")
I think the simplest solution seems to be to define the derivative as `sech(x)^2` without reusing the primal. IMO, as @stevengj suggested, if users or a package want a faster and more inaccurate version, they should define and use a `tanh_approx` function with a less accurate derivate.
Branches discussed here would be optimized for `Float64` but users would still run into the same problems with e.g. `Float32` (or they might be inefficient for something like `Float128`).
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [December 13, 2023, 2:38pm UTC](https://discourse.julialang.org/t/accurate-derivative-for-tanh/107568/12 "2023-12-13T14:38:18Z")
> [@devmotion](#):
>
> Branches discussed here would be optimized for `Float64` but users would still run into the same problems with e.g. `Float32` (or they might be inefficient for something like `Float128`).
No, in this case my suggested crossover point `abs(real(x)) < 1` should be independent of precision (at least for real `x` … I haven’t thought too much about the complex case), because that’s where `tanh(x)^2` is around 0.5 or less.
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [December 13, 2023, 2:43pm UTC](https://discourse.julialang.org/t/accurate-derivative-for-tanh/107568/13 "2023-12-13T14:43:34Z")
> [@Tamas\_Papp](#):
>
> I should have explanded, but I thought we could keep CSE by also calculating `tanh` from `exp(2*x)` etc.
`tanh(x)` is [computed from polynomial approximations](https://github.com/JuliaLang/julia/blob/15244663a345a98e2919837fbd6d9e0ad997d5ba/base/special/hyperbolic.jl#L127-L159) for many values of `x`. Worse, as far as I can tell, `tanh` is not inlined by the compiler, which will prevent the compiler from CSE-ing expressions computed inside `tanh` with expressions outside `tanh`:
```julia
julia> f(x) = 1+tanh(x)
f (generic function with 1 method)
julia> @code_llvm f(0.2)
; @ REPL[42]:1 within `f`
define double @julia_f_445(double %0) #0 {
top:
%1 = call double @j_tanh_447(double %0) #0
; ┌ @ promotion.jl:410 within `+` @ float.jl:408
%2 = fadd double %1, 1.000000e+00
; └
ret double %2
}
```
---
### Author:  [@devmotion](https://discourse.julialang.org/u/devmotion)
#### Post date: [December 13, 2023, 2:43pm UTC](https://discourse.julialang.org/t/accurate-derivative-for-tanh/107568/14 "2023-12-13T14:43:45Z")
True, I was thinking about the suggestion of a cutoff at 14. I’m not sure how much we would gain from a branch using `abs(real(x)) <= 1` in practice though.