# AD in ill-conditioned z/√(s²+z²) without custom rules

**URL:** <https://discourse.julialang.org/t/ad-in-ill-conditioned-z-s-z-without-custom-rules/140047>\
**Category:** Numerics\
**Tags:** question, numerics, ad\
**Created:** [October 16, 2026, 5:37am UTC](https://discourse.julialang.org/t/ad-in-ill-conditioned-z-s-z-without-custom-rules/140047 "2026-10-16T05:37:08Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [October 16, 2026, 5:37am UTC](https://discourse.julialang.org/t/ad-in-ill-conditioned-z-s-z-without-custom-rules/140047/1 "2026-10-16T05:37:08Z")

</div>

I would like to implement a `z/sqrt(s^2+z^2)` that

1. works for when `z` and `s` have wildly different magnitudes, but are both finite,
2. also gives the correct result when AD’s via the most common frameworks.

I would prefer to **avoid writing custom rules if possible**. This part makes the exercise tricky.

Here is an MWE of failures using Enzyme as a demo because it traps `NaN`s. I would appreciate suggestions on how to do this. Or maybe someone convincing me that this is not possible without custom rules.

```julia
using Enzyme, Test

Enzyme.Compiler.CheckNan[] = true # error for NaNs

function _z_over_hypot(z, s)
    r = z / s
    if abs(r) ≤ 1
        r / sqrt(one(r) + r^2)
    else
        u = inv(r)
        sign(r) / sqrt(one(u) + u^2)
    end
end

@testset "z over hypot torture test" begin
    # V = [1.0, 2.0, 3.0] # for simple tests
    V = [1e-180, -1e-180, 0.5e-100, 1e-100, 2e-100, 1e-50, 1e100, 1e200]
    for s in V
        for z in V
            for dz in V
                @show s, z, dz
                dx, x = Enzyme.autodiff(Enzyme.ForwardWithPrimal, z -> SK._z_over_hypot(z, s),
                                        Enzyme.Duplicated(z, dz))
                _z = BigFloat(z)
                _s = BigFloat(s)
                _r = _z / _s
                _q = 1 + _r^2
                _x = z / √(z^2 + s^2)
                _dx = 1 / _s / (_q * √_q)
                @test x ≈ Float64(_x)
                @test dx ≈ Float64(_dx) * dz
            end
        end
    end
end

```

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [October 16, 2026, 5:55am UTC](https://discourse.julialang.org/t/ad-in-ill-conditioned-z-s-z-without-custom-rules/140047/2 "2026-10-16T05:55:51Z")

</div>

What’s wrong with the naive `_z_over_hypot(z, s) = z / hypot(z,s)`?

I haven’t tried Enzyme, but a simplified version of your test that simply checks whether it matches the `BigFloat` result passes:

```julia-auto
julia> @testset "z over hypot torture test" begin
           # V = [1.0, 2.0, 3.0] # for simple tests
           V = [1e-180, -1e-180, 0.5e-100, 1e-100, 2e-100, 1e-50, 1e100, 1e200]
           for s in V
               for z in V
                   @test _z_over_hypot(z,s) ≈ Float64(_z_over_hypot(big(z),big(s)))
               end
           end
       end
Test Summary: | Pass Total Time
z over hypot torture test | 64 64 0.0s

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [October 16, 2026, 6:20am UTC](https://discourse.julialang.org/t/ad-in-ill-conditioned-z-s-z-without-custom-rules/140047/3 "2026-10-16T06:20:55Z")

</div>

It fails for AD. That’s why Enzyme is in the tests MWE.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [October 16, 2026, 6:55am UTC](https://discourse.julialang.org/t/ad-in-ill-conditioned-z-s-z-without-custom-rules/140047/4 "2026-10-16T06:55:33Z")

</div>

I see. I guess the basic problem here is that the naive quotient rule for \partial/\partial z of this function (which is what AD will do by default, probably) gives:

\frac{\partial}{\partial z} \frac{z}{\operatorname{hypot}(z,s)} = \frac{\operatorname{hypot}(z,s) - \frac{z^2}{\operatorname{hypot}(z,s)}}{\operatorname{hypot}(z,s)^2} = \cdots = \frac{s^2}{\operatorname{hypot}(z,s)^{3}}

which suffers catastrophic cancellation in the naive numerator for |z| \gg |s| unless you analytically simplify it, which AD won’t do (unless you re-arrange the input expression). And even the simplified expression with s^2 in the numerator, which has no catastrophic cancellation, is susceptible to spurious over/underflow if both s and z are very large or small (that is, the numerator and denominator can separately under/overflow, even if the final answer would be in the representable range).

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [October 16, 2026, 7:07am UTC](https://discourse.julialang.org/t/ad-in-ill-conditioned-z-s-z-without-custom-rules/140047/5 "2026-10-16T07:07:15Z")

</div>

> [@stevengj](#):
>
> And even the simplified expression with s^2 𝑠2 in the numerator, which has no catastrophic cancellation, is susceptible to spurious over/underflow if both s 𝑠 and z 𝑧 are very large or small.

I suspect that the overflow/underflow issue, in particular, will be hard to solve without a custom rule — it’s really hard to avoid spurious underflow of intermediate terms in expressions like this unless you have control over the arrangements of the terms. Your control over the intermediate terms that AD generates seems like it will inevitably be pretty fragile — even if you find a formulation that works, an update to the AD system could easily break it.

If you don’t have to worry about over/underflow, e.g. if your magnitudes are at worst 10^{\pm 50} or so, then the problem becomes much easier.