Debugging non-finite Jacobian elements (w/ ForwardDiff)

Specific Domains Numerics
,
1

I have a rather complex calculation that maps a vector of floats to a longer vector of floats.

At some inputs, ForwardDiff gives me a Jacobian where some elements are NaN. Inputs are of course finite, and so is the function value.

When checked with FiniteDifferences and that works fine. I would like to understand what is going on (I am suspecting a numerical issue with a rule or something). I have thought of adding

_finiteD(x::Real) = true
_finiteD(x::ForwardDiff.Dual) = all(isfinite, x.partials)
_finiteD(a::AbstractArray) = all(_finiteD, a)

and then peppering the code with

@assert _finiteD(some_interim_value)

but this will be tedious. Is there a shortcut?

2

maybe use DifferentiationInterface to try different backends?

1 Like
3

Sometimes using NaN safe mode has fixed such issues for me, Advanced Usage Guide · ForwardDiff.

But otherwise I’ve done as you and resorted to manually print if I detect nan/inf and then figure out why. Sometimes, it was needed to modify the equation with eps() (to avoid division by zero, for example, for square roots). Would be nice if there was an automatic detection that did require so much manual insertion of checks!

1 Like
4

I tried Enzyme which fails (reported the issue), is there any other backend suitable for forward mode?

5

In case anyone is curious, the MWE for the offending code is

function f(param)
    decay = exp(param)              # make it positive
    α = exp(-decay)
end

This “works” (for a given value of “works” :smiley:) for eg param = 800.0 but produces NaNs with AD. This is called inside a numerical solver where the solution is around param = 0.7, but large values are visited while the parameter space is explored.

Since the purpose of the first exp is to convert a parameter in \mathbb{R} to a positive number, I replaced it with logaddexp(zero(param), param) which does not suffer from numerical issues.

In the end I found it with the _finiteD checks as above.