# Testing and tolerances for implicitly calculated functions and their AD

**URL:** <https://discourse.julialang.org/t/testing-and-tolerances-for-implicitly-calculated-functions-and-their-ad/133575>\
**Category:** Numerics\
**Tags:** question, testing, ad, implicit-equation\
**Created:** [October 31, 2025, 12:19pm UTC](https://discourse.julialang.org/t/testing-and-tolerances-for-implicitly-calculated-functions-and-their-ad/133575 "2025-10-31T12:19:23Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [October 31, 2025, 12:19pm UTC](https://discourse.julialang.org/t/testing-and-tolerances-for-implicitly-calculated-functions-and-their-ad/133575/1 "2025-10-31T12:19:23Z")

</div>

To fix ideas, suppose we have a function x = g(p) defined by f(x(p), p) = 0, both arguments are scalars.

The numerical implementation involves finding an x such that | f(x, p)| \le \mathrm{tol}, either by Newton’s method or Brent, depending on circumstances.

Once x is found, the derivative x'(p) = - (\partial f/\partial p)/(\partial f/\partial x) can be readily calculated and incorporated into an AD rule, which I will want to unit test.

I am running into two issues:

1. most test frameworks use the excellent FiniteDifferences.jl, so one can pass on a finite difference method for checking the derivative. However, I am unsure how to parametrize it, there is a `factor = ...` argument but it refers to _relative_ noise. But even the absolute, let alone the relative error of these calculations depends on the p and the tolerance. Is there anything else I should be doing?

2. When it comes to unit testing, it is unclear how to choose tolerances. In practice I found that some tests fail if I don’t choose high tolerances (think `rtol = 0.05` etc). But then I am worried that it’s just my code having errors that pass silently.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [November 1, 2025, 12:22pm UTC](https://discourse.julialang.org/t/testing-and-tolerances-for-implicitly-calculated-functions-and-their-ad/133575/2 "2025-11-01T12:22:56Z")

</div>

> [@Tamas\_Papp](#):
>
> most test frameworks use the excellent [FiniteDifferences.jl](https://juliaregistries.github.io/General/packages/redirect_to_repo/FiniteDifferences), so one can pass on a finite difference method for checking the derivative. However, I am unsure how to parametrize it, there is a `factor = ...` argument but it refers to _relative_ noise. But even the absolute, let alone the relative error of these calculations depends on the p and the tolerance. Is there anything else I should be doing?

Easiest thing to do, which is what NonlinearSolve.jl does for its testing here, is to just test ForwardDiff vs Forward-Mode rule vs Reverse-mode rule, so one diffs the algorithm and the other two check the rules.

FiniteDifferences.jl is actually a really bad idea for this kind of thing because of numerical instabilities so it shouldn’t be used as a way to check rules but 🤷 I brought it up before. At least for the simple cases in ChainRules.jl that doesn’t really matter but for most things you’d see in SciML it’s just not even a convergent way to do things considering the numerical error so yeah, highly do not suggest that people ever use that package in practice. Either FiniteDiff.jl as a fallback because it basically is just the caching AD interfaces with the simplest finite difference methods or nothing, but higher order FiniteDifferences.jl is a huge can of worms in most real cases you’d want to test.

> [@Tamas\_Papp](#):
>
> 1. When it comes to unit testing, it is unclear how to choose tolerances. In practice I found that some tests fail if I don’t choose high tolerances (think `rtol = 0.05` etc). But then I am worried that it’s just my code having errors that pass silently.

This can be difficult because for implicit problems it can depend on the numerical solving technique used and the condition number of the Jacobian as to how close to convergence you get. There is a pullback error in the rule that is proportional to both of these factors (I need to write down that somewhere 😅) and so yes, you can have cases where differentiating the algorithm vs the implicit function rule differ by 1e-3 even though the solver tolerance is 1e-12 (and once you see the proofs of this error it’s not hard to construct such edge cases). So the answer is, it is a “it depends”

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 4, 2025, 1:24pm UTC](https://discourse.julialang.org/t/testing-and-tolerances-for-implicitly-calculated-functions-and-their-ad/133575/3 "2025-11-04T13:24:59Z")

</div>

> [@ChrisRackauckas](#):
>
> Easiest thing to do, which is what [NonlinearSolve.jl](https://juliaregistries.github.io/General/packages/redirect_to_repo/NonlinearSolve) does for its testing here, is to just test ForwardDiff vs Forward-Mode rule vs Reverse-mode rule, so one diffs the algorithm and the other two check the rules.

Are you referring to [this test set](https://github.com/SciML/NonlinearSolve.jl/blob/f23f29469a0aad6afd5a5d6df150f8a70371ba8b/test/forward_ad_tests.jl)? (and the similar file for adjoints)?

> [@ChrisRackauckas](#):
>
> so yes, you can have cases where differentiating the algorithm vs the implicit function rule differ by 1e-3 even though the solver tolerance is 1e-12 (and once you see the proofs of this error it’s not hard to construct such edge cases). So the answer is, it is a “it depends”

Thanks, this is what I found too. The approach you recommend makes sense, but it then bypasses the various other correctness and sanity checks that AD suites provide. Ideally, such suites should allow users to provide a function that calculates derivatives (using the arguments and the return value), instead of fdm.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 2, 2025, 12:49pm UTC](https://discourse.julialang.org/t/testing-and-tolerances-for-implicitly-calculated-functions-and-their-ad/133575/4 "2025-12-02T12:49:17Z")

</div>

I came up with another solution that I find rather useful.

Suppose that the equation that defines the solution implicitly is `f(x, p)`, and `x = g(p)` solves it. Then define

```julia
h(p) = f(g(p), p)

```

and test the _gradient_ of `h` to be \approx 0 (when `f` evaluates to a scalar, otherwise, if `f` has a vector output, use a Jacobian).

One can do this with AD directly, no comparison to finite differences required. This implicitly checks the AD rule for `g`, but does that in the context of `f`, which is what defines it.