# Continuous relative/absolute residual

**URL:** <https://discourse.julialang.org/t/continuous-relative-absolute-residual/22949>\
**Category:** Optimization (Mathematical)\
**Tags:** question\
**Created:** [April 9, 2019, 7:43am UTC](https://discourse.julialang.org/t/continuous-relative-absolute-residual/22949 "2019-04-09T07:43:01Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [April 9, 2019, 7:43am UTC](https://discourse.julialang.org/t/continuous-relative-absolute-residual/22949/1 "2019-04-09T07:43:01Z")

</div>

This is, strictly speaking, not a Julia but a math question, but I am running into this with AD and optimization.

I am solving a parametric system

f(x, \theta) = g(x, \theta)

for x given \theta. I could implement this as

```julia
function residual(x, θ)
    F = f(x, θ)
    G = g(x, θ)
    F .- G
end

```

but it may make more sense near the optimum to use a criterion like `@. (F - G)/F` or `@. (F - G)/G`. But this may fail if either `F ≈ 0` or `G ≈ 0`.

Some texbooks recommend something like `@. (F - G)/max(1, F, G)`, but then the derivatives are not continuous.

Is there a “standard” way of doing what I want in a continuously differentiable way? I think I could combine a softmax to get this, but thought I would ask first here.

---

<div class="post-metadata">

**Author:** ![Bernard\_GODARD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bernard_godard/32/4155_2.png) [@Bernard\_GODARD](https://discourse.julialang.org/u/Bernard_GODARD)\
**Post date:** [April 9, 2019, 10:44am UTC](https://discourse.julialang.org/t/continuous-relative-absolute-residual/22949/2 "2019-04-09T10:44:21Z")

</div>

Not sure about a standard way, but:

```julia
@. (F - G)/( 1 + abs(F) + abs(G) )

```

will prevent derivative discontinuities when F-G oscillates around zero which is likely to happen in such a problem.  
You still get discontinuities if either F or G are oscillating around zeros, but this can be solved by creating a smoothed version of abs()

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 9, 2019, 11:23am UTC](https://discourse.julialang.org/t/continuous-relative-absolute-residual/22949/3 "2019-04-09T11:23:03Z")

</div>

The way it’s done in differential equations is to mix them:

```julia
@. (F - G)/max(abstol + max(F,G)*reltol) < 1

```

Then those tolerances can of course either be scalars or arrays that match the size of `F` and `G`.

---

<div class="post-metadata">

**Author:** ![Bernard\_GODARD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bernard_godard/32/4155_2.png) [@Bernard\_GODARD](https://discourse.julialang.org/u/Bernard_GODARD)\
**Post date:** [April 9, 2019, 11:34am UTC](https://discourse.julialang.org/t/continuous-relative-absolute-residual/22949/4 "2019-04-09T11:34:53Z")

</div>

> [@ChrisRackauckas](#):
>
> @. (F - G)/max(abstol + max(F,G)\*reltol) \< 1

With max(F,G) you probably mean max(abs(F),abs(G)) ?

This can be modified to make its derivative continuous around F-G = 0 into:

```julia
@. (F - G)/( abstol + 0.5 * reltol * ( abs(F) + abs(G) ) )

```

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 9, 2019, 11:41am UTC](https://discourse.julialang.org/t/continuous-relative-absolute-residual/22949/6 "2019-04-09T11:41:46Z")

</div>

> [@Bernard\_GODARD](#):
>
> With max(F,G) you probably mean max(abs(F),abs(G)) ?

Yes

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [April 9, 2019, 12:23pm UTC](https://discourse.julialang.org/t/continuous-relative-absolute-residual/22949/7 "2019-04-09T12:23:45Z")

</div>

I found this [nice approximation integrating a sigmoid function](https://math.stackexchange.com/questions/728094/approximate-x-with-a-smooth-function) to |x|, which motivates

```julia
using PGFPlotsX, StatsFuns

approx_abs(x, k) = 2/k*(log1pexp(k*x) - log(2)) - x
g(x, y) = abs(x - y) / (1 + abs(x) + abs(y)) # we approximate this
f(x, y, k) = approx_abs(x - y, k) / (1 + approx_abs(x, k) + approx_abs(y, k))

# graphs
grid = range(-10, 10; length = 1000)
@pgf Axis({ legend_pos = "south east", legend_cell_align = "left"},
          PlotInc({ no_marks }, Table(grid, g.(grid, 0.2))),
          LegendEntry("g"),
          PlotInc({ no_marks }, Table(grid, f.(grid, 0.2, 1))),
          LegendEntry("f; k = 1"),
          PlotInc({ no_marks }, Table(grid, f.(grid, 0.2, 10))),
          LegendEntry("f; k = 10"))

```

 ![abs](https://global.discourse-cdn.com/julialang/original/3X/7/c/7cd85a4f4d2c10f25dab3e42bcadfddd3f950e6d.png)

Now I need to think about the numerical corner cases (after a ☕) and whether to package this somehow, or find a home in an existing package.

---

<div class="post-metadata">

**Author:** ![mauro3](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mauro3/32/292_2.png) [@mauro3](https://discourse.julialang.org/u/mauro3)\
**Post date:** [April 9, 2019, 12:40pm UTC](https://discourse.julialang.org/t/continuous-relative-absolute-residual/22949/8 "2019-04-09T12:40:28Z")

</div>

I got a bunch of smoothing functions [VAWTools.jl/smoothing-functions.jl at j0.6 · mauro3/VAWTools.jl · GitHub](https://github.com/mauro3/VAWTools.jl/blob/j0.6/src/smoothing-functions.jl), which could use a home too. Note, you could maybe use the `hyperpola` function in above.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [April 9, 2019, 1:26pm UTC](https://discourse.julialang.org/t/continuous-relative-absolute-residual/22949/9 "2019-04-09T13:26:24Z")

</div>

If I understand it correctly, the approximation for |x| would be

\sqrt{x^2 + d^2}

for d \> 0. This is similar to something I saw [on stackexchange](https://math.stackexchange.com/q/728094), and could work very well. It is definitely simpler and easier to reason about (for me).

I will experiment and get back here.

---

<div class="post-metadata">

**Author:** ![dpsanders](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dpsanders/32/3573_2.png) [@dpsanders](https://discourse.julialang.org/u/dpsanders)\
**Post date:** [April 9, 2019, 1:29pm UTC](https://discourse.julialang.org/t/continuous-relative-absolute-residual/22949/10 "2019-04-09T13:29:34Z")

</div>

Is this a nonlinear system? How are you solving this?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [April 9, 2019, 1:33pm UTC](https://discourse.julialang.org/t/continuous-relative-absolute-residual/22949/11 "2019-04-09T13:33:03Z")

</div>

It is a highly nonlinear system. Currently solving with `Optim` (minimizing squared residuals), will experiment with the other nonlinear solvers.