# How to smooth the if function

Is there currently a library that provides a smoothing function to ensure the differentiability of the judgment function?

For example, I have a simple formula here:
f(x) = x > 1 ? x : x^2

Is there a smooth function here so that the following formula can obtain approximate results?

f_smooth(x) = smooth_func(x-1)*x+smooth_func(1-x)*x^2

Currently I have built a smooth_func, see below:

smooth_func(x) = (tanh(5.0 * x) + 1.0) * 0.5

There are lots of ways to smooth the ifelse function, so the first question is why do you need it? And why is the one you found not good enough?

2 Likes

This may help:

2 Likes

Firstly, this âsmooth functionâ is something I initially encountered in a paper. It seems to be used to replace the âifelseâ function to ensure the differentiability of the expression. I attempted to calculate the derivatives using both the âsmoothâ and âifelseâ approaches separately with âForwardDiffâ:

using ForwardDiff

f1(x) = x > 1 ? x^3 : x^2
smooth_func(x) = (tanh(5.0 * x) + 1.0) * 0.5
f2(x) = smooth_func(x-1) * x + smooth_func(1-x) * x^2

ForwardDiff.derivative(f1, 0.1)
ForwardDiff.derivative(f2, 0.1)


I found that functions based on âifelseâ can also be differentiated using âForwardDiffâ. Being a beginner in this area, I started to question whether the âsmooth functionâ is necessary when dealing with situations involving âifelseâ.

Moving on to the second question, the âsmooth functionâ doesnât produce the expected results when the input approaches zero. For example:

julia> smooth_func(1e-6)
0.5000025

julia> smooth_func(-1e-6)
0.49999750000000004

julia> smooth_func(1e-2)
0.52497918747894

julia> smooth_func(1e-3)
0.5024999791668749

julia> smooth_func(1e-1)
0.7310585786300049


This greatly affects my subsequent calculations and the results of differentiation. After multiple attempts, I found that adding the constant 5.0 as the first term can effectively avoid this issue. It appears that there is a significant correlation between the input value and the âsmooth functionâ. When abs(5.0*x) >> 1, the result matches expectations.

1 Like

Thank you, this seems to be what Iâm looking for. Iâll go take a look.

Thatâs why Iâm asking about your use case. In most cases, you donât care about the behavior near the non-differentiable point (in this case a discontinuity). ForwardDiff makes an arbitrary choice, other autodiff backends may make other choices, but without knowing what you do itâs hard to decide if these choices matter at all.

What is your âexpected resultâ near zero? It seems to approach 0.5, which is a reasonable behavior.

Can you provide a complete example?

2 Likes

I have written a demo (but it does not fully reproduce the issue I encountered before), see below:

using OrdinaryDiffEq, ModelingToolkit
using ModelingToolkit: t_nounits as t, D_nounits as D

@parameters Ď Ď Î˛
@variables x(t)

step_func(v) = (tanh(Î˛ * v) + 1.0) * 0.5

eqs = [
D(x) ~ step_func(t - Ď) * Ď
]

@mtkbuild sys = ODESystem(eqs, t)

u0 = [x => 1.0]

p = [
Ď => 1.0,
Ď => 5.0,
Î˛ => 1.0]

tspan = (4.5, 5.5)
prob = ODEProblem(sys, u0, tspan, p, jac=true)

for i in [0.1, 1.0, 5.0, 10.0, 50.0]
new_prob = remake(prob, p=[Î˛ => i])
new_sol = solve(new_prob, Tsit5(), saveat=0.1)
println(vcat(new_sol.u...))
end


I keep adjusting the parameters of the smooth function, and the calculation results also show deviations.
Using the smooth function in my project may lead to even worse situations, such as the issue of ordinary differential equations having no solutions, and the calculation results include unexpected negative numbers.

What is your âexpected resultâ near zero? It seems to approach 0.5, which is a reasonable behavior.

I hope to nearly perfectly simulate the function v > 0 ? 1 : 0, but when it approaches 0, the result of the smooth function has a slight discrepancy, for example,

(tanh(5.0 * 0.1) + 1.0) * 0.5 = 0.731058
(tanh(5.0 * -0.1) + 1.0) * 0.5 = 0.268941


Just increase the âgainâ

(tanh(50.0 * 0.1) + 1.0) * 0.5 = 0.99995460213

(tanh(50.0 * -0.1) + 1.0) * 0.5=0.00004539786



But It have side efrects since the derivative at 0 will increase and It could be a problem, it deppends on your exact applicationâŚ

1 Like

Yes, improving gain can indeed prevent this issue.

The standard solution to this is to âcompactifyâ the transition region of a sigmoid. The classic function you see in differential geometry is

\verb|smooth_func|(x) = \begin{cases} 0 & x \leq -1, \\ \frac{1+\tanh[x/(1-x^2)]}{2} & -1 < x < 1, \\ 1 & x \geq 1. \end{cases}

This turns out to be a smooth (infinitely differentiable, C^\infty) function. Hereâs an implementation Iâve used that I think does a pretty good job of dealing with the various roundoff/overflow issues that can arise with this function. Iâve written it for vector input, but you could convert to accept a single point if you want:

function smooth_func(x::AbstractArray, xâ, xâ, yâ=0, yâ=1)
@assert xâ â¤ xâ
y = similar(x)
xâ, xâ, yâ, yâ, mÂ˛ = convert.(eltype(y), (xâ, xâ, yâ, yâ, 3))
m = âmÂ˛
Îy = (yâ - yâ) / 2
Îx = (xâ - xâ) / 2
yĚ = (yâ + yâ) / 2
xĚ = (xâ + xâ) / 2
for i â eachindex(x, y)
xĚ = (x[i] - xĚ) / Îx
# Note: These || cases are nearly redundant,
# but protect against very rare roundoff issues
if xĚ â¤ -1 || x[i] â¤ xâ
y[i] = yâ
elseif xĚ âĽ 1 || x[i] âĽ xâ
y[i] = yâ
else
y[i] = yĚ + Îy * tanh(m * xĚ / (1 - xĚ^2))
end
end
return y
end


You have to tell it the x_0 value where you want it to start transitioning and the x_1 value where you want it to stop transitioning. Optionally, you can also adjust the y values before and after the transition. Then, you can use it in the standard way:

x0 = -1
x1 = 1
Ď = smooth_func(x, x0, x1)
f_smooth = (1-Ď) * x + Ď * x^2


For example, it can cause optimization algorithms to converge very slowly (e.g. because the second derivative matrix may be badly conditioned in a multi-variate problem). Or an ODE solver may converge very slowly (require tiny time steps).

Really, people are just stabbing in the dark here without knowing the application. There is no generically âcorrectâ or âgoodâ way to make a non-differentiable function differentiable, because the question itself is underspecified.

2 Likes

In an ODE context, itâs much much better to just put in the correct discontinuity analytically, e.g. via callbacks. There is no conceptual problem with having discontinuities or (distributional) derivatives thereof on the rhs of an ODE, and implementing this isnât too hard either.

3 Likes