# Smooth approximation to max(0, x)

**URL:** <https://discourse.julialang.org/t/smooth-approximation-to-max-0-x/109383>\
**Category:** Numerics\
**Tags:** question\
**Created:** [January 29, 2024, 10:21am UTC](https://discourse.julialang.org/t/smooth-approximation-to-max-0-x/109383 "2024-01-29T10:21:38Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [January 29, 2024, 10:21am UTC](https://discourse.julialang.org/t/smooth-approximation-to-max-0-x/109383/1 "2024-01-29T10:21:38Z")

</div>

I need a “smooth” approximation f to x \to \max(0, x), such that

1. \lim\_{x \to -\infty} f(x) = 0,
2. \lim\_{x \to \infty} f(x) = x,
3. f(x) \> 0, f'(x) \> 0, both continuous; ideally continuous higher derivatives
4. not too expensive to calculate.

I am aware of \log(a + \exp(x)), which can be implemented with `LogExpFunctions.log1pexp`, is there anything else?

---

<div class="post-metadata">

**Author:** ![lrnv](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lrnv/32/19373_2.png) [@lrnv](https://discourse.julialang.org/u/lrnv)\
**Post date:** [January 29, 2024, 10:29am UTC](https://discourse.julialang.org/t/smooth-approximation-to-max-0-x/109383/2 "2024-01-29T10:29:03Z")

</div>

Would you accept a function that approximates only around 0 and is exactly equal to 0 before a constant -a and to x after a constant b ? If so, we could provide a very efficient solution (in term of computational time) with e.g. a bezier curve

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [January 29, 2024, 10:35am UTC](https://discourse.julialang.org/t/smooth-approximation-to-max-0-x/109383/3 "2024-01-29T10:35:08Z")

</div>

You mean a cubic polynomial matching the derivatives and the values at -a and b? I could try that, but I am looking for something smoother.

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [January 29, 2024, 10:36am UTC](https://discourse.julialang.org/t/smooth-approximation-to-max-0-x/109383/4 "2024-01-29T10:36:36Z")

</div>

Have you seen “mish” and “squareplus” from  
[Rectifier (neural networks) - Wikipedia](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)), or the [Smooth Maximum Unit](https://arxiv.org/pdf/2111.04682.pdf)?

This paper considers a number of smoothed functions [When are smooth-ReLUs ReLU-like? | OpenReview](https://openreview.net/forum?id=0qnryNf6XwR)

---

<div class="post-metadata">

**Author:** ![cdawg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cdawg/32/9811_2.png) [@cdawg](https://discourse.julialang.org/u/cdawg)\
**Post date:** [January 29, 2024, 10:45am UTC](https://discourse.julialang.org/t/smooth-approximation-to-max-0-x/109383/5 "2024-01-29T10:45:40Z")

</div>

by pattern matching Id guess maybe anything that is invf(f(0) + f(x)) where f is increasing monotonically and has range (0,inf) ?? First guess is usually wrong tho

---

<div class="post-metadata">

**Author:** ![SortofDamocles](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sortofdamocles/32/15709_2.png) [@SortofDamocles](https://discourse.julialang.org/u/SortofDamocles)\
**Post date:** [January 29, 2024, 11:33am UTC](https://discourse.julialang.org/t/smooth-approximation-to-max-0-x/109383/6 "2024-01-29T11:33:15Z")

</div>

That’s a good pattern for sure. Similar to the “smooth absolute value” sqrt(x^2 + c)

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [January 29, 2024, 11:57am UTC](https://discourse.julialang.org/t/smooth-approximation-to-max-0-x/109383/7 "2024-01-29T11:57:00Z")

</div>

> [@cdawg](#):
>
> anything that is invf(f(0) + f(x)) where f is increasing monotonically and has range (0,inf)

I may be misunderstanding something, but that’s precisely the kind of function I am looking for. sure, if I have it, I can transform it a lot of ways.

> [@jar1](#):
>
> Have you seen “mish” and “squareplus” from  
> [Rectifier (neural networks) - Wikipedia](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)), or the [Smooth Maximum Unit](https://arxiv.org/pdf/2111.04682.pdf)?

No, but thanks, this is very helpful.

---

<div class="post-metadata">

**Author:** ![barucden](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barucden/32/26154_2.png) [@barucden](https://discourse.julialang.org/u/barucden)\
**Post date:** [January 29, 2024, 12:41pm UTC](https://discourse.julialang.org/t/smooth-approximation-to-max-0-x/109383/8 "2024-01-29T12:41:57Z")

</div>

Does \log(a + \exp(x)) have an unwanted property, or do you want to know other functions just in general?

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [January 29, 2024, 12:42pm UTC](https://discourse.julialang.org/t/smooth-approximation-to-max-0-x/109383/9 "2024-01-29T12:42:49Z")

</div>

How about:

```julia
t = (1/4)^(1/3)
t1, t2 = t^4 - t, t^4

```

```julia
julia> relu(x) = ifelse(x > 0, x, 0)
relu (generic function with 1 method)

julia> lineplot(-0.5,0.5,x-> t1 < x < t2 ? (x-t1)^4 : relu(x))
            ┌────────────────────────────────────────┐       
        0.5 │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⡞│ #57(x)
            │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⠎⠀│       
            │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡰⠃⠀⠀│       
            │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡖⠁⠀⠀⠀│       
            │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⠎⠀⠀⠀⠀⠀│       
            │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡠⠃⠀⠀⠀⠀⠀⠀│       
            │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡜⠁⠀⠀⠀⠀⠀⠀⠀│       
   f(x) │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⢠⠏⠀⠀⠀⠀⠀⠀⠀⠀⠀│       
            │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⡤⠃⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│       
            │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⢀⡔⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│       
            │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⡠⠊⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│       
            │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⢀⡔⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│       
            │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⢀⡔⠃⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│       
            │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⡤⡗⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│       
          0 │⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⡤⠤⠔⠒⠋⠁⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│       
            └────────────────────────────────────────┘       
            ⠀-0.5⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀0.5⠀       
            ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀x⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀       

```

It has a conditional, but it’s kind of simple to think about. And 4th power easy to calc using double squaring. Conditional doesn’t need to mean branching code i.e.

```julia
ifelse( t1 < x < t2, (x-t1)^4, ifelse(x < 0, 0, x))

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [January 29, 2024, 12:47pm UTC](https://discourse.julialang.org/t/smooth-approximation-to-max-0-x/109383/10 "2024-01-29T12:47:27Z")

</div>

> [@barucden](#):
>
> do you want to know other functions just in general?

yes.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [January 29, 2024, 1:00pm UTC](https://discourse.julialang.org/t/smooth-approximation-to-max-0-x/109383/11 "2024-01-29T13:00:38Z")

</div>

I like `squareplus` but the numerical evaluation at `x < 0` involves a delicate cancelation. Two options are to switch to

\frac{b}{2\left(\sqrt{x^2+b}-x\right)}

when x \< 0 (you get that just by multiplying numerator and denominator by x - \sqrt{x^2 + b}). But generally I just use

x\_+ = \begin{cases} \sqrt{x^2 + \sigma^2} - \sigma & x \ge 0; \\ 0 & \textrm{otherwise}. \end{cases}

There’s also a delicate cancelation here, but it’s not one that depends strongly on x, which to me is an advantage. Pick a \sigma for which \sqrt{\sigma^2} - \sigma = 0 in floating-point arithmetic, e.g., any power of 2. Also, use `hypot(x, σ)` if you care about precision.

---

<div class="post-metadata">

**Author:** ![PeterSimon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petersimon/32/25193_2.png) [@PeterSimon](https://discourse.julialang.org/u/PeterSimon)\
**Post date:** [January 30, 2024, 2:10am UTC](https://discourse.julialang.org/t/smooth-approximation-to-max-0-x/109383/12 "2024-01-30T02:10:14Z")

</div>

> [@Tamas\_Papp](#):
>
> but I am looking for something smoother.

This isn’t too practical because each function evaluation requires a numeric integration, but it was a fun way to revisit some math I hadn’t looked at in decades. I smoothed your original function by convolving it with a scaled [bump function](https://en.wikipedia.org/wiki/Bump_function). This is motivated by 2 facts:

1. Convolving a distribution (your function) with a test (bump) function results in an infinitely differentiable function.
2. The scaled bump functions approach the delta function in the limit as \epsilon \rightarrow 0 so that you can get an arbitrarily good approximation to your original distribution.

The code and resulting plots for \epsilon = 0.01 are shown below.

```julia
using QuadGK: quadgk
using SpecialFunctions: besselk
using Plots

f(x) = max(zero(x), x) # function to be smoothed

half = 0.5 # Can use higher precision type here if desired
const I1 = exp(-half) * (besselk(1, half) - besselk(0, half)) # Integral of unnormalized bump function
bump_unnormalized(x) = abs(x) > 1 ? zero(x) : exp(-1 / (1 - x^2))
bump(x, ϵ=one(x)) = bump_unnormalized(x/ϵ) / (ϵ * I1) # normalized, scaled bump function

# convolve scaled bump function with f
f_smooth(x,ϵ) = quadgk(t -> f(x-t) * bump(t, ϵ), -ϵ, ϵ)[1]

ϵ = 0.01

p = plot(xlabel="x", ylabel="f_smooth(x)", title="ϵ = $ϵ", legend=false, ratio=1)
plot!(p, range(-10ϵ, 10ϵ, 500), x -> f_smooth(x, ϵ), legend=false, ratio=1)
display(p)

p = plot(xlabel="x", ylabel="f_smooth(x)", title="ϵ = $ϵ", legend=false, ratio=1)
plot!(p, range(-ϵ, ϵ, 500), x -> f_smooth(x, ϵ))
display(p)

p = plot(xlabel="x", ylabel="|f(x) - f_smooth(x)|", title="ϵ = $ϵ", legend=false)
plot!(p, range(-10ϵ, 10ϵ, 500), x -> abs(f(x) - f_smooth(x, ϵ)))

```

 ![image](https://global.discourse-cdn.com/julialang/original/3X/f/c/fc24b778074aae4790f900d2378eb932d97d0201.png)

 ![image](https://global.discourse-cdn.com/julialang/original/3X/a/1/a1dee6a84677faace954227f20f46c2b602d0a09.png)

![f_smooth_error](https://global.discourse-cdn.com/julialang/original/3X/6/0/606a7fcc55c8a2c2d2af7c08b2e38eff99d1ccd1.png)

On my machine execution time for invoking `f_smooth(x, ϵ)` ranges from about 200 nsec for `x < ϵ` to about 3.5 usec for `x > ϵ`.

Edit:  
Maybe this approach can be cheap enough (on average), since you only need evaluate the smoothed function within the transition region:

```julia
f_smoothfaster(x, ϵ) = abs(x) < ϵ ? fsmooth(x, ϵ) : f(x)

```

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [January 30, 2024, 5:13am UTC](https://discourse.julialang.org/t/smooth-approximation-to-max-0-x/109383/13 "2024-01-30T05:13:11Z")

</div>

While cool, you can also get arbitrarily accurate and analytic without needing to integrate with `f(x,ϵ) = log1p(exp(x/ϵ))*ϵ`  
 ![image](https://global.discourse-cdn.com/julialang/original/3X/4/0/4080d31cad7ea5f5dee327c4abcca87a23a71882.png)

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [March 31, 2024, 1:43am UTC](https://discourse.julialang.org/t/smooth-approximation-to-max-0-x/109383/14 "2024-03-31T01:43:42Z")

</div>

> [@tim.holy](#):
>
> I like `squareplus` but the numerical evaluation at `x < 0` involves a delicate cancelation. Two options are to switch to
> 
> \frac{b}{2\left(\sqrt{x^2+b}-x\right)}
> 
> b2(√x2+b−x)\frac{b}{2\left(\sqrt{x^2+b}-x\right)}
> 
> when x \< 0x\<0x \< 0 (you get that just by multiplying numerator and denominator by x - \sqrt{x^2 + b}x−√x2+bx - \sqrt{x^2 + b}).

Another alternative; we have {\sqrt{x^2 + b} + x} = {{\sqrt{b}} \cdot {e^{asinh{\frac{x}{\sqrt{b}}}}}}, so there are now three alternatives for `squareplus`:

```julia
squareplus_wikipedia(x, b) = (x + sqrt(x*x + b))/2

squareplus_timholy(x, b) = b/(2 * (sqrt(x*x + b) - x))

function squareplus_exp_asinh(x, b)
  sb = sqrt(b)
  exp(asinh(x/sb))*sb/2
end

```