# What's going on with exp() and --math-mode=fast?

**URL:** <https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619>\
**Category:** General Usage\
**Tags:** fast-math\
**Created:** [July 14, 2021, 8:34am UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619 "2021-07-14T08:34:03Z")\
**Posts on this page:** 10\
**Page:** 2

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 14, 2021, 6:29pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/21 "2021-07-14T18:29:02Z")

</div>

> [@Sukera](#):
>
> So I’d argue that the library _not_ resetting the flag was faulty behavior and a library compild with `-Ofast` shouldn’t affect the caller either way 😛

No, the fact is that the compiler changes the global CPU setting:  
[https://twitter.com/vchuravy/status/1216788090079076354](https://twitter.com/vchuravy/status/1216788090079076354)

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [July 14, 2021, 6:31pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/22 "2021-07-14T18:31:16Z")

</div>

I’m aware - I’m arguing that the compiler should insert code switching that flag to whatever it was when the function was called when returning from those library functions 🙂 The flag being CPU global doesn’t help things of course, but I’d rather not discuss in this thread how modern CPUs aren’t/shouldn’t behave like a PDP-11 on steroids 😃

---

<div class="post-metadata">

**Author:** ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)\
**Post date:** [July 15, 2021, 5:11am UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/23 "2021-07-15T05:11:10Z")

</div>

I’ve submitted an issue with a proposed (naive) solution of forcing the subtraction with llvmcall:

> <https://github.com/JuliaLang/julia/issues/41592>
>
> As discussed on the forum, the accuracy of exp() suffers when using fast-math be…cause llvm optimizes away a numerical trick that is used for rounding:
> https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/21
> 
> While some inaccuracy is expected when using fast-math, it seems like this specific situation could be improved by instructing llvm to not optimize away the numerical rounding trick.
> 
> There's probably a better way to do this, but I was able to get exp() working with fast-math by replacing the subtraction:
> \`\`\`N\_float -= MAGIC\_ROUND\_CONST(T)\`\`\`
> with
> \`\`\`N\_float = forcesub(N\_float, MAGIC\_ROUND\_CONST(T))\`\`\`
> where
> \`\`\`
> function forcesub(a,b)
> Base.llvmcall("%3 = fsub double %0, %1\\n ret double %3", Float64, Tuple{Float64,Float64}, a, b)
> end
> \`\`\`
> 
> Before (with --math-mode=fast):
> \`\`\`
> julia\> exp(1.0)
> 2.7158546124258023
> 
> julia\> exp(1e-3)
> 1.0
> \`\`\`
> After (with --math-mode=fast):
> \`\`\`
> julia\> exp(1.0)
> 2.718281828459045
> 
> julia\> exp(1e-3)
> 1.0010005001667084

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [July 15, 2021, 9:06am UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/24 "2021-07-15T09:06:17Z")

</div>

I do want to point out that if your wasn’t written with any particular order in mind, `@fastmath` will often be more accurate, e.g. it will use `fma` instead of `*` followed by `+` on computers with `fma`-support, which incurs fewer rounding steps.  
Or, using the problematic example here, `1.0 * 369.3299304675746 + 6.755399441055744e15 - 6.755399441055744e15 == 369.3299304675746` if rounding weren’t a problem. So in some sense, the fastmath version – by returning `369.3299304675746` instead of `369.0` – is more accurate here.  
However, `369.0` was the intended result.  
The `exp` code was deliberately written with floating point behavior and rounding in mind. The IEEE version does precisely what the author intended.

So I would suggest that if you didn’t have specific intentions about order or rounding behavior – e.g. the order you specified was picked out of all possibilities out of convenience – then switching from one arbitrary order to another specified by `@fastmath` will increase accuracy more often than not.

But when these were chosen deliberately, i.e. not arbitrarily, results can be catastrophic for accuracy. Starting Julia with `--math-mode=fast`:

```julia
julia> x = randn(10)'
1×10 adjoint(::Vector{Float64}) with eltype Float64:
 0.411952 -1.20621 -0.780723 0.69826 -1.60801 -1.88011 0.363784 0.663141 -0.874339 0.613377

julia> sinpi.(x)
1×10 Matrix{Float64}:
 0.0 -0.0 -0.0 0.0 -0.0 -0.0 0.0 0.0 -0.0 0.0

julia> cospi.(x)
1×10 Matrix{Float64}:
 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0

```

I think it’d be nice to use fastmath in some more places, e.g. in `(add/mul)_fast(::ForwardDiff.Dual, ...)` to optionally avoid some of the multiply-by and add-zeros from ForwardDiff that require some of fast flags to actually eliminate.  
But it shouldn’t be applied to code written by others who may have implemented algorithms requiring specific sequences of operations.

Global fastmath is pretty bad IMO.  
Which is part of the reason I’m somewhat encouraging applying it (or the safer alternative: [@muladd](https://github.com/SciML/MuladdMacro.jl)) more locally: get rid of the incentive to use it globally by having the places that can use it safely already benefit from fma instructions.

EDIT: I think `@muladd` and `@simd` should be preferred over `@fastmath`, as the two of them cover the most important optimizations enabled by `@fastmath` with less of the safety concerns (e.g., handling of `NaN`s).

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [July 15, 2021, 12:28pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/25 "2021-07-15T12:28:20Z")

</div>

[fix `exp` in `--math-mode=fast` by oscardssmith · Pull Request #41600 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/pull/41600) should fix it.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [August 26, 2021, 12:32pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/26 "2021-08-26T12:32:26Z")

</div>

It was approved to disable the (global) --math-mode=fast: [https://github.com/JuliaLang/julia/pull/41607#issuecomment-897887059](https://github.com/JuliaLang/julia/pull/41607#issuecomment-897887059)

and I have the PR for it (that one is closed, but still the valid one, it only needs to be merged, and it’s taking some time, maybe since it’s the closed one?):

[https://github.com/JuliaLang/julia/pull/41638](https://github.com/JuliaLang/julia/pull/41638)

---

<div class="post-metadata">

**Author:** ![johanter](https://avatars.discourse-cdn.com/v4/letter/j/8e7dd6/32.png) [@johanter](https://discourse.julialang.org/u/johanter)\
**Post date:** [October 21, 2021, 4:49pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/27 "2021-10-21T16:49:41Z")

</div>

Hi, I don’t know if this is related or fixed already. I tracked down an issue ([https://github.com/JuliaRobotics/IncrementalInference.jl/issues/1382](https://github.com/JuliaRobotics/IncrementalInference.jl/issues/1382)) in julia 1.7rc1 to a function that uses `@fastmath` in [https://github.com/JuliaRobotics/KernelDensityEstimate.jl](https://github.com/JuliaRobotics/KernelDensityEstimate.jl) .

for julia 1.7.0-rc1

```julia
julia> versioninfo()
Julia Version 1.7.0-rc1
Commit 9eade6195e (2021-09-12 06:45 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i5-7300HQ CPU @ 2.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake)
Environment:
  JULIA_EDITOR = code
  JULIA_NUM_THREADS = 4

julia> foo(x) = @fastmath exp(-0.5*x)
foo (generic function with 1 method)

julia> foo(2835.0)
-7.888656068042933

julia> exp(-0.5*2835.0)
0.0

```

for julia 1.6.3

```julia
julia> versioninfo()
Julia Version 1.6.3
Commit ae8452a9e0 (2021-09-23 17:34 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i5-7300HQ CPU @ 2.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)

julia> foo(x) = @fastmath exp(-0.5*x)
foo (generic function with 1 method)

julia> foo(2835.0)
0.0

julia> exp(-0.5*2835.0)
0.0

```

---

<div class="post-metadata">

**Author:** ![dehann](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dehann/32/2003_2.png) [@dehann](https://discourse.julialang.org/u/dehann)\
**Post date:** [October 21, 2021, 4:58pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/28 "2021-10-21T16:58:17Z")

</div>

I get the same issue with Julia 1.7-rc1 while 1.6.3 works fine.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [October 21, 2021, 5:05pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/29 "2021-10-21T17:05:58Z")

</div>

This is my fault. I had a PR that fixed it, but I forgot about it. Will remake today and get it merged before 1.7.0

---

<div class="post-metadata">

**Author:** ![johanter](https://avatars.discourse-cdn.com/v4/letter/j/8e7dd6/32.png) [@johanter](https://discourse.julialang.org/u/johanter)\
**Post date:** [October 23, 2021, 5:13pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/30 "2021-10-23T17:13:28Z")

</div>

Thanks, the tests are now passing on the julia nightly build.

[Previous page](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619.md?page=1)