# What's going on with exp() and --math-mode=fast?

**URL:** <https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619>\
**Category:** General Usage\
**Tags:** fast-math\
**Created:** [July 14, 2021, 8:34am UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619 "2021-07-14T08:34:03Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)\
**Post date:** [July 14, 2021, 8:34am UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/1 "2021-07-14T08:34:03Z")

</div>

I was surprised to see results that are significantly different (by much more than I would except when using normal ffast-math optimizations) when running julia with `--math-mode=fast`.

It seems like a very rough approximation is being used for `exp()` ( 0.1% error for `exp(1.0)`). Is this documented anywhere? And why don’t I see the same behavior with `@fastmath` or when calling `Base.Math.exp_fast`?

```julia
$ ~/julia/bin/julia
               _
   _ _ _(_)_ | Documentation: https://docs.julialang.org
  (_) | (_) (_) |
   _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` | |
  | | |_| | | | (_| | | Version 1.8.0-DEV.178 (2021-07-14)
 _/ |\ __'_|_|_|\__'_| | Commit 696cb1ba7b (0 days old master)
|__/ |

julia> exp(1.0)
2.718281828459045

julia> @fastmath exp(1.0)
2.718281828459045

julia> Base.Math.exp_fast(1.0)
2.718281828459045

$ ~/julia/bin/julia --math-mode=fast
               _
   _ _ _(_)_ | Documentation: https://docs.julialang.org
  (_) | (_) (_) |
   _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` | |
  | | |_| | | | (_| | | Version 1.8.0-DEV.178 (2021-07-14)
 _/ |\ __'_|_|_|\__'_| | Commit 696cb1ba7b (0 days old master)
|__/ |

julia> exp(1.0)
2.7158546124258023

```

---

<div class="post-metadata">

**Author:** ![gustaphe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gustaphe/32/18174_2.png) [@gustaphe](https://discourse.julialang.org/u/gustaphe)\
**Post date:** [July 14, 2021, 9:00am UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/2 "2021-07-14T09:00:01Z")

</div>

I don’t know, but try doing the `@fastmath` call first? Maybe the compiler knows the result should be the same and just reuses it?

---

<div class="post-metadata">

**Author:** ![heliosdrm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/heliosdrm/32/3851_2.png) [@heliosdrm](https://discourse.julialang.org/u/heliosdrm)\
**Post date:** [July 14, 2021, 9:23am UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/3 "2021-07-14T09:23:49Z")

</div>

This is old (before Julia 1.0), but I don’t know if the situation has changed. If not, that would explain the difference you observed:

> [@How to tell VS Code to use the fast math option?](https://discourse.julialang.org/t/how-to-tell-vs-code-to-use-the-fast-math-option/18257/2):
>
> You really don’t want to use --math-mode=fast. You’re telling the compiler to do something that’s fundamentally broken, e.g. [sinpi and cospi return incorrect results when using `--math-mode=fast` · Issue #30073 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/30073). The global flag is much worse than the occasional use of the @fastmath macro, which is far less broken, but still may cause incorrect results. You shouldn’t use --math-mode=fast unless you’re willing to take the risk that all of your results might be wrong.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [July 14, 2021, 9:36am UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/4 "2021-07-14T09:36:12Z")

</div>

The beginning of the code for `exp` looks something like:

```julia
const a = 369.3299304675746
const b = 6.755399441055744e15

function f(x::Float64)
    z = muladd(x, a, b)
    z -= b
end

```

Without `--math-mode=fast` this gives the result `369.0`, with it it gives `369.3299304675746`. Looking at the generated code, in fast mode, it has simplified this code to `x*a`:

```julia
julia> @code_llvm f(1.0)
...
  %1 = fmul fast double %0, 0x40771547652B82FE
  ret double %1
}

```

while it otherwise computes things as given:

```julia
julia> @code_llvm f(1.0)
...
   %1 = fmul contract double %0, 0x40771547652B82FE
   %2 = fadd contract double %1, 0x4338000000000000
   %3 = fadd double %2, 0xC338000000000000
  ret double %3
}

```

There can be an arbitrary error in the final result from such a change.

---

<div class="post-metadata">

**Author:** ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)\
**Post date:** [July 14, 2021, 11:53am UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/5 "2021-07-14T11:53:01Z")

</div>

That seems unfortunate. Is there any way to add something like `@nofastmath` to julia’s implementation of `exp()`?

I see a similar constant and `muladd` in the glibc implementation, but that generally won’t be compiled with -ffast-math. Also for some reason, julia’s implementation is 24% slower than glibc’s, even with the `muladd` eliding.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [July 14, 2021, 1:06pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/6 "2021-07-14T13:06:39Z")

</div>

Why are we allowing changing a muladd to a mul? That just seems totally broken.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 14, 2021, 1:34pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/7 "2021-07-14T13:34:42Z")

</div>

> [@robsmith11](#):
>
> That seems unfortunate

I mean, the whole point of fast-math is trading off speed with correctness. If fast-math was to give always the correct results, it wouldn’t be fast-math, it would be the standard way of doing math.

---

<div class="post-metadata">

**Author:** ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)\
**Post date:** [July 14, 2021, 1:47pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/8 "2021-07-14T13:47:59Z")

</div>

> [@giordano](#):
>
> I mean, the whole point of fast-math is trading off speed with correctness. If fast-math was to give always the correct results, it wouldn’t be fast-math, it would be the standard way of doing math.

Right, but generally when I use fast-math, I can reason about my own code and conclude that doing something like swapping the order of some operations will have a minimal impact on my results.

But here we have a standard library function that’s severely affected to the point that it’s unusable for many purposes. And many people who aren’t familiar with julia’s internals may assume that it’s calling glibc math functions, as most languages do.

These errors are orders of magnitudes worse than what one would get when doing other common operations with numbers that are in a reasonable range:

```julia
julia> log(exp(1.0))
0.9991066782289837

julia> log(exp(1e-3))
0.0

```

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [July 14, 2021, 2:05pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/9 "2021-07-14T14:05:55Z")

</div>

> [@Oscar\_Smith](#):
>
> Why are we allowing changing a muladd to a mul? That just seems totally broken.

It simplifies

```julia
z = x * a + b # z = muladd(x, a, b)
z -= b

```

to

```julia
z = x * a

```

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [July 14, 2021, 2:25pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/10 "2021-07-14T14:25:07Z")

</div>

Oh, that actually should be fixable. Rewriting that to unsafe\_trunc should retain almost all the performance and remove the ability of fastmath to mess it up.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [July 14, 2021, 2:46pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/11 "2021-07-14T14:46:44Z")

</div>

More generally, however, I don’t think `@fastmath` should be applied to inlined code from other functions. There are just way too many pitfalls there.

Basically, `@fastmath` is used by people who think “this code is insensitive to rearrangements that are exact in exact arithmentic (e.g. re-association)”, but it is not reasonable to ask anyone to make that judgement for code they _can’t even see_ (because it is inlined from somewhere else).

Update: ~~filed [Issues · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/41582~~) Sorry, my misunderstanding — it seems that `@fastmath` already doesn’t apply to inlined code.

---

<div class="post-metadata">

**Author:** ![simonbyrne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simonbyrne/32/19_2.png) [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)\
**Post date:** [July 14, 2021, 4:17pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/12 "2021-07-14T16:17:59Z")

</div>

> [@giordano](#):
>
> I mean, the whole point of fast-math is trading off speed with correctness. If fast-math was to give always the correct results, it wouldn’t be fast-math, it would be the standard way of doing math.

That’s why we should rename it to `@unsafemath`.

---

<div class="post-metadata">

**Author:** ![antoine-levitt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antoine-levitt/32/4008_2.png) [@antoine-levitt](https://discourse.julialang.org/u/antoine-levitt)\
**Post date:** [July 14, 2021, 4:45pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/13 "2021-07-14T16:45:10Z")

</div>

Yes, fast math doesn’t apply to inlined code (which also makes it more annoying to use).

I think the general agreement is that math-mode=fast is currently massively broken, as this thread shows, and should never be used. I would imagine the view of core devs is “why is this horrid thing still here,let’s kill it”, which is a pity because it could actually be useful with some care. For that to happen , we need a `@nofastmath` macro and apply it judiciously to functions that do floating point trickery (eg special summations, implementations of special functions, etc). If this is done, it should be relatively OK (in the sense that all bets are still off and arbitrary loss of precision is always possible, but it has a pretty good chance of not losing too many digits in practice)? One thing I’m not sure about is how that interacts with precompilation, though.

---

<div class="post-metadata">

**Author:** ![kimikage](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kimikage/32/14534_2.png) [@kimikage](https://discourse.julialang.org/u/kimikage)\
**Post date:** [July 14, 2021, 5:08pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/14 "2021-07-14T17:08:00Z")

</div>

`unsafe_trunc` uses `RoundToZero`. It causes more numerical errors.

BTW, even if you are not using fastmath, LLVM may replace a muladd with a mul.  
[https://github.com/JuliaLang/julia/issues/34988](https://github.com/JuliaLang/julia/issues/34988)

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [July 14, 2021, 5:10pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/15 "2021-07-14T17:10:00Z")

</div>

The whole point of `@fastmath` is to use floating point trickery for more speed. It doesn’t even apply to non-floating point operations. One consequence of that trickery is arbitrary loss of precision, _which is what you can’t avoid when you assume floating point ops are associative, not matter how little you apply it._ Really, modes like `-Ofast` in Fortran or C for floating point ops were a well-meant feature that has horrid consequences when used in a language like julia, where code almost always gets compiled wholesale through all calls, i.e. without calling already-compiled code from a dynamically linked library.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 14, 2021, 5:46pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/16 "2021-07-14T17:46:12Z")

</div>

Options like `-Ofast` can affect global states. That can even happen upon loading a shared library and without actually calling any functions in it: [benchmarking ("exponent", "subnorm", "Float32") produced the DomainError when ApproxFun package is also used · Issue #253 · JuliaCI/BaseBenchmarks.jl · GitHub](https://github.com/JuliaCI/BaseBenchmarks.jl/issues/253)

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [July 14, 2021, 5:58pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/17 "2021-07-14T17:58:16Z")

</div>

Also note that setting the `math-mode=fast` as a startup flag to Julia is much more invasive than locally using fastmath. [julia/intrinsics.cpp at 23dbf169f0e4b831adfd20d666bedd1f669527ca · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/blob/23dbf169f0e4b831adfd20d666bedd1f669527ca/src/intrinsics.cpp#L735)

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [July 14, 2021, 6:16pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/18 "2021-07-14T18:16:34Z")

</div>

Yes, but I’m talking about gcc flags like `-Ofast`, which only affect the current compilation unit and don’t affect linked library code (which is already compiled, unlike the vast majority of linked julia code - here it’s far more common to compile the code of dependencies as well), as far as I’m aware.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 14, 2021, 6:18pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/19 "2021-07-14T18:18:44Z")

</div>

> [@Sukera](#):
>
> Yes, but I’m talking about gcc flags like `-Ofast`

Me too 🙂

> [@Sukera](#):
>
> which only affect the current compilation unit and don’t affect linked library code

Did you enjoy reading the whole debugging trip in the bug report I linked? 🙂

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [July 14, 2021, 6:26pm UTC](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619/20 "2021-07-14T18:26:40Z")

</div>

Yes, I’ve read that thread before - I have the (sometimes useless) ability to read way too many interesting issues 🙂

Let me quote you right back at yourself though:

> From [Subnormal number - Wikipedia](https://en.wikipedia.org/wiki/Denormal_number#Disabling_denormal_floats_at_the_code_level)
> 
> > well-behaved software should save and restore the denormal mode before returning to the caller or calling out to unsuspecting library/OS code

From [here](https://github.com/JuliaCI/BaseBenchmarks.jl/issues/253#issuecomment-573831731).

So I’d argue that the library _not_ resetting the flag was faulty behavior and a library compiled with `-Ofast` shouldn’t affect the caller either way 😛 (especially not influence how julia compiles its own code, which is the topic here, instead of getting incorrect results from correctly compiled code, as in that issue)

[Next page](https://discourse.julialang.org/t/whats-going-on-with-exp-and-math-mode-fast/64619.md?page=2)