# Why doesn’t Julia fma automatically?

**URL:** <https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982>\
**Category:** General Usage\
**Created:** [October 13, 2023, 3:31am UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982 "2023-10-13T03:31:53Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![mj2984](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mj2984/32/43833_2.png) [@mj2984](https://discourse.julialang.org/u/mj2984)\
**Post date:** [October 13, 2023, 3:31am UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/1 "2023-10-13T03:31:53Z")

</div>

> [@Julia & Mojo Mandelbrot Benchmark](https://discourse.julialang.org/t/julia-mojo-mandelbrot-benchmark/103638/5):
>
> changing `z = z * z + c` to `z = muladd(z, z, c)` gets about 25% faster right off the bat

Is there any reason why the compiler doesn’t optimize these? I feel like having to do this is a detriment to high level programming, and especially the math like syntax that I can write in Julia (same expression I use in my paper).

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [October 13, 2023, 3:34am UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/2 "2023-10-13T03:34:00Z")

</div>

fma changes results because IEEE floating point numbers are weird.

---

<div class="post-metadata">

**Author:** ![ParadaCarleton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paradacarleton/32/20005_2.png) [@ParadaCarleton](https://discourse.julialang.org/u/ParadaCarleton)\
**Post date:** [October 14, 2023, 8:20pm UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/3 "2023-10-14T20:20:57Z")

</div>

Yes, it changes them to be more accurate (only 1 roundoff instead of 2), so this still really doesn’t make sense.

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [October 14, 2023, 8:23pm UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/4 "2023-10-14T20:23:35Z")

</div>

> Yes, it changes them to be more accurate (only 1 roundoff instead of 2), so this still really doesn’t make sense.

It changes them to be _unpredictable_

more accurate or not is mostly irrelevant here

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [October 14, 2023, 10:30pm UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/5 "2023-10-14T22:30:24Z")

</div>

I guess this was already discussed lots of times. One relevant thread:

> [@X \* y + z does not automatically use FMA instruction](https://discourse.julialang.org/t/x-y-z-does-not-automatically-use-fma-instruction/100640):
>
> I am new to Julia and was wondering why the code x \* y + z does not use fmadd where all values are floats, but instead fmul followed by fadd. On the other hand, if I use the @fastmath decorator, the @code\_native changes to use the fmadd instruction. My question is, if FMA is faster and more precise, why won’t it use fmadd to begin with on supported machines? Machine: M1 apple silicon on 13.4 Ventura Julia: 1.9.1 Optimization: -O3 and -O2

The TLDR is **because of correctness** , it is assumed that the user uses the exact FP operations that are necessary, and changing the operations in such a way that the result is changed would be incorrect. This is consistent with other optimizations that compilers (not just Julia) do, in general an optimization is allowed if it doesn’t change any observable result.

The thread also gives specific examples of FP code that can break in such cases. Search for “error-free transforms” or “multi-word arithmetics” or “floating-point expansions” for more. An example Julia package that relies on the current behavior is MultiFloats. Many implementations of transcendental math functions would also be broken.

---

<div class="post-metadata">

**Author:** ![ParadaCarleton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paradacarleton/32/20005_2.png) [@ParadaCarleton](https://discourse.julialang.org/u/ParadaCarleton)\
**Post date:** [October 15, 2023, 1:54am UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/6 "2023-10-15T01:54:44Z")

</div>

> [@adienes](#):
>
> It changes them to be _unpredictable_

They’re perfectly predictable; whenever you see `A x + b`, it gives the same answer, consistently, so people familiar with FMA can predict exactly what it will do. People who aren’t familiar with FMA can also predict exactly what it will do: It will calculate `A x + b`, and anything beyond that doesn’t matter to them, and because if your code relies on whether `A x + b` rounds exactly once or exactly twice, it’s bad code.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [October 15, 2023, 1:59am UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/7 "2023-10-15T01:59:11Z")

</div>

The problem is `a*b+c*d` there are two ways to fuse this that will give different answers.

---

<div class="post-metadata">

**Author:** ![ParadaCarleton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paradacarleton/32/20005_2.png) [@ParadaCarleton](https://discourse.julialang.org/u/ParadaCarleton)\
**Post date:** [October 15, 2023, 1:59am UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/8 "2023-10-15T01:59:21Z")

</div>

> [@nsajko](#):
>
> This is consistent with other optimizations that compilers (not just Julia) do,

It’s consistent with _some_ other compilers (like C or C++) but not with others (like Haskell or every Python DSL for machine learning).

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [October 15, 2023, 2:06am UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/9 "2023-10-15T02:06:01Z")

</div>

This post was temporarily hidden by the community for possibly being off-topic, unfocused, inappropriate, or spammy.

---

<div class="post-metadata">

**Author:** ![ParadaCarleton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paradacarleton/32/20005_2.png) [@ParadaCarleton](https://discourse.julialang.org/u/ParadaCarleton)\
**Post date:** [October 15, 2023, 2:11am UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/10 "2023-10-15T02:11:01Z")

</div>

Yeah, you’re right that this is an unfortunate case. That being said, I think it still makes sense to just pick a standard interpretation, since `(a*b + (c*d))` or vice-versa will both usually be a bit more accurate than `(a*b) + (c*d)`. Alternatively, we could choose to interpret anything more complex than `a * b + c` in strict numerical order, and just stick to the easy win of optimizing `a * b + c`.

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [October 15, 2023, 8:14am UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/11 "2023-10-15T08:14:31Z")

</div>

```julia
f(a,b) = a * b
f(a,b) + c

```

would then depend on inlining? Are floating point results allowed to depend on inlining?

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [October 15, 2023, 10:19am UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/12 "2023-10-15T10:19:18Z")

</div>

Not sure what you mean. If I make a program that prints “hello world”, in whichever language, I wouldn’t expect it to print “hello Earth”. Unless that was somehow explicitly allowed by some documented semantics, of course.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [October 15, 2023, 7:50pm UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/13 "2023-10-15T19:50:30Z")

</div>

> [@jar1](#):
>
> ```julia
> f(a,b) = a * b
> f(a,b) + c
> 
> ```
> 
> would then depend on inlining?

Or rather would depend on inlining NOT happening, or at least if inlining done, not behaving as if written as one function. And that would be reasonable to do.

---

<div class="post-metadata">

**Author:** ![ParadaCarleton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paradacarleton/32/20005_2.png) [@ParadaCarleton](https://discourse.julialang.org/u/ParadaCarleton)\
**Post date:** [October 16, 2023, 1:54am UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/14 "2023-10-16T01:54:53Z")

</div>

> [@nsajko](#):
>
> Not sure what you mean. If I make a program that prints “hello world”, in whichever language, I wouldn’t expect it to print “hello Earth”. Unless that was somehow explicitly allowed by some documented semantics, of course.

Because changing exact floating-point values isn’t (and can’t) be considered a breaking change, or it would imply that we could never change any numerical function. If you figure out an optimization that lets you double the speed of `sum`, too bad, can’t be merged, it’s a breaking change. Upgrading LLVM would be a breaking change too (because LLVM doesn’t guarantee exact floating-point results). Pretty much any kind of optimization would end up becoming illegal (e.g. broadcast fusion would have to be thrown out as well).

Whenever I write `a * x + b`, I’m already asking for an abstract thing, and accepting that what I get will only be approximately correct (that’s how floats work!); I can’t get hung up on the exact values of the last one or two bits.

---

<div class="post-metadata">

**Author:** ![ParadaCarleton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paradacarleton/32/20005_2.png) [@ParadaCarleton](https://discourse.julialang.org/u/ParadaCarleton)\
**Post date:** [October 16, 2023, 2:01am UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/15 "2023-10-16T02:01:11Z")

</div>

> [@jar1](#):
>
> would then depend on inlining? Are floating point results allowed to depend on inlining?

Yes, they are (and already do in a few situations). The exact value of the last bit isn’t guaranteed.

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [October 16, 2023, 2:03am UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/16 "2023-10-16T02:03:19Z")

</div>

pretty false equivalency between changing `sum` and changing `+`

they have different contracts

---

<div class="post-metadata">

**Author:** ![ParadaCarleton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paradacarleton/32/20005_2.png) [@ParadaCarleton](https://discourse.julialang.org/u/ParadaCarleton)\
**Post date:** [October 16, 2023, 2:25am UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/17 "2023-10-16T02:25:43Z")

</div>

Neither of them guarantees exact floating-point values in their contract (and they can’t, because LLVM doesn’t guarantee exact floating-point values). But if you want a more analogous case, `A * B * C` chooses the execution order that has the best performance, rather than exactly left-to-right.

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [October 16, 2023, 2:45am UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/18 "2023-10-16T02:45:01Z")

</div>

> `A * B * C` chooses the execution order that has the best performance, rather than exactly left-to-right.

and this is indeed already a controversial design choice itself. but in any case `(A*B)*C` does not. where do you draw the line? if you allow rewriting `a*b + c`, why not rewrite more complicated arithmetic? will parentheses save you? or only function barriers? or neither?

> because LLVM doesn’t guarantee exact floating-point values

I am not an LLVM expert, but I’d venture a guess that `*` and `+` mean one thing and one thing only. without fast math flags, doesn’t it follow IEEE754?

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [October 16, 2023, 3:02pm UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/19 "2023-10-16T15:02:26Z")

</div>

It’s very unclear that LLVM does NOT fuse by default, or at least isn’t allowed to, or that IEEE disallows that. IEEE demands FMA _available_ now; since 2008 revision? At least 2019 revision. LLVM is an implementation detail, Clang based on it does FMA by default:

[https://clang.llvm.org/docs/UsersManual.html#cmdoption-ffp-contract](https://clang.llvm.org/docs/UsersManual.html#cmdoption-ffp-contract)

> The C standard permits intermediate floating-point results within an expression to be computed with more precision than their type would normally allow. This permits operation fusing, and Clang takes advantage of this by default. This behavior can be controlled with the FP\_CONTRACT and clang fp contract pragmas. Please refer to the pragma documentation for a description of how the pragmas interact with this option.
> 
> Valid values are:
> 
> - fast (fuse across statements disregarding pragmas, default for CUDA)
> - on (fuse in the same statement unless dictated by pragmas, default for languages other than CUDA/HIP)
> - off (never fuse)
> - fast-honor-pragmas (fuse across statements unless dictated by pragmas, default for HIP)

> **[IEEE 754 |](https://en.wikipedia.org/wiki/IEEE_754#2019)**
>
> The first standard for floating-point arithmetic, IEEE 754-1985, was published in 1985. It covered only binary floating-point arithmetic.
> A new version, IEEE 754-2008, was published in August 2008, following a seven-year revision process, chaired by Dan Zuras and edited by Mike Cowlishaw. It replaced both IEEE 754-1985 (binary floating-point arithmetic) and IEEE 854-1987 Standard for Radix-Independent Floating-Point Arithmetic. The binary formats in the original standard are included in this new...

It’s not clear to me IEEE demands fused _used_, i.e. not just available. It’s very unlikely it demands it used, but neither clear it demands it NOT used.

So it’s up to Julia, and it can and would still claim IEEE compatibility. I think there might be exceptions to Julia wanting to do this, e.g. for complex values. But otherwise I think we should do the same “on” the default (and I guess “fast” for CUDA), as above. But that also means we need some pragma equivalent to opt out. I suppose that would be a macro in Julia. I think we should do this to compete with C/C++ (in recent compilers; and Mojo also, I suppose it defaults to fused too).

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [October 16, 2023, 3:23pm UTC](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982/20 "2023-10-16T15:23:31Z")

</div>

> [@ParadaCarleton](#):
>
> Whenever I write `a * x + b`, I’m already asking for an abstract thing, and accepting that what I get will only be approximately correct (that’s how floats work!); I can’t get hung up on the exact values of the last one or two bits.

That is **not** how floats work. They have an exact arithmetic with a limited number of “decimal” places. They just snap to the nearest representable value if you land somewhere that’s in between the values they can represent — but that value is an exact number. It’s not a “fuzzy” thing. There are many algorithms that explicitly depend upon this.

[Next page](https://discourse.julialang.org/t/why-doesn-t-julia-fma-automatically/104982.md?page=2)