# Moving from \`Float64\` to \`Float32\` not improving performance

**URL:** <https://discourse.julialang.org/t/moving-from-float64-to-float32-not-improving-performance/88516>\
**Category:** Performance\
**Created:** [October 10, 2022, 9:43am UTC](https://discourse.julialang.org/t/moving-from-float64-to-float32-not-improving-performance/88516 "2022-10-10T09:43:55Z")\
**Posts on this page:** 6\
**Page:** 2

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [October 10, 2022, 1:31pm UTC](https://discourse.julialang.org/t/moving-from-float64-to-float32-not-improving-performance/88516/21 "2022-10-10T13:31:19Z")

</div>

> [@lmiq](#):
>
> Sometimes writing the loop is just simpler:

Realize also that the whole vectorized style is probably leading you astray here.

If I’m not mistaken, it looks to me like nearly the _whole_ body of the `while t < end_t` loop can be written as a _single_ loop, and that several of the temporary arrays (like `surface_gradient_s` and `new_thick`) can be eliminated _entirely_. Not only will the code be (probably) significantly faster, but I’m betting it will also be clearer.

Matlab/Numpy really trains people in a contorted style of programming.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [October 10, 2022, 2:45pm UTC](https://discourse.julialang.org/t/moving-from-float64-to-float32-not-improving-performance/88516/22 "2022-10-10T14:45:18Z")

</div>

> [@lmiq](#):
>
> it is not about memory usage, but how many operations the processors can do simultaneously.

LoopVectorization.jl targets Intel’s AVX, i.e. SIMD registers, and you get practically more operations with Float32 (i.e. lower precision), I think that’s the point of `@turbo`. Yes, you run as many _instructions_, but you work on twice as many operands in each register, so in effect more _operations_.

And even if not, when you’re memory-bound, not CPU-bound, the memory usage is important.

And you really should look towards the future, where there is no Float64 (EDIT: I mean in this new optimized instruction set, it would still exist as fallback in the default set), in Intel AMX instructions (not even Float32 except for accumulator)

> **[Advanced Matrix Extension (AMX) - x86 - WikiChip](https://en.wikichip.org/wiki/x86/amx)**
>
> Advanced Matrix Extension (AMX) is an x86 extension that introduces a matrix register file and new instructions for operating on matrices.

[https://www.corsix.org/content/contrasting-intel-amx-and-apple-amx](https://www.corsix.org/content/contrasting-intel-amx-and-apple-amx)

> Intel has an x64 instruction set extension called [AMX](https://en.wikichip.org/wiki/x86/amx), meanwhile Apple has a _totally different_ aarch64 instruction set extension also called [AMX](https://github.com/corsix/amx).  
> […]  
> Intel AMX: Multiplicands are always 32-bit, either `i8[4]` or `u8[4]` or `bf16[2]`, combined via dot product. Accumulators are always 32-bit, either `i32` or `u32` or `f32`.
> 
> Apple AMX: Multiplicands are scalars, any of `i8` / `u8` / `i16` / `u16` / `f16` / `f32` / `f64`. Accumulators are any of `i16` / `u16` / `i32` / `u32` / `f16` / `f32` / `f64`. Note `f16` (i.e. IEEE 754 half-precision with 5-bit exponent and 10-bit fraction) rather than `bf16` (8-bit exponent, 7-bit fraction).

I bit off-topic here, but my “ ~~At best Float32 will give you 2x speedup on CPUs~~ ” might be wrong, I at least see (more than 2x faster, for half the size, is likely an artifact of neural networks):

> **[Int4 Precision for AI Inference | NVIDIA Technical Blog](https://developer.nvidia.com/blog/int4-for-ai-inference/)**
>
> If there’s one constant in AI and deep learning, it’s never-ending optimization to wring every possible bit of performance out of a given platform. Many inference applications benefit from reduced…

> ### _INT4 Precision Can Bring an Additional 59% Speedup Compared to INT8_

> **[Convolutional Neural Network With INT4 Optimization](https://semiengineering.com/convolutional-neural-network-with-int4-optimization/)**
>
> INT8 provides better performance with comparable precision than floating point for AI inference. But when INT8 is unable to meet the desired performance with limited resources, INT4 optimization is the answer. This INT4 optimization achieves up to a...

> INT8 provides better performance with comparable precision than floating point for AI inference. But when INT8 is unable to meet the desired performance with limited resources, INT4 optimization is the answer. This INT4 optimization achieves up to a 77% performance boost on real hardware in comparison with the current INT8 solution.

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [October 10, 2022, 2:54pm UTC](https://discourse.julialang.org/t/moving-from-float64-to-float32-not-improving-performance/88516/23 "2022-10-10T14:54:16Z")

</div>

`Float32` can also help with cache locality - at various array sizes `Float64` arrays may no longer fit in a cache, moving access from L1 to L2, or L2 to L3 caches, while the `Float32` version of the same array will still fit in the smaller more localized cache.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [October 10, 2022, 3:09pm UTC](https://discourse.julialang.org/t/moving-from-float64-to-float32-not-improving-performance/88516/24 "2022-10-10T15:09:16Z")

</div>

Float64 isn’t going anywhere. That said, HPC is starting to augment Float64 with lower precisions in some places. One example of this is iterative methods. Traditionally, all the computation would be done in full Float64 precision, but recently there have been good results demonstrated from doing initial steps in lower precision and moving to higher precision as the error decreases.

---

<div class="post-metadata">

**Author:** ![JordiBolibar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jordibolibar/32/24307_2.png) [@JordiBolibar](https://discourse.julialang.org/u/JordiBolibar)\
**Post date:** [October 10, 2022, 3:15pm UTC](https://discourse.julialang.org/t/moving-from-float64-to-float32-not-improving-performance/88516/25 "2022-10-10T15:15:42Z")

</div>

> Matlab/Numpy really trains people in a contorted style of programming.

Just for clarification, this little experiment started as a translation from a Python script. Just for fun 🙂 So this explains why many parts still look quite a lot like Python.

I already managed to get a 30x acceleration, so I’m more than happy so far with that for my needs. Readability is also important in my context (i.e. including other Earth sciences users). So some of the suggested improvements are just a bit too convoluted to make sense for this case.

Anyways, thanks everybody for the feedback. I really learnt quite a lot with this.

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [October 10, 2022, 5:20pm UTC](https://discourse.julialang.org/t/moving-from-float64-to-float32-not-improving-performance/88516/26 "2022-10-10T17:20:35Z")

</div>

> [@Palli](#):
>
> And you really should look towards the future, where there is no Float64 (EDIT: I mean in this new optimized instruction set, it would still exist as fallback in the default set), in Intel AMX instructions (not even Float32 except for accumulator)

For me, AMX is not the future until it supports `Float64` or some type with comparable properties.

[Previous page](https://discourse.julialang.org/t/moving-from-float64-to-float32-not-improving-performance/88516.md?page=1)