I am trying to understand what happens when I subtract a Float32 from a Float64 where the numbers are “very small”. I understand what happens when both are Float64 julia> 1.401298464324817e-45 - 1.0e-45 4.012984643248171e-46 but if the second number is Float32 I get zero julia> 1.401298464324817…

[image] DNF: What happened? Actually 1.0f-45 is not 1.00000000000f-45 but rather julia> big(1.0f-45) 1.401298464324817070923729583289916131280261941876515771757068283889791082685861e-45 In particular, 1.0f-45 is internally rounded to a “subnormal” Float32 value that effectively has only a si…

[image] Motz: This surprised me. What is the general rule that leads to this result? The general rule when doing arithmetic on numbers of different types is that they each get promoted to a common type. For Float32 and Float64 the former gets promoted to Float64 and the latter stays unchanged…

See also [image] PSA: floating-point arithmetic Offtopic Sometimes people are surprised by the results of floating-point calculations such as julia> 5/6 0.8333333333333334 # shouldn't the last digit be 3? julia> 2.6 - 0.7 - 1.9 2.220…

[image] stevengj: In fact, for performance reasons, you sometimes want to just treat subnormal values as zero : This should be doubly treated with caution, as that setting affects every floating point calculation on the same process, as far as I remember. So changing this “for performance” i…

Setting underflows to zero is one of those things that “works great, except when it doesn’t”. I’ve never seen the performance benefit amount to much and @Sukera makes a very sound point about the risks. I tried this for a while and abandoned it.

[image] ctkelley: I’ve never seen the performance benefit amount to much and @Sukera makes a very sound point about the risks. As a counter-example, I’ve seen order-of-magnitude slowdowns from subnormals in real applications involving wave simulations, where the solutions can be exponentiall…

Of course, it’s not a black/white issue at all - but it’s also important to note that disabling subnormals is not a performance panacea. Whether computing them is actually slower or not strongly depends on how it’s implemented in hardware, and how many normal-subnormal interactions (which are usuall…

I had a case last week when a colleague found that some of her neural networks were several times slower than other ones when running inference in ONNXRunTime on CPU. It turned out that the slow ones had subnormals among the weights. There the solution was trivial though, just flush those coefficien…

Thank you @DNF and @stevengj for the explanation. Looking at the Julia documentation you referenced Treat-Subnormal-Numbers-as-Zeros I think I am safe to use set_zero_subnormals(true) in my code. My code has similar behaviour to the example in the documentation so I may get a performance improvement…

Offtopic(?): [image] stevengj: you sometimes want to just treat subnormal values as zero Subnormals could be documented better. They are, and I have the get_zero_subnormals() function in mind, which is in docs, just it not mentioned there. So I’m thinking is it very rare to actually use it? …

Subtract Float32 number from Float64 number - what's the rule?

New to Julia

Sukera August 29, 2023, 10:58am 3

Topic		Replies	Views
Slowdown due to subnormal float, coming from neural net training Performance	20	972	October 27, 2022
Quick probably basic question on Floats General Usage question , float	5	908	August 2, 2020
Why is 1.0f-99 not a subnormal number? General Usage question	1	141	October 2, 2024
Float32 performance fluctuating over time Performance	1	314	October 22, 2021
PSA: floating-point arithmetic Offtopic	14	7679	July 8, 2021

Subtract Float32 number from Float64 number - what's the rule?

Related topics