# Sum of float64 vector gives slightly incorrect answer
**URL:** https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577
**Category:** Performance
**Tags:** question
**Created:** [January 24, 2018, 10:38pm UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577 "2018-01-24T22:38:46Z")
**Posts on this page:** 20
**Page:** 2
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [January 25, 2018, 6:03pm UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/21 "2018-01-25T18:03:39Z")
> [@John\_Gibson](#):
>
> Seems like it’s time for a “PSA: floating-point arithmetic”, like Thomas\_Papp’s “PSA: How to quote code with backticks”. I’ll write one unless someone beats me to it.
Unfortunately, floating-point arithmetic is complicated when you get down to brass tacks, and it is tricky to write something both short and correct.
If you google for “floating-point introduction” or similar, you’ll find lots of tutorials online. Many of them are written for people with computer-science backgrounds and are rather technical, e.g. this one is well known:
[What Every Computer Scientist Should Know About Floating-Point Arithmetic](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html)
Here are a couple of gentler introductions, although both of them spend more time than I would like on fixed-point arithmetic:
[Floating Point Demystified, Part 1](http://blog.reverberate.org/2014/09/what-every-computer-programmer-should.html)
[http://homerreid.dyndns.org/teaching/18.330/Notes/MachineArithmetic.pdf](http://homerreid.dyndns.org/teaching/18.330/Notes/MachineArithmetic.pdf)
---
### Author:  [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)
#### Post date: [January 25, 2018, 6:17pm UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/22 "2018-01-25T18:17:29Z")
Agreed, don’t start with the Goldberg article: it really isn’t that good, and a lot of the stuff isn’t that relevant (e.g. every computer nowadays uses guard bits).
I personally think the first few chapters of [Nick Higham’s book](http://www.maths.manchester.ac.uk/~higham/asna/index.php) give a really good introduction.
---
### Author:  [@John\_Gibson](https://discourse.julialang.org/u/John_Gibson)
#### Post date: [January 25, 2018, 6:33pm UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/23 "2018-01-25T18:33:52Z")
I was thinking it would be useful to explain
```julia
julia> 5/6
0.8333333333333334
julia> 0.4 - 0.3 - 0.1
2.7755575615628914e-17
```
in a paragraph getting no deeper than binary rep, 64 bits, 52 (53) for mantissa, and how this is not a Juliaism but an IEE standard burned into hardware, with links to deeper explanations and mentions of alternatives like `BigFloat`. Seems like it’s needed twice a week lately.
---
### Author:  [@nalimilan](https://discourse.julialang.org/u/nalimilan)
#### Post date: [January 25, 2018, 10:34pm UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/24 "2018-01-25T22:34:35Z")
This should definitely be added to the FAQ. R has [FAQ 7.31](https://cran.r-project.org/doc/FAQ/R-FAQ.html#Why-doesn_0027t-R-think-these-numbers-are-equal_003f) for this and it’s used everytime somebody raises this issue.
---
### Author:  [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)
#### Post date: [January 25, 2018, 11:25pm UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/25 "2018-01-25T23:25:22Z")
If the data is not sensitive, would you create a [gist](http://gist.com) with 7 or so rows and post the link. A simple, runnable sequence that does the transform which has been a bother would be helpful, too.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [January 26, 2018, 7:08am UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/26 "2018-01-26T07:08:04Z")
Great idea. On a single page, you can explain the _problem_, not the solution. Eg something along the lines of
1. how computers represent floating point numbers,
2. a few examples like sums that don’t behave like their decimal counterpart, or numbers that are _per se_ unrepresentable eg the `5/6` that came up in another thread,
3. Julia’s built-in facilities for mitigating, but not eliminating the problem (eg fancy algs in `sum`, `BigFloat`),
4. further reading (just a few nice articles)
Very little would go a long way. Just as a substitute for explaining this every time.
---
### Author:  [@GantZA](https://discourse.julialang.org/u/GantZA)
#### Post date: [January 27, 2018, 5:43pm UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/27 "2018-01-27T17:43:31Z")
Been trying to use the DecFP package now, but any numbers of type Dec64 (other DecXX im assuming as well) don’t work with the `cond()` or other eigen related functions. Any ways around this?
---
### Author:  [@jebej](https://discourse.julialang.org/u/jebej)
#### Post date: [January 27, 2018, 6:38pm UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/28 "2018-01-27T18:38:20Z")
If you are doing linear algebra, you should be OK with Float64 numbers. As you noticed, most linear algebra routines work only with standard base 2 numbers (Float64 and Float32), because that’s the fastest with computer architectures.
In any case, what you are doing _most likely_ does not require decimal numbers.
As explained above, decimal numbers will incur the same representation issue as you had originally posted. This is because most real numbers cannot be exactly represented in any basis (be it 2 or 10 or 36) with a finite amount of memory. It was suggested to use DecFP because you were originally concerned with addition of decimal numbers. More complex operations will certainly return numbers not representable in finite precision base 10, in which case DecFP becomes inappropriate.
If you tell us more about your application we might be able to suggest better solutions.
---
### Author:  [@GantZA](https://discourse.julialang.org/u/GantZA)
#### Post date: [January 27, 2018, 7:10pm UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/29 "2018-01-27T19:10:55Z")
I’ve just tested using BigFloat and the LinearAlgebra package which provides the missing functionality. Unfortunately the slow down is quite a lot and the increased accuracy was only present after the 10th decimal or so, so as you said I will be fine with Float64.
My application is a portfolio optimization and backtest, the reason I started this thread was due to a weird round-off occurring which has since disappeared (weird in that it occurred rarely and but had a pattern). The conversation above made me want to check how much of an impact arbitrary precision decimals may have and its not significant (for financial application), even with a large dataset and many iterations
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [January 28, 2018, 7:51am UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/30 "2018-01-28T07:51:27Z")
> [@GantZA](#):
>
> using BigFloat and the LinearAlgebra package which provides the missing functionality.
I don’t think it does that. It just has a smaller error (the magnitude of which you have some control over) that you may not notice.
---
### Author:  [@GantZA](https://discourse.julialang.org/u/GantZA)
#### Post date: [January 28, 2018, 8:02am UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/31 "2018-01-28T08:02:24Z")
By “missing functionality” I mean the ` cond(Matrix)` function which didn’t work with Dec64 or BigFloat matrices. The LinearAlgebra package provides the cond and other required functions for BigFloat but not Dec64.
Does this clear it up?
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [January 28, 2018, 8:10am UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/32 "2018-01-28T08:10:18Z")
Yes, I thought you meant that it gets rid of floating point inaccuracy.
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [January 28, 2018, 2:17pm UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/33 "2018-01-28T14:17:19Z")
> [@GantZA](#):
>
> By “missing functionality” I mean the cond(Matrix) function which didn’t work with Dec64 or BigFloat matrices. The LinearAlgebra package provides the cond and other required functions for BigFloat but not Dec64.
The built-in `cond` function relies on the SVD computation provided by the external library LAPACK, which only works with `Float32` and `Float64`. To do this for other number types, you can use the [GitHub - JuliaLinearAlgebra/GenericSVD.jl: Singular Value Decomposition for generic number types](https://github.com/JuliaMath/GenericSVD.jl) package.
---
### Author:  [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)
#### Post date: [January 28, 2018, 6:00pm UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/34 "2018-01-28T18:00:50Z")
What has happened to that `cond` function, has it been moved out to `stdlib` along with `LinAlg`?
How does the generic code in `GenericSVD` compare performance-wise when used with `Float64`?
It hasn’t yet been updated to the v0.6/v0.7 syntax though.
---
### Author:  [@davide](https://discourse.julialang.org/u/davide)
#### Post date: [April 26, 2019, 8:57am UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/35 "2019-04-26T08:57:36Z")
A related problem, which I find disconcerting: floating point arithmetic has no commutative property.
```julia
julia> 0.1 + 0.1 + 100.0 == 100.0 + 0.1 + 0.1
false
julia> a = [0.1, 0.1, 100.0];
julia> sum(a)
100.2
julia> sum(reverse(a))
100.19999999999999
```
Is there a reliable solution for this or it is just one of the many flavors in which floating point rounding inaccuracy unavoidably emerges?
---
### Author:  [@ffevotte](https://discourse.julialang.org/u/ffevotte)
#### Post date: [April 26, 2019, 10:01am UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/36 "2019-04-26T10:01:25Z")
This is not a loss of commutativity, but rather of associativity, i.e. for any given binary operation, the order of both operands is irrelevant.
```julia
julia> 100.0 + 0.1 == 0.1 + 100.0
true
```
However, for any sequence of operations (e.g. additions), the global order of operations matters (i.e. it is necessary to parenthesize all basic operations to uniquely specify the result)
```julia
julia> 100. + 0.1 + 0.1 == 0.1 + 0.1 + 100.
false
julia> 100. + 0.1 + 0.1 == 0.1 + (0.1 + 100.)
true
```
> [@davide](#):
>
> Is there a reliable solution for this or it is just one of the many flavors in which floating point rounding inaccuracy unavoidably emerges?
There are many reliable solutions to ensure that you always get a correctly rounded result from your sum (i.e. a result that is rounded to the nearest representable floating-point value, in the same way that the exact mathematical result would). But all these solutions come at a cost.
Using rational numbers based on big integers is a relatively simple example of such techniques. If you are specifically computing a sum, there exist specific accurate algorithms (for example Kulisch accumurators or, more recently, Demmel’s accurate summation algorithm)
---
### Author:  [@davide](https://discourse.julialang.org/u/davide)
#### Post date: [April 29, 2019, 8:15am UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/37 "2019-04-29T08:15:54Z")
Thanks for the explanation. I see, it’s a lack of associativity.
A fast solution I adopted to get consistent results (not exact results) is to sort the vectors before applying a function, e.g. summation:
```julia
julia> a = [0.1, 0.1, 100.0];
julia> b = [100, 0.1, 0.1];
julia> sum(sort(a)) == sum(sort(b))
true
```
This works also with other functions (I spotted the problem calculating entropies of discrete distributions, where a different order of the same values resulted in different entropies).
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [April 29, 2019, 8:18am UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/38 "2019-04-29T08:18:37Z")
> [@davide](#):
>
> get consistent results
I am curious why you need that. Also note that the exact results may still depend on the software and hardware environment, so it is only consistent in one particular setting.
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [April 29, 2019, 8:30am UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/39 "2019-04-29T08:30:56Z")
It would probably be a bit better to separate into positive and negative values and sort each half by ascending absolute value. But as @Tamas_Papp said, this will only get you so far and may still not produce portable results.
---
### Author:  [@davide](https://discourse.julialang.org/u/davide)
#### Post date: [April 29, 2019, 9:11am UTC](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577/40 "2019-04-29T09:11:16Z")
I was comparing entropies (and other measures) of probability distributions, and calculating with them (e.g. H1 - H2, and getting not zero values when I expected zero).
What I would like is consistency within my setting and comparisons, so no need for portability (at the moment).
Thanks @StefanKarpinski for the suggestion.
[Previous page](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577.md?page=1)
[Next page](https://discourse.julialang.org/t/sum-of-float64-vector-gives-slightly-incorrect-answer/8577.md?page=3)