# Different Float64 sum on different architectures

**URL:** <https://discourse.julialang.org/t/different-float64-sum-on-different-architectures/46949>\
**Category:** General Usage\
**Created:** [September 20, 2020, 9:16am UTC](https://discourse.julialang.org/t/different-float64-sum-on-different-architectures/46949 "2020-09-20T09:16:26Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![greg\_plowman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/greg_plowman/32/8100_2.png) [@greg\_plowman](https://discourse.julialang.org/u/greg_plowman)\
**Post date:** [September 20, 2020, 9:16am UTC](https://discourse.julialang.org/t/different-float64-sum-on-different-architectures/46949/1 "2020-09-20T09:16:26Z")

</div>

I sometimes get different results for summing a vector of `Float64` numbers on different Windows machines. The vectors are identical, so it seems `sum` can produce different results across hardware architectures (even if os is always Windows).

Is this expected?

Is there a way to guarantee the same results?

Would `sum_kbn` from `KahanSummation`, or `xsum` from `Xsum` guarantee the same result?  
Both of these alternative sums produce identical results across machines on my particular data.

[https://github.com/JuliaMath/KahanSummation.jl](https://github.com/JuliaMath/KahanSummation.jl)  
[https://github.com/JuliaMath/Xsum.jl](https://github.com/JuliaMath/Xsum.jl)

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [September 20, 2020, 9:37am UTC](https://discourse.julialang.org/t/different-float64-sum-on-different-architectures/46949/2 "2020-09-20T09:37:41Z")

</div>

Why do you need the exact same result?

> [@greg\_plowman](#):
>
> Is this expected?

Yes, the results will be hardware dependent. Based on the width of the CPU’s vector registers, a different order of adding the numbers will be optimal. These different orders can slightly change rounding.

> [@greg\_plowman](#):
>
> Is there a way to guarantee the same results?

You can use `foldl(+, x)` instead of `sum(x)` to guarantee a particular order.  
Most Kahan Summations implementations aren’t SIMD and thus will be exactly the same across architectures ([AccurateArithmetic](https://github.com/JuliaMath/AccurateArithmetic.jl) is), and of course Xsum promises to be exactly rounded, so you obviously shouldn’t see any rounding differences there.

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [September 20, 2020, 10:44am UTC](https://discourse.julialang.org/t/different-float64-sum-on-different-architectures/46949/3 "2020-09-20T10:44:36Z")

</div>

See here for a simple example:

> [@Different inner product if vector type is not set](https://discourse.julialang.org/t/different-inner-product-if-vector-type-is-not-set/35359/8):
>
> I think you do, but you may be surprised to know that floating point addition is not associative (due to, essentially, rounding). julia\> @show (-1e30+1e30)+1.0 -1e30+(1e30+1.0); (-1.0e30 + 1.0e30) + 1.0 = 1.0 -1.0e30 + (1.0e30 + 1.0) = 0.0

Basically, summation is not associative and there is no guarantee for the order of execution of the sum.

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [September 20, 2020, 10:55am UTC](https://discourse.julialang.org/t/different-float64-sum-on-different-architectures/46949/4 "2020-09-20T10:55:07Z")

</div>

[Here](https://discourse.julialang.org/t/array-ordering-and-naive-summation/1929) is a fun example of the lack of associativity (thanks to Stefan Karpinski):

```julia
function sumsto(x::Float64)
    0 <= x < exp2(970) || throw(ArgumentError("sum must be in [0,2^970)"))
    n, p₀ = Base.decompose(x) # integers such that `n*exp2(p₀) == x`
    [floatmax(); [exp2(p) for p in -1074:969 if iseven(n >> (p-p₀))]
    -floatmax(); [exp2(p) for p in -1074:969 if isodd(n >> (p-p₀))]]
end

```

Example:

```julia
julia> x = sumsto(2.3);

julia> y = sumsto(1e18);

julia> sort(x) == sort(y)
true

julia> foldl(+,x)
2.3

julia> foldl(+,y)
1.0e18

```

---

<div class="post-metadata">

**Author:** ![greg\_plowman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/greg_plowman/32/8100_2.png) [@greg\_plowman](https://discourse.julialang.org/u/greg_plowman)\
**Post date:** [September 20, 2020, 11:48am UTC](https://discourse.julialang.org/t/different-float64-sum-on-different-architectures/46949/5 "2020-09-20T11:48:16Z")

</div>

Thanks for the responses.  
Very interesting and enlightening!

It seems the same sum result can be guaranteed with:  
`foldl(+, x) # accumulates rounding error`  
`xsum(x) # no rounding error`

I’ll just have to add a method to allow `dims` argument for arrays.  
I’m surprised a `dims` kwarg isn’t defined for `foldl` (or `reduce`)

---

<div class="post-metadata">

**Author:** ![greg\_plowman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/greg_plowman/32/8100_2.png) [@greg\_plowman](https://discourse.julialang.org/u/greg_plowman)\
**Post date:** [September 20, 2020, 12:24pm UTC](https://discourse.julialang.org/t/different-float64-sum-on-different-architectures/46949/6 "2020-09-20T12:24:14Z")

</div>

Is there a way to disable SIMD and/ or whatever else is reordering the additions, so then standard `sum` executes deterministically independent of processor?  
A `@nosimd` macro perhaps?

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [September 20, 2020, 1:10pm UTC](https://discourse.julialang.org/t/different-float64-sum-on-different-architectures/46949/7 "2020-09-20T13:10:51Z")

</div>

You can get generators over rows and columns with `eachrow` and `eachcol` respectively.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [September 20, 2020, 1:58pm UTC](https://discourse.julialang.org/t/different-float64-sum-on-different-architectures/46949/8 "2020-09-20T13:58:51Z")

</div>

While that sounds appealing, it actually makes the summation both slower and less accurate. If you really want that just use `foldl(+, a)`.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [September 20, 2020, 2:37pm UTC](https://discourse.julialang.org/t/different-float64-sum-on-different-architectures/46949/9 "2020-09-20T14:37:51Z")

</div>

> [@Elrod](#):
>
> Why do you need the exact same result?

I think this is the key question for all these discussions (wrt float arithmetic, reproducibility of random sequences, etc).

If the results are close, this should not matter and things should be compared with some variant of `isapprox`.

If they are not, and appear to be very sensitive to random seeds or low-level questions of floating point arithmetic, they are not to be trusted anyway.

---

<div class="post-metadata">

**Author:** ![greg\_plowman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/greg_plowman/32/8100_2.png) [@greg\_plowman](https://discourse.julialang.org/u/greg_plowman)\
**Post date:** [September 20, 2020, 9:31pm UTC](https://discourse.julialang.org/t/different-float64-sum-on-different-architectures/46949/10 "2020-09-20T21:31:31Z")

</div>

> [@StefanKarpinski](#):
>
> While that sounds appealing, it actually makes the summation both slower and less accurate. If you really want that just use `foldl(+, a)` .

Yes, of course.

I think my use of standard `sum` did not convey my intention.  
I did not mean for `Base.sum` to be re-implemented for everyone.

What I meant was could I use `@nosimd sum(a)` to get a deterministic sum. The advantage over `foldl(+, a)` is the extra methods available to `sum`. In particular, `sum(a, dims=x)`.

This is no biggy though, I definitely can work with `foldl(+, a)`.

Thanks again for all the responses.