# How to match performance of sum(A, dims=1)?

**URL:** <https://discourse.julialang.org/t/how-to-match-performance-of-sum-a-dims-1/87543>\
**Category:** Performance\
**Created:** [September 21, 2022, 4:32am UTC](https://discourse.julialang.org/t/how-to-match-performance-of-sum-a-dims-1/87543 "2022-09-21T04:32:02Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![torrance](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/torrance/32/38990_2.png) [@torrance](https://discourse.julialang.org/u/torrance)\
**Post date:** [September 21, 2022, 4:32am UTC](https://discourse.julialang.org/t/how-to-match-performance-of-sum-a-dims-1/87543/1 "2022-09-21T04:32:02Z")

</div>

Well, this is a bit embarrassing… I’m writing up a set of slides to present to colleagues about Julia, and I wanted to show an example how explicit looping and so on is fine and dandy and as fast as any other language construct.

However! - I actually can’t get my `mysum()` function to match Julia’s `sum(A, dims=1)` function (summing a 2D array along the first axis):

```julia
arr = rand(1000, 100_000)

@btime sum($arr, dims=1);
# ~ 56 ms

function mysum(arr)
    result = zeros(eltype(arr), size(arr, 2))

    for j in axes(arr, 2)
        for i in axes(arr, 1)
            result[j] += arr[i, j]
        end
    end
    return result
end

@btime mysum($arr);
# ~ 80 ms

```

I’ve checked the obvious: its column-major iteration, there’s no allocations (except the initial result array), it’s type stable. I’ve also tried initialising an undef `results` array, and summing with a temporary variable for each row (in case it was the memory accesses on the heap that were slowing me down), also with no effect.

How do I speed this up?

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [September 21, 2022, 4:35am UTC](https://discourse.julialang.org/t/how-to-match-performance-of-sum-a-dims-1/87543/2 "2022-09-21T04:35:55Z")

</div>

Did you try `@inbounds`?

---

<div class="post-metadata">

**Author:** ![torrance](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/torrance/32/38990_2.png) [@torrance](https://discourse.julialang.org/u/torrance)\
**Post date:** [September 21, 2022, 4:36am UTC](https://discourse.julialang.org/t/how-to-match-performance-of-sum-a-dims-1/87543/3 "2022-09-21T04:36:54Z")

</div>

Yes, indeed. No effect.

---

<div class="post-metadata">

**Author:** ![torrance](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/torrance/32/38990_2.png) [@torrance](https://discourse.julialang.org/u/torrance)\
**Post date:** [September 21, 2022, 4:46am UTC](https://discourse.julialang.org/t/how-to-match-performance-of-sum-a-dims-1/87543/4 "2022-09-21T04:46:43Z")

</div>

Just upgraded from 1.8.0 to 1.8.1, and my version is now twice as slow. That’s a really curious performance degradation.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [September 21, 2022, 4:47am UTC](https://discourse.julialang.org/t/how-to-match-performance-of-sum-a-dims-1/87543/5 "2022-09-21T04:47:54Z")

</div>

have you tried `@turbo`?

---

<div class="post-metadata">

**Author:** ![torrance](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/torrance/32/38990_2.png) [@torrance](https://discourse.julialang.org/u/torrance)\
**Post date:** [September 21, 2022, 4:53am UTC](https://discourse.julialang.org/t/how-to-match-performance-of-sum-a-dims-1/87543/6 "2022-09-21T04:53:25Z")

</div>

@Oscar_Smith i have yes, but I wouldn’t expect it to have an effect since this is a reduction and SIMD doesn’t easily apply.

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [September 21, 2022, 5:01am UTC](https://discourse.julialang.org/t/how-to-match-performance-of-sum-a-dims-1/87543/7 "2022-09-21T05:01:00Z")

</div>

I think SIMD applies. I recall finding this out while preparing lecture for students last year. But `@simd` was sufficient.

Here is the link to source of our lecture

> <https://github.com/JuliaTeachingCTU/Scientific-Programming-in-Julia/blob/master/docs/src/lecture_05/lecture.md>

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [September 21, 2022, 5:05am UTC](https://discourse.julialang.org/t/how-to-match-performance-of-sum-a-dims-1/87543/8 "2022-09-21T05:05:26Z")

</div>

`@turbo` works for me.

```julia
julia> function turbosum(arr)
           result = zeros(eltype(arr), size(arr, 2))
       
           @turbo for j in axes(arr, 2)
               for i in axes(arr, 1)
                   result[j] += arr[i, j]
               end
           end
           return result
       end
turbosum (generic function with 1 method)

julia> @btime turbosum($arr);
  31.189 ms (2 allocations: 781.30 KiB)

julia> @btime sum($arr; dims=1);
  42.052 ms (2 allocations: 781.30 KiB)

```

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [September 21, 2022, 5:06am UTC](https://discourse.julialang.org/t/how-to-match-performance-of-sum-a-dims-1/87543/9 "2022-09-21T05:06:57Z")

</div>

> [@torrance](#):
>
> Just upgraded from 1.8.0 to 1.8.1, and my version is now twice as slow. That’s a really curious performance degradation.

That sounds pretty bad, I can confirm that results are quite bad for the manual version on v1.8.1, but I get equally bad timings on v1.7.3

---

<div class="post-metadata">

**Author:** ![torrance](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/torrance/32/38990_2.png) [@torrance](https://discourse.julialang.org/u/torrance)\
**Post date:** [September 21, 2022, 5:15am UTC](https://discourse.julialang.org/t/how-to-match-performance-of-sum-a-dims-1/87543/10 "2022-09-21T05:15:53Z")

</div>

Right, so with 1.8.1, `@turbo` indeed has a big improvement. But this is no longer the teaching moment I was hoping for, since ‘sprinkle with `@turbo`’ doesn’t really help people understand what’s going on here.

Also, I wonder how Base is doing this, without access to LoopVectorization magic…

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [September 21, 2022, 5:27am UTC](https://discourse.julialang.org/t/how-to-match-performance-of-sum-a-dims-1/87543/11 "2022-09-21T05:27:21Z")

</div>

You could just look at the base code with `@edit`?

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [September 21, 2022, 5:39am UTC](https://discourse.julialang.org/t/how-to-match-performance-of-sum-a-dims-1/87543/12 "2022-09-21T05:39:34Z")

</div>

The base code is created by macros and not easy to read.

> <https://github.com/JuliaLang/julia/blob/master/base/reducedim.jl#L988-L1025>

---

<div class="post-metadata">

**Author:** ![jishnub](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jishnub/32/33620_2.png) [@jishnub](https://discourse.julialang.org/u/jishnub)\
**Post date:** [September 21, 2022, 5:41am UTC](https://discourse.julialang.org/t/how-to-match-performance-of-sum-a-dims-1/87543/13 "2022-09-21T05:41:32Z")

</div>

The difference seems entirely due to `@simd`:

```julia

julia> function mysum(arr)
                  result = zeros(eltype(arr), size(arr, 2))
                  @inbounds for j in axes(arr, 2)
                      t = zero(eltype(arr))
                      for i in axes(arr, 1)
                          t += arr[i, j]
                      end
                      result[j] = t
                  end
                  return result
              end
mysum (generic function with 1 method)

julia> @btime mysum($arr);
  165.281 ms (2 allocations: 781.30 KiB)

julia> function mysum(arr)
                  result = zeros(eltype(arr), size(arr, 2))
                  @inbounds for j in axes(arr, 2)
                      t = zero(eltype(arr))
                      @simd for i in axes(arr, 1)
                          t += arr[i, j]
                      end
                      result[j] = t
                  end
                  return result
              end
mysum (generic function with 1 method)

julia> @btime mysum($arr);
  53.112 ms (2 allocations: 781.30 KiB)

julia> @btime sum($arr, dims=1);
  52.268 ms (2 allocations: 781.30 KiB)

```

---

<div class="post-metadata">

**Author:** ![torrance](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/torrance/32/38990_2.png) [@torrance](https://discourse.julialang.org/u/torrance)\
**Post date:** [September 21, 2022, 5:48am UTC](https://discourse.julialang.org/t/how-to-match-performance-of-sum-a-dims-1/87543/14 "2022-09-21T05:48:30Z")

</div>

> [@nilshg](#):
>
> You could just look at the base code with `@edit`?

It’s turtles all the way down, but I eventually got to the all-important `Base.mapreduce_impl()` which does a little magic with block sizes and `@simd`.

> [@jishnub](#):
>
> The difference seems entirely due to `@simd`:

You’re right. I need both `@simd` **and** a temporary loop variable for it to work.

I’m a little confused here though, since I didn’t think `@simd` worked with reduction variables, and now it seems to. Am I guaranteed that the sum won’t suffer from race conditions?

---

<div class="post-metadata">

**Author:** ![jishnub](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jishnub/32/33620_2.png) [@jishnub](https://discourse.julialang.org/u/jishnub)\
**Post date:** [September 21, 2022, 6:25am UTC](https://discourse.julialang.org/t/how-to-match-performance-of-sum-a-dims-1/87543/15 "2022-09-21T06:25:50Z")

</div>

I don’t think race conditions are an issue, since IIUC, we’re not accessing the same memory location from multiple tasks. What `@simd` does is [reorder associative operations](https://discourse.julialang.org/t/when-shouldnt-we-use-simd/18276/11), so it may evaluate `a + (b+c)` instead of `(a+b) + c`. This means that if we use `@simd` in reductions, it will likely change the result due to floating-point rounding errors. We may check that `mysum(arr)` differs from `vec(sum(arr, dims=1))`, although they’re approximately equal.