# How to "reduce" an array?

**URL:** <https://discourse.julialang.org/t/how-to-reduce-an-array/92945>\
**Category:** General Usage\
**Tags:** cuda, arrays\
**Created:** [January 13, 2023, 11:29pm UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945 "2023-01-13T23:29:11Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ahmed\_Salih](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ahmed_salih/32/206579_2.png) [@Ahmed\_Salih](https://discourse.julialang.org/u/Ahmed_Salih)\
**Post date:** [January 13, 2023, 11:29pm UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/1 "2023-01-13T23:29:11Z")

</div>

Hello

The wording is a bit bad in the title and I struggle to explain what I want to do properly, but basically what I want to do is;

Suppose I have an array I consisting of indices:

```julia
I = [1 1 1 2 2 2 3 3 3]

```

And an array of results, R:

```julia
R = [2 4 7 8 5 1 9 8 1]

```

Then I want to produce a sum of R based on the indices of I, such that a final result would be:

```julia
FR = [13 14 18]

```

Where 13 comes from the first 3 indices being 1 1 1 in I.

Hope someone got what I mean… 😅

EDIT: And I want to do it using a broadcasting operation, no for loops

Kind regards

---

<div class="post-metadata">

**Author:** ![jmair](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jmair/32/35117_2.png) [@jmair](https://discourse.julialang.org/u/jmair)\
**Post date:** [January 13, 2023, 11:33pm UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/2 "2023-01-13T23:33:30Z")

</div>

I am not sure how practical this for your actual use case, but this could work:

```julia
sum(reshape(R, 3, :), dims=1)

```

This obviously only works if the sums are the same sizes and adjacent.

---

<div class="post-metadata">

**Author:** ![Ahmed\_Salih](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ahmed_salih/32/206579_2.png) [@Ahmed\_Salih](https://discourse.julialang.org/u/Ahmed_Salih)\
**Post date:** [January 13, 2023, 11:35pm UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/3 "2023-01-13T23:35:05Z")

</div>

> [@jmair](#):
>
> `sum(reshape(R, 3, :), dims=1)`

Thank you for your comment!

No that does not work for me, I could also have been:

```julia
I = [1 2 3 3 2 1 1 1 2]

```

Just wanted to provide a simple example since I struggled with explaining it 🙂

Kind regards

---

<div class="post-metadata">

**Author:** ![jmair](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jmair/32/35117_2.png) [@jmair](https://discourse.julialang.org/u/jmair)\
**Post date:** [January 13, 2023, 11:36pm UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/4 "2023-01-13T23:36:34Z")

</div>

I am not sure how to do this without a `for` loop, is there any reason you can’t use one?

---

<div class="post-metadata">

**Author:** ![stillyslalom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stillyslalom/32/45687_2.png) [@stillyslalom](https://discourse.julialang.org/u/stillyslalom)\
**Post date:** [January 13, 2023, 11:37pm UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/5 "2023-01-13T23:37:42Z")

</div>

```julia
julia> (unique(I) .== I)*R'
3×1 Matrix{Int64}:
 13
 14
 18

```

…but a for-loop would be more efficient

---

<div class="post-metadata">

**Author:** ![Ahmed\_Salih](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ahmed_salih/32/206579_2.png) [@Ahmed\_Salih](https://discourse.julialang.org/u/Ahmed_Salih)\
**Post date:** [January 13, 2023, 11:41pm UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/6 "2023-01-13T23:41:06Z")

</div>

Doing GPU programming, want to see if I can do it without making a kernel - cannot use scalar indexing

---

<div class="post-metadata">

**Author:** ![Ahmed\_Salih](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ahmed_salih/32/206579_2.png) [@Ahmed\_Salih](https://discourse.julialang.org/u/Ahmed_Salih)\
**Post date:** [January 13, 2023, 11:41pm UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/7 "2023-01-13T23:41:51Z")

</div>

Thank you for the suggestion!

I see why it would work, but as you say quite slow

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [January 13, 2023, 11:59pm UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/8 "2023-01-13T23:59:03Z")

</div>

Maybe something like this:

```julia
function sumind(a, ind)
    val = zeros(Int, maximum(a))
    for i in eachindex(ind)
        val[a[i]] += ind[i] # fixed typo, thanks @rafael.guerra 
    end
    return val
end 

```

BTW, vectors are more fundamental and efficient than matrices. So I suggest making vectors

```julia
I = [1, 1, 1, 2, 2, 2, 3, 3, 3]
R = [2, 4, 7, 8, 5, 1, 9, 8, 1]

```

instead of 1xN matrices. It will probably not make a noticeable difference in this example, but it does help the compiler know what is going on.

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [January 14, 2023, 12:38am UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/9 "2023-01-14T00:38:36Z")

</div>

> [@DNF](#):
>
> `val[a[i]] += R[i]`

Typo: here `R` should be ` ind`

---

<div class="post-metadata">

**Author:** ![Ahmed\_Salih](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ahmed_salih/32/206579_2.png) [@Ahmed\_Salih](https://discourse.julialang.org/u/Ahmed_Salih)\
**Post date:** [January 14, 2023, 12:39am UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/10 "2023-01-14T00:39:34Z")

</div>

Thank you for all the suggestions guys!

I ended up writing my own kernel as:

```julia
I = CuArray([1,1,1,2,2,2,3,3,3])
R = CuArray([2,4,7,8,5,1,9,8,1])
FR = CuArray([0,0,0])

function gpu_add_simple!(y, x, I)
    index = threadIdx().x
    stride = blockDim().x
    for i = index:stride:length(I)
        @inbounds y[I[i]] += x[i]
    end
    return nothing
end

@cuda gpu_add_simple!(FR,R,I)

FR
3-element CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}:
 13
 14
 18

```

Inspired by some GPU Julia docs.

I am sure some with GPU knowledge could make the function even more functional, but this does what I want.

EDIT: And I had enabled “CUDA.allowscalar(false)”. I wrongly thought for loops would break this assumption.

Kind regards

---

<div class="post-metadata">

**Author:** ![jmair](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jmair/32/35117_2.png) [@jmair](https://discourse.julialang.org/u/jmair)\
**Post date:** [January 14, 2023, 12:45am UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/11 "2023-01-14T00:45:46Z")

</div>

I was thinking of something like this, but doesn’t this have a race condition for an arbitrary index array?

---

<div class="post-metadata">

**Author:** ![Ahmed\_Salih](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ahmed_salih/32/206579_2.png) [@Ahmed\_Salih](https://discourse.julialang.org/u/Ahmed_Salih)\
**Post date:** [January 14, 2023, 12:49am UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/12 "2023-01-14T00:49:35Z")

</div>

Seems like it does, results get wrong when using multiple threads/blocks - hoping someone more experienced can help out 🙂

---

<div class="post-metadata">

**Author:** ![mcabbott](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcabbott/32/6603_2.png) [@mcabbott](https://discourse.julialang.org/u/mcabbott)\
**Post date:** [January 14, 2023, 12:52am UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/13 "2023-01-14T00:52:27Z")

</div>

Yes to the race condition. This is `NNlib.scatter(+, R, I)`, for which there’s a kernel which should be written so as to avoid it.

---

<div class="post-metadata">

**Author:** ![Ahmed\_Salih](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ahmed_salih/32/206579_2.png) [@Ahmed\_Salih](https://discourse.julialang.org/u/Ahmed_Salih)\
**Post date:** [January 14, 2023, 1:01am UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/14 "2023-01-14T01:01:05Z")

</div>

Thank you for the comment!

I just tried it out and it does work!

I have one question though, how come it does not work when R is a vector of static arrays instead of just numbers?

Shouldn’t I be able to add those together too?

And is it easy to learn how to modify my custom kernel to avoid the race condition?

Kind regards

---

<div class="post-metadata">

**Author:** ![Ahmed\_Salih](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ahmed_salih/32/206579_2.png) [@Ahmed\_Salih](https://discourse.julialang.org/u/Ahmed_Salih)\
**Post date:** [January 14, 2023, 1:05am UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/15 "2023-01-14T01:05:54Z")

</div>

![image](https://global.discourse-cdn.com/julialang/original/3X/d/4/d4495b5e527e2249530c773206fbc8d18e1a821f.png)

It breaks for me when trying to use CuArray as index - but if I dont use CuArray the calculation becomes slow on the CPU

Kind regards

---

<div class="post-metadata">

**Author:** ![mcabbott](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcabbott/32/6603_2.png) [@mcabbott](https://discourse.julialang.org/u/mcabbott)\
**Post date:** [January 14, 2023, 1:11am UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/16 "2023-01-14T01:11:27Z")

</div>

Without looking closely I’d assume the failure on a vector of SVector is a bug. Make an issue (with a minimal example, and code not a screenshot)… or better see if you can fix it, code is [here](https://github.com/FluxML/NNlib.jl/blob/master/src/scatter.jl) and [here](https://github.com/FluxML/NNlibCUDA.jl/blob/master/src/scatter.jl).

---

<div class="post-metadata">

**Author:** ![Ahmed\_Salih](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ahmed_salih/32/206579_2.png) [@Ahmed\_Salih](https://discourse.julialang.org/u/Ahmed_Salih)\
**Post date:** [January 14, 2023, 1:17am UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/17 "2023-01-14T01:17:21Z")

</div>

Thanks done

---

<div class="post-metadata">

**Author:** ![Ahmed\_Salih](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ahmed_salih/32/206579_2.png) [@Ahmed\_Salih](https://discourse.julialang.org/u/Ahmed_Salih)\
**Post date:** [January 14, 2023, 2:33pm UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/18 "2023-01-14T14:33:57Z")

</div>

> [@Ahmed\_Salih](#):
>
> ```julia
> function gpu_add_simple!(y, x, I)
> index = threadIdx().x
> stride = blockDim().x
> for i = index:stride:length(I)
> @inbounds y[I[i]] += x[i]
> end
> return nothing
> end
> 
> ```

Anyone who knows how to remove the race condition?

The code I am working on is being held back a lot by it being so slow only with 1 thread and block…

Kind regards

---

<div class="post-metadata">

**Author:** ![jmair](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jmair/32/35117_2.png) [@jmair](https://discourse.julialang.org/u/jmair)\
**Post date:** [January 14, 2023, 4:49pm UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/19 "2023-01-14T16:49:43Z")

</div>

I think you can use `CUDA.@atomic`:

```julia
function gpu_add_scatter!(y, x, I)
    index = threadIdx().x + (blockIdx().x - 1) * blockDim().x
    @inbounds if index<length(y)
        CUDA.@atomic y[I[index]] += x[index]
    end
    nothing
end

```

More in the source code of `NNLibCUDA.jl`: [NNlibCUDA.jl/scatter.jl at master · FluxML/NNlibCUDA.jl · GitHub](https://github.com/FluxML/NNlibCUDA.jl/blob/master/src/scatter.jl)

---

<div class="post-metadata">

**Author:** ![Ahmed\_Salih](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ahmed_salih/32/206579_2.png) [@Ahmed\_Salih](https://discourse.julialang.org/u/Ahmed_Salih)\
**Post date:** [January 14, 2023, 10:58pm UTC](https://discourse.julialang.org/t/how-to-reduce-an-array/92945/20 "2023-01-14T22:58:13Z")

</div>

Thank you for the suggestion!

I have not tested it, but ended up using the Flux library, which afaik refers to that package you link to, when CuArrays are input.

Instead of using StaticArrays, I had to split the invidiual coordinates out in arrays and then recombine to staticarrays again, pseudo code:

```julia
function GPUCalculateKernelGradient!(WgI,αD,Q,xᵢⱼ,H,I_,J_,N)
    WgL = αD*5 .* (Q .- 2).^3 .* Q ./ (8H .* (Q * H .+ 1e-6)) .* xᵢⱼ

    WgIx = NNlib.scatter(+, getindex.(WgL,1), I_; dstsize = N) - NNlib.scatter(+, getindex.(WgL,1), J_; dstsize = N)
    WgIy = NNlib.scatter(+, getindex.(WgL,2), I_; dstsize = N) - NNlib.scatter(+, getindex.(WgL,2), J_; dstsize = N)
    WgIz = NNlib.scatter(+, getindex.(WgL,3), I_; dstsize = N) - NNlib.scatter(+, getindex.(WgL,3), J_; dstsize = N)
    WgI .= map((x,y,z)->SVector(x,y,z),WgIx,WgIy,WgIz)

    return WgL
end

```

Where WgL is a CuArray of SVector{3,Float64}

Just to help others out in the future, until StaticArrays can be used directly.

Kind regards