# MPI.API.MPI\_Reduce\_scatter ERRORS

**URL:** <https://discourse.julialang.org/t/mpi-api-mpi-reduce-scatter-errors/90093>\
**Category:** Julia at Scale\
**Tags:** parallel, mpi, distributed, api, mpijl\
**Created:** [November 11, 2022, 3:00am UTC](https://discourse.julialang.org/t/mpi-api-mpi-reduce-scatter-errors/90093 "2022-11-11T03:00:08Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Neo](https://avatars.discourse-cdn.com/v4/letter/n/9fc348/32.png) [@Neo](https://discourse.julialang.org/u/Neo)\
**Post date:** [November 11, 2022, 3:00am UTC](https://discourse.julialang.org/t/mpi-api-mpi-reduce-scatter-errors/90093/1 "2022-11-11T03:00:08Z")

</div>

From [low-level API in MPI.jl](https://juliaparallel.org/MPI.jl/stable/reference/api/), we know

MPI.API.MPI\_Reduce\_scatter(sendbuf, recvbuf, recvcounts, datatype, op, comm)

When I ran the code below with mpiexecjl -n 3 julia testReduce\_scatter.jl:

```julia
# testReduce_scatter.jl
using MPI, LinearAlgebra
MPI.Init()
comm = MPI.COMM_WORLD
rank = MPI.Comm_rank(comm)
counts = [2, 2, 1]
V = ones(3,5)
V_vbuf = VBuffer(V, counts .* 3)
Y = zeros(3,counts[rank+1])
Y_buf = MPI.Buffer(Y)
MPI.API.MPI_Reduce_scatter(V, Y, Y_buf.count, V_vbuf.datatype, +, comm)
println(rank, Y)

```

I got:

```julia
ERROR: MethodError: Cannot `convert` an object of type typeof(+) to an object of type Int32
Closest candidates are:
  convert(::Type{T}, ::T) where T<:Number at /export/pkgs/linux-u22/julia-1.7.3/share/julia/base/number.jl:6
  convert(::Type{T}, ::Number) where T<:Number at /export/pkgs/linux-u22/julia-1.7.3/share/julia/base/number.jl:7
  convert(::Type{T}, ::Base.TwicePrecision) where T<:Number at /export/pkgs/linux-u22/julia-1.7.3/share/julia/base/twiceprecision.jl:262
  ...
Stacktrace:
 [1] Type, x::Type, x::VBuffer{Matrix{Float64}})
   @ VBuffer{Matrix{Float64}})
   @ cconvert(Base ./Base ./T::essentials.jl:417
 [2] MPI_Reduce_scatter(sendbuf::VBuffer{Matrix{Float64}}, recvbuf::Matrix{Float64}, recvcounts::Int32, datatype::MPI.Datatype, op::Function, comm::MPI.Comm)
   @ MPI.API ~/.julia/packages/MPI/tJjHF/src/api/generated_api.jl:765
 [3] top-level scope
   @ /export/users/neo/Documents/testReduce_scatter.jl:11

```

Does anyone happen to know how to fix this? Why it converts an operator + to Int32?

---

<div class="post-metadata">

**Author:** ![simonbyrne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simonbyrne/32/19_2.png) [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)\
**Post date:** [November 11, 2022, 5:42am UTC](https://discourse.julialang.org/t/mpi-api-mpi-reduce-scatter-errors/90093/2 "2022-11-11T05:42:20Z")

</div>

The low level API matches the C API, so the objects have to match. In this case you need to wrap it in a MPI.Op object. See how the MPI.Reduce function is defined for an example

---

<div class="post-metadata">

**Author:** ![Neo](https://avatars.discourse-cdn.com/v4/letter/n/9fc348/32.png) [@Neo](https://discourse.julialang.org/u/Neo)\
**Post date:** [November 11, 2022, 3:44pm UTC](https://discourse.julialang.org/t/mpi-api-mpi-reduce-scatter-errors/90093/3 "2022-11-11T15:44:24Z")

</div>

Do you mean something like this?

```julia
using MPI, LinearAlgebra
MPI.Init()
comm = MPI.COMM_WORLD
rank = MPI.Comm_rank(comm)
counts = [2, 2, 1]
V = ones(3,5)
V_vbuf = VBuffer(V, counts .* 3)
Y = zeros(3,counts[rank+1])
Y_buf = MPI.Buffer(Y)
add = MPI.Op(+, iscommutative=true)
MPI.API.MPI_Reduce_scatter(V, Y, Y_buf.count, V_vbuf.datatype, add, comm)
println(rank, Y)

```

I added _add = MPI.Op(+, iscommutative=true)_ but got another error:

```julia
ERROR: LoadError: MethodError: no method matching unsafe_convert(::Type{Ptr{Int32}}, ::Int32)
Closest candidates are:
  unsafe_convert(::Type{Ptr{T}}, ::Base.ReshapedArray{T}) where T at /export/pkgs/linux-u22/julia-1.7.3/share/julia/base/reshapedarray.jl:281
  unsafe_convert(::Type{Ptr{T}}, ::Base.ReinterpretArray{T, N, S, A} where {N, A<:(AbstractArray{S})}) where {T, S} at /export/pkgs/linux-u22/julia-1.7.3/share/julia/base/reinterpretarray.jl:315
  unsafe_convert(::Type{Ptr{T}}, ::Base.Threads.Atomic{T}) where T at /export/pkgs/linux-u22/julia-1.7.3/share/julia/base/atomics.jl:328
  ...
Stacktrace:
 [1] MPI_Reduce_scatter(sendbuf::Matrix{Float64}, recvbuf::Matrix{Float64}, recvcounts::Int32, datatype::MPI.Datatype, op::MPI.Op, comm::MPI.Comm)
   @ MPI.API ~/.julia/packages/MPI/tJjHF/src/api/generated_api.jl:765
 [2] top-level scope

```

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [November 11, 2022, 4:05pm UTC](https://discourse.julialang.org/t/mpi-api-mpi-reduce-scatter-errors/90093/4 "2022-11-11T16:05:00Z")

</div>

You might need to wrap `Y_buf.count` into a `Ref` or similar.

---

<div class="post-metadata">

**Author:** ![Neo](https://avatars.discourse-cdn.com/v4/letter/n/9fc348/32.png) [@Neo](https://discourse.julialang.org/u/Neo)\
**Post date:** [November 11, 2022, 5:09pm UTC](https://discourse.julialang.org/t/mpi-api-mpi-reduce-scatter-errors/90093/5 "2022-11-11T17:09:40Z")

</div>

I tried to wrap `Y_buf.count` into `Ref{Int32}(Y_buf.count)`, `Ptr{Int32}(Y_buf.count)`, and `Ptr{Int64}(Int64(Y_buf.count))`, but none of them worked.  
For example:

```julia
# testReduce_scatter.jl
using MPI, LinearAlgebra
MPI.Init()
comm = MPI.COMM_WORLD
rank = MPI.Comm_rank(comm)
counts = [2, 2, 1]
V = ones(3,5)
V_vbuf = VBuffer(V, counts .* 3)
Y = zeros(3,counts[rank+1])
Y_buf = MPI.Buffer(Y)
add = MPI.Op(+, Float64, iscommutative=true)
MPI.API.MPI_Reduce_scatter(V, Y_buf.data, Ref{Int32}(Y_buf.count), V_vbuf.datatype, add, comm)
for i in 0:2
   if i == rank
      println(rank, Y)
   end
   MPI.Barrier(comm)
end

```

I got

```julia
% mpiexecjl -n 1 julia testReduce_scatter.jl
0[1.0 1.0; 1.0 1.0; 1.0 1.0]

```

and

```julia
% mpiexecjl -n 2 julia testReduce_scatter.jl
0[2.0 2.0; 2.0 2.0; 2.0 2.0]
1[0.0 0.0; 0.0 0.0; 0.0 0.0]

```

and

```julia
% mpiexecjl -n 3 julia testReduce_scatter.jl
ERROR: ERROR: LoadError: LoadError: MPIError(201992706): Invalid count, error stack:
internal_Reduce_scatter(141): MPI_Reduce_scatter(sendbuf=0x7ff2bd73dd20, recvbuf=0x7ff2bbc1b610, recvcounts=0x7ff2bdb5ca30, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD) failed
internal_Reduce_scatter(96).: Negative count, value is -1312333680
Stacktrace:
 [1] MPIError(666114): Invalid count, error stack:
internal_Reduce_scatter(141): MPI_Reduce_scatter(sendbuf=0x7f9faa53dd20, recvbuf=0x7f9fa8cf2f20, recvcounts=0x7f9faa95ca30, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD) failed
internal_Reduce_scatter(96).: Negative count, value is -1633197936
Stacktrace:
 [1] MPI_Reduce_scatter(sendbuf::Matrix{Float64}, recvbuf::Matrix{Float64}, recvcounts::Base.RefValue{Int32}, datatype::MPI.Datatype, op::MPI.Op, comm::MPI.Comm)
   @ MPI_Reduce_scatter(MPI.API ~/.julia/packages/MPI/tJjHF/src/api/sendbuf::Matrix{Float64}, recvbuf::Matrix{Float64}, recvcounts::Base.RefValue{Int32}, datatype::MPI.Datatype, op::MPI.Op, comm::MPI.Comm)
   @ generated_api.jl:765

```

In the second case, the output `Y` for `rank 1` is expected to be `[2.0 2.0; 2.0 2.0; 2.0 2.0]` but got `[0.0 0.0; 0.0 0.0; 0.0 0.0]`. In the third case, the count is negative. I also tried `Ptr{Int32}` and `Ptr{Int64}` but neither worked. Any insights? Please help.

---

<div class="post-metadata">

**Author:** ![Neo](https://avatars.discourse-cdn.com/v4/letter/n/9fc348/32.png) [@Neo](https://discourse.julialang.org/u/Neo)\
**Post date:** [November 11, 2022, 6:21pm UTC](https://discourse.julialang.org/t/mpi-api-mpi-reduce-scatter-errors/90093/6 "2022-11-11T18:21:08Z")

</div>

Thanks @carstenbauer and @simonbyrne for help. I figured it out.  
The code below works.

```julia
# testReduce_scatter.jl
using MPI, LinearAlgebra
MPI.Init()
comm = MPI.COMM_WORLD
rank = MPI.Comm_rank(comm)
counts = [2, 2, 1]
V = ones(3,5)
V_vbuf = VBuffer(V, counts .* 3)
Y = zeros(3,counts[rank+1])
Y_buf = MPI.Buffer(Y)
add = MPI.Op(+, Float64, iscommutative=true)
MPI.API.MPI_Reduce_scatter(V, Y, V_vbuf.counts, Y_buf.datatype, add, comm)
for i in 0:2
   if i == rank
      println(rank, Y)
   end
   MPI.Barrier(comm)
end

```

```julia
% mpiexecjl -n 3 julia testReduce_scatter.jl
0[3.0 3.0; 3.0 3.0; 3.0 3.0]
1[3.0 3.0; 3.0 3.0; 3.0 3.0]
2[3.0; 3.0; 3.0;;]

```

---

<div class="post-metadata">

**Author:** ![simonbyrne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simonbyrne/32/19_2.png) [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)\
**Post date:** [November 14, 2022, 10:05pm UTC](https://discourse.julialang.org/t/mpi-api-mpi-reduce-scatter-errors/90093/7 "2022-11-14T22:05:31Z")

</div>

Glad you made it work. Please do consider opening a PR against MPI.jl so that we can support it going forward.

---

<div class="post-metadata">

**Author:** ![jishnub](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jishnub/32/33620_2.png) [@jishnub](https://discourse.julialang.org/u/jishnub)\
**Post date:** [November 15, 2022, 6:09am UTC](https://discourse.julialang.org/t/mpi-api-mpi-reduce-scatter-errors/90093/8 "2022-11-15T06:09:09Z")

</div>

It would be good to have this usage documented in MPI.jl. @Neo would you mind updating the documentation to include such an example?