MPI.API.MPI_Reduce_scatter ERRORS

Specific Domains Julia at Scale
, , , ,
1

From low-level API in MPI.jl, we know

MPI.API.MPI_Reduce_scatter(sendbuf, recvbuf, recvcounts, datatype, op, comm)

When I ran the code below with mpiexecjl -n 3 julia testReduce_scatter.jl:

# testReduce_scatter.jl
using MPI, LinearAlgebra
MPI.Init()
comm = MPI.COMM_WORLD
rank = MPI.Comm_rank(comm)
counts = [2, 2, 1]
V = ones(3,5)
V_vbuf = VBuffer(V, counts .* 3)
Y = zeros(3,counts[rank+1])
Y_buf = MPI.Buffer(Y)
MPI.API.MPI_Reduce_scatter(V, Y, Y_buf.count, V_vbuf.datatype, +, comm)
println(rank, Y)

I got:

ERROR: MethodError: Cannot `convert` an object of type typeof(+) to an object of type Int32
Closest candidates are:
  convert(::Type{T}, ::T) where T<:Number at /export/pkgs/linux-u22/julia-1.7.3/share/julia/base/number.jl:6
  convert(::Type{T}, ::Number) where T<:Number at /export/pkgs/linux-u22/julia-1.7.3/share/julia/base/number.jl:7
  convert(::Type{T}, ::Base.TwicePrecision) where T<:Number at /export/pkgs/linux-u22/julia-1.7.3/share/julia/base/twiceprecision.jl:262
  ...
Stacktrace:
 [1] Type, x::Type, x::VBuffer{Matrix{Float64}})
   @ VBuffer{Matrix{Float64}})
   @ cconvert(Base ./Base ./T::essentials.jl:417
 [2] MPI_Reduce_scatter(sendbuf::VBuffer{Matrix{Float64}}, recvbuf::Matrix{Float64}, recvcounts::Int32, datatype::MPI.Datatype, op::Function, comm::MPI.Comm)
   @ MPI.API ~/.julia/packages/MPI/tJjHF/src/api/generated_api.jl:765
 [3] top-level scope
   @ /export/users/neo/Documents/testReduce_scatter.jl:11

Does anyone happen to know how to fix this? Why it converts an operator + to Int32?

2

The low level API matches the C API, so the objects have to match. In this case you need to wrap it in a MPI.Op object. See how the MPI.Reduce function is defined for an example

1 Like
3

Do you mean something like this?

using MPI, LinearAlgebra
MPI.Init()
comm = MPI.COMM_WORLD
rank = MPI.Comm_rank(comm)
counts = [2, 2, 1]
V = ones(3,5)
V_vbuf = VBuffer(V, counts .* 3)
Y = zeros(3,counts[rank+1])
Y_buf = MPI.Buffer(Y)
add = MPI.Op(+, iscommutative=true)
MPI.API.MPI_Reduce_scatter(V, Y, Y_buf.count, V_vbuf.datatype, add, comm)
println(rank, Y)

I added add = MPI.Op(+, iscommutative=true) but got another error:

ERROR: LoadError: MethodError: no method matching unsafe_convert(::Type{Ptr{Int32}}, ::Int32)
Closest candidates are:
  unsafe_convert(::Type{Ptr{T}}, ::Base.ReshapedArray{T}) where T at /export/pkgs/linux-u22/julia-1.7.3/share/julia/base/reshapedarray.jl:281
  unsafe_convert(::Type{Ptr{T}}, ::Base.ReinterpretArray{T, N, S, A} where {N, A<:(AbstractArray{S})}) where {T, S} at /export/pkgs/linux-u22/julia-1.7.3/share/julia/base/reinterpretarray.jl:315
  unsafe_convert(::Type{Ptr{T}}, ::Base.Threads.Atomic{T}) where T at /export/pkgs/linux-u22/julia-1.7.3/share/julia/base/atomics.jl:328
  ...
Stacktrace:
 [1] MPI_Reduce_scatter(sendbuf::Matrix{Float64}, recvbuf::Matrix{Float64}, recvcounts::Int32, datatype::MPI.Datatype, op::MPI.Op, comm::MPI.Comm)
   @ MPI.API ~/.julia/packages/MPI/tJjHF/src/api/generated_api.jl:765
 [2] top-level scope
4

You might need to wrap Y_buf.count into a Ref or similar.

5

I tried to wrap Y_buf.count into Ref{Int32}(Y_buf.count), Ptr{Int32}(Y_buf.count), and Ptr{Int64}(Int64(Y_buf.count)), but none of them worked.
For example:

# testReduce_scatter.jl
using MPI, LinearAlgebra
MPI.Init()
comm = MPI.COMM_WORLD
rank = MPI.Comm_rank(comm)
counts = [2, 2, 1]
V = ones(3,5)
V_vbuf = VBuffer(V, counts .* 3)
Y = zeros(3,counts[rank+1])
Y_buf = MPI.Buffer(Y)
add = MPI.Op(+, Float64, iscommutative=true)
MPI.API.MPI_Reduce_scatter(V, Y_buf.data, Ref{Int32}(Y_buf.count), V_vbuf.datatype, add, comm)
for i in 0:2
   if i == rank
      println(rank, Y)
   end
   MPI.Barrier(comm)
end

I got

% mpiexecjl -n 1 julia testReduce_scatter.jl
0[1.0 1.0; 1.0 1.0; 1.0 1.0]

and

% mpiexecjl -n 2 julia testReduce_scatter.jl
0[2.0 2.0; 2.0 2.0; 2.0 2.0]
1[0.0 0.0; 0.0 0.0; 0.0 0.0]

and

% mpiexecjl -n 3 julia testReduce_scatter.jl
ERROR: ERROR: LoadError: LoadError: MPIError(201992706): Invalid count, error stack:
internal_Reduce_scatter(141): MPI_Reduce_scatter(sendbuf=0x7ff2bd73dd20, recvbuf=0x7ff2bbc1b610, recvcounts=0x7ff2bdb5ca30, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD) failed
internal_Reduce_scatter(96).: Negative count, value is -1312333680
Stacktrace:
 [1] MPIError(666114): Invalid count, error stack:
internal_Reduce_scatter(141): MPI_Reduce_scatter(sendbuf=0x7f9faa53dd20, recvbuf=0x7f9fa8cf2f20, recvcounts=0x7f9faa95ca30, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD) failed
internal_Reduce_scatter(96).: Negative count, value is -1633197936
Stacktrace:
 [1] MPI_Reduce_scatter(sendbuf::Matrix{Float64}, recvbuf::Matrix{Float64}, recvcounts::Base.RefValue{Int32}, datatype::MPI.Datatype, op::MPI.Op, comm::MPI.Comm)
   @ MPI_Reduce_scatter(MPI.API ~/.julia/packages/MPI/tJjHF/src/api/sendbuf::Matrix{Float64}, recvbuf::Matrix{Float64}, recvcounts::Base.RefValue{Int32}, datatype::MPI.Datatype, op::MPI.Op, comm::MPI.Comm)
   @ generated_api.jl:765

In the second case, the output Y for rank 1 is expected to be [2.0 2.0; 2.0 2.0; 2.0 2.0] but got [0.0 0.0; 0.0 0.0; 0.0 0.0]. In the third case, the count is negative. I also tried Ptr{Int32} and Ptr{Int64} but neither worked. Any insights? Please help.

6

Thanks @carstenbauer and @simonbyrne for help. I figured it out.
The code below works.

# testReduce_scatter.jl
using MPI, LinearAlgebra
MPI.Init()
comm = MPI.COMM_WORLD
rank = MPI.Comm_rank(comm)
counts = [2, 2, 1]
V = ones(3,5)
V_vbuf = VBuffer(V, counts .* 3)
Y = zeros(3,counts[rank+1])
Y_buf = MPI.Buffer(Y)
add = MPI.Op(+, Float64, iscommutative=true)
MPI.API.MPI_Reduce_scatter(V, Y, V_vbuf.counts, Y_buf.datatype, add, comm)
for i in 0:2
   if i == rank
      println(rank, Y)
   end
   MPI.Barrier(comm)
end

% mpiexecjl -n 3 julia testReduce_scatter.jl
0[3.0 3.0; 3.0 3.0; 3.0 3.0]
1[3.0 3.0; 3.0 3.0; 3.0 3.0]
2[3.0; 3.0; 3.0;;]
1 Like
7

Glad you made it work. Please do consider opening a PR against MPI.jl so that we can support it going forward.

8

It would be good to have this usage documented in MPI.jl. @Neo would you mind updating the documentation to include such an example?