I have something like
struct Struct
a::Vector
b::Vector
i::Integer
end
mystruct = Struct(rand(10), rand(5), 1)
How can I pass mystruct around with MPI.jl? If it’s not possible as it is, what’s the easiest workaround?
IIRC, you’re struct must be isbitstype, which yours isn’t. First of all, all your field types are abstract since Integer is abstract and you didn’t specify the type parameters of Vector. This is bad for performance even before thinking about parallelization with MPI.
However, even if you would write your struct as
struct S
a::Vector{Float64}
b::Vector{Float64}
i::Int64
end
it wouldn’t be isbitstype, because Vector doesn’t (statically) indicate the length of the vector.
If you know the length of the vectors, though, you can replace Vector by SVector from StaticArrays.jl to get
julia> struct StaticS
a::SVector{10, Float64}
b::SVector{10, Float64}
i::Int64
end
julia> isbitstype(StaticS)
true
Objects of this type can then be passed around via e.g. MPI.Send.
A working example:
using MPI
using StaticArrays
using Random
# must be isbitstype so we're using SVector here
struct Particle
x::Float32
y::Float32
z::Float32
velocity::Float32
name::SVector{10,Char}
mass::Float64
end
function Particle()
return Particle(
rand(Float32),
rand(Float32),
rand(Float32),
rand(Float32),
collect(randstring(10)),
rand(),
)
end
function main()
# init MPI as always ...
MPI.Init()
rank = MPI.Comm_rank(MPI.COMM_WORLD)
world_sz = MPI.Comm_size(MPI.COMM_WORLD)
# send a vector of Particles
N = 5
if rank == 0
println("Rank 0: Sending...")
ps = [Particle() for _ in 1:N]
for p in ps
println(p)
end
println()
# automatically calls create_struct under the hood
MPI.Send(ps, 1, 0, MPI.COMM_WORLD)
else
recvbuf = Vector{Particle}(undef, N)
MPI.Recv!(recvbuf, 0, 0, MPI.COMM_WORLD)
println("Rank 1: Receiving...")
for p in recvbuf
println(p)
end
end
return MPI.Finalize()
end
main()
Possible output:
Rank 0: Sending...
Particle(0.46500897f0, 0.5270344f0, 0.9695773f0, 0.5295187f0, ['m', 'Y', 'L', 'o', 'k', 'Q', 'J', 'l', '9', 'P'], 0.02892147340795659)
Particle(0.28375614f0, 0.866768f0, 0.06535137f0, 0.059826612f0, ['W', 'E', 'C', 'd', 'A', 'v', 'q', '7', 'e', '1'], 0.1901452287781975)
Particle(0.22925854f0, 0.23061156f0, 0.3626809f0, 0.8808433f0, ['M', '8', 'a', '9', 'a', 'r', '1', '5', 'W', 'o'], 0.25132079245344663)
Particle(0.17365086f0, 0.510172f0, 0.20950961f0, 0.29564202f0, ['2', 'f', '0', '2', 't', 'i', 'n', '2', 'e', 'c'], 0.5372436257786044)
Particle(0.72283494f0, 0.3763957f0, 0.6775358f0, 0.9669912f0, ['N', 'd', '1', 'q', 'b', 'q', 'P', 't', '7', 'c'], 0.8375645365266158)
Rank 1: Receiving...
Particle(0.46500897f0, 0.5270344f0, 0.9695773f0, 0.5295187f0, ['m', 'Y', 'L', 'o', 'k', 'Q', 'J', 'l', '9', 'P'], 0.02892147340795659)
Particle(0.28375614f0, 0.866768f0, 0.06535137f0, 0.059826612f0, ['W', 'E', 'C', 'd', 'A', 'v', 'q', '7', 'e', '1'], 0.1901452287781975)
Particle(0.22925854f0, 0.23061156f0, 0.3626809f0, 0.8808433f0, ['M', '8', 'a', '9', 'a', 'r', '1', '5', 'W', 'o'], 0.25132079245344663)
Particle(0.17365086f0, 0.510172f0, 0.20950961f0, 0.29564202f0, ['2', 'f', '0', '2', 't', 'i', 'n', '2', 'e', 'c'], 0.5372436257786044)
Particle(0.72283494f0, 0.3763957f0, 0.6775358f0, 0.9669912f0, ['N', 'd', '1', 'q', 'b', 'q', 'P', 't', '7', 'c'], 0.8375645365266158)
5 Likes
Thank you for the detailed answer. I already knew that concrete parametrisation substantially improves performance, but I do understand why you might have wanted to clarify it. In fact, my struct example is more like
struct S{a_T, b_T}
a::Vector{a_T}
b::Vector{b_T}
i::Int64
end
where both a_T and b_T are isbitstype, which is somewhat equivalent to your case. So, the second part of your answer is really what I was looking for. Also thank you for that working example.
What if the size of those vectors can’t be set a priori (e.g. they might collect the result of an adaptive ODE solver)? Is there a workaround in that case?
Depends on what precisely a priori means. It is enough if the length is known when creating an object of type S, in which case one could get away with
julia> struct S{a_T, b_T, L}
a::SVector{L, a_T}
b::SVector{L, b_T}
i::Int64
end
julia> isbitstype(S{Float64, Float64, 10})
true
Does that suffice for your usecase?
Another option might be to make the static vectors large enough (upper bound) and adding an extra field length which indicates the “dynamic” length of the vector content.
If the vectors really need to be dynamically sized, I don’t know a solution from the top of my head.
The problem is that MPI generally expects both sides of the communication to know how much data is being communicated. So you can either communicate the lengths separately, then communicate the arrays individually, or you can use the lowercase MPI.send/MPI.recv functions, which will serialize the data.
How do these functions (capitalised vs lowercase) differ? Or, better, what does serialisation mean? I’ve checked the MPI.jl documentation, but I guess I’m still a newbie in terms of MPI.
Serialization isn’t something MPI specific. See
https://docs.julialang.org/en/v1/stdlib/Serialization/#Serialization
and
Serialization lets you write arbitrary data to a buffer, see Serialization · The Julia Language
The lowercase MPI.send will serialize the data then call MPI.Send (MPI.isend is the non-blocking version of MPI.send); MPI.recv will probe the message to find out how long it is, allocate an appropriate buffer, call MPI.Recv, then deserialize the buffer and return the object.
These are naturally slower as the copy data and are not type stable, there are some benchmarks in the MPI.jl paper
Thank you both. That was illuminating.