Is it possible to pass a struct in MPI.jl?

I have something like

struct Struct
   a::Vector
   b::Vector
   i::Integer
end

mystruct = Struct(rand(10), rand(5), 1)

How can I pass mystruct around with MPI.jl? If it’s not possible as it is, what’s the easiest workaround?

IIRC, you’re struct must be isbitstype, which yours isn’t. First of all, all your field types are abstract since Integer is abstract and you didn’t specify the type parameters of Vector. This is bad for performance even before thinking about parallelization with MPI.

However, even if you would write your struct as

struct S
    a::Vector{Float64}
    b::Vector{Float64}
    i::Int64
end

it wouldn’t be isbitstype, because Vector doesn’t (statically) indicate the length of the vector.

If you know the length of the vectors, though, you can replace Vector by SVector from StaticArrays.jl to get

julia> struct StaticS
           a::SVector{10, Float64}
           b::SVector{10, Float64}
           i::Int64
       end

julia> isbitstype(StaticS)
true

Objects of this type can then be passed around via e.g. MPI.Send.

A working example:

using MPI
using StaticArrays
using Random

# must be isbitstype so we're using SVector here
struct Particle
    x::Float32
    y::Float32
    z::Float32
    velocity::Float32
    name::SVector{10,Char}
    mass::Float64
end

function Particle()
    return Particle(
        rand(Float32),
        rand(Float32),
        rand(Float32),
        rand(Float32),
        collect(randstring(10)),
        rand(),
    )
end

function main()
    # init MPI as always ...
    MPI.Init()
    rank = MPI.Comm_rank(MPI.COMM_WORLD)
    world_sz = MPI.Comm_size(MPI.COMM_WORLD)

    # send a vector of Particles
    N = 5
    if rank == 0
        println("Rank 0: Sending...")
        ps = [Particle() for _ in 1:N]
        for p in ps
            println(p)
        end
        println()
        # automatically calls create_struct under the hood
        MPI.Send(ps, 1, 0, MPI.COMM_WORLD)
    else
        recvbuf = Vector{Particle}(undef, N)
        MPI.Recv!(recvbuf, 0, 0, MPI.COMM_WORLD)
        println("Rank 1: Receiving...")
        for p in recvbuf
            println(p)
        end
    end

    return MPI.Finalize()
end

main()

Possible output:

Rank 0: Sending...
Particle(0.46500897f0, 0.5270344f0, 0.9695773f0, 0.5295187f0, ['m', 'Y', 'L', 'o', 'k', 'Q', 'J', 'l', '9', 'P'], 0.02892147340795659)
Particle(0.28375614f0, 0.866768f0, 0.06535137f0, 0.059826612f0, ['W', 'E', 'C', 'd', 'A', 'v', 'q', '7', 'e', '1'], 0.1901452287781975)
Particle(0.22925854f0, 0.23061156f0, 0.3626809f0, 0.8808433f0, ['M', '8', 'a', '9', 'a', 'r', '1', '5', 'W', 'o'], 0.25132079245344663)
Particle(0.17365086f0, 0.510172f0, 0.20950961f0, 0.29564202f0, ['2', 'f', '0', '2', 't', 'i', 'n', '2', 'e', 'c'], 0.5372436257786044)
Particle(0.72283494f0, 0.3763957f0, 0.6775358f0, 0.9669912f0, ['N', 'd', '1', 'q', 'b', 'q', 'P', 't', '7', 'c'], 0.8375645365266158)

Rank 1: Receiving...
Particle(0.46500897f0, 0.5270344f0, 0.9695773f0, 0.5295187f0, ['m', 'Y', 'L', 'o', 'k', 'Q', 'J', 'l', '9', 'P'], 0.02892147340795659)
Particle(0.28375614f0, 0.866768f0, 0.06535137f0, 0.059826612f0, ['W', 'E', 'C', 'd', 'A', 'v', 'q', '7', 'e', '1'], 0.1901452287781975)
Particle(0.22925854f0, 0.23061156f0, 0.3626809f0, 0.8808433f0, ['M', '8', 'a', '9', 'a', 'r', '1', '5', 'W', 'o'], 0.25132079245344663)
Particle(0.17365086f0, 0.510172f0, 0.20950961f0, 0.29564202f0, ['2', 'f', '0', '2', 't', 'i', 'n', '2', 'e', 'c'], 0.5372436257786044)
Particle(0.72283494f0, 0.3763957f0, 0.6775358f0, 0.9669912f0, ['N', 'd', '1', 'q', 'b', 'q', 'P', 't', '7', 'c'], 0.8375645365266158)
4 Likes

Thank you for the detailed answer. I already knew that concrete parametrisation substantially improves performance, but I do understand why you might have wanted to clarify it. In fact, my struct example is more like

struct S{a_T, b_T}
    a::Vector{a_T}
    b::Vector{b_T}
    i::Int64
end

where both a_T and b_T are isbitstype, which is somewhat equivalent to your case. So, the second part of your answer is really what I was looking for. Also thank you for that working example.

What if the size of those vectors can’t be set a priori (e.g. they might collect the result of an adaptive ODE solver)? Is there a workaround in that case?

Depends on what precisely a priori means. It is enough if the length is known when creating an object of type S, in which case one could get away with

julia> struct S{a_T, b_T, L}
           a::SVector{L, a_T}
           b::SVector{L, b_T}
           i::Int64
       end

julia> isbitstype(S{Float64, Float64, 10})
true

Does that suffice for your usecase?

Another option might be to make the static vectors large enough (upper bound) and adding an extra field length which indicates the “dynamic” length of the vector content.

If the vectors really need to be dynamically sized, I don’t know a solution from the top of my head.

The problem is that MPI generally expects both sides of the communication to know how much data is being communicated. So you can either communicate the lengths separately, then communicate the arrays individually, or you can use the lowercase MPI.send/MPI.recv functions, which will serialize the data.

How do these functions (capitalised vs lowercase) differ? Or, better, what does serialisation mean? I’ve checked the MPI.jl documentation, but I guess I’m still a newbie in terms of MPI.

Serialization isn’t something MPI specific. See

https://docs.julialang.org/en/v1/stdlib/Serialization/#Serialization

and

Serialization lets you write arbitrary data to a buffer, see Serialization · The Julia Language

The lowercase MPI.send will serialize the data then call MPI.Send (MPI.isend is the non-blocking version of MPI.send); MPI.recv will probe the message to find out how long it is, allocate an appropriate buffer, call MPI.Recv, then deserialize the buffer and return the object.

These are naturally slower as the copy data and are not type stable, there are some benchmarks in the MPI.jl paper

Thank you both. That was illuminating.