How can I allocate a buffer contains different types of data (for MPI communication, for example)?

General Usage Performance
,
1

I need to communicate data of Vector{Int8} and Vector{Float64} using MPI.jl. How can I achieve this with a single communication?

2

The easiest way is probably to put these into a struct and then communicate the struct. I am not sure whether

matters for performance. So before you make large changes to your code that might reduce readability, I recommend to benchmark whether these changes help at all.

3

Thanks for your advice. The problem of communicating a struct is that the struct should satisfy isbitstype(T) is true. Thus no AbstractArray type can be used, which will cause severe performance problem if the data is changed frequently. It is indeed necessary to test whether multiple communications will affect performance, and I will conduct this test afterwards.

4

Ah now I understand better what you are getting at. You say you need a isbitstype for communication? Then ordinary AbstractArrays already don’t work, so packing them into a struct does of course not work as well.

But then I don’t think that your approach does make sense. Say you have two arrays (of different type if you wish). Then you’d need to allocate some appropriate Buffer (could try e.g. a Memory in 1.11), copy all of your array’s contents into there and then communicate that. For anything but the smallest arrays, I would assume that this overhead (allocation and copying) would dwarf the cost of a second communication. But then again I have never used MPI myself, so can’t speak from practical experience.

If you know the amount of memory needed beforehand, then you could try to preallocate a sufficient amount of memory and wrap sections of that into Array{T} to work with it. Then you communicate the MemoryRef (which is isbits) of the whole memory.

5

If you know the amount of memory needed beforehand, then you could try to preallocate a sufficient amount of memory and wrap sections of that into Array{T} to work with it. Then you communicate the MemoryRef (which is isbits) of the whole memory.

That’s exactly what I want to know. How to preallocate the memory according to the size of data? And how to wrap one segment into Array{T}? This can be done easily in C, but I have no idea in Julia…

6

I don’t quite understand what you want to do. Do you want to copy the arrays into a buffer before sending with mpi? Do you create a MPI_Type_struct? What does the C-program doing this look like?

Or do you want something like this:

buf = Memory{UInt8}(undef, 32)
bufptr = pointer(buf)

ivec = unsafe_wrap(Array, Ptr{Int}(bufptr), (2,), own=false)
fvec = unsafe_wrap(Array, Ptr{Float64}(bufptr+16), (2,), own=false)

where ivec uses the first part of buf, and fvec uses the second part?

2 Likes
7

Ah, yes, that’s what I want. So we need to use UInt8 to allocate the memory in Julia. It’s a bit strange that Julia doesn’t provide a more direct interface for allocating memory of known size. Thanks for your reply!

8

Oh, you can always do

bufptr = @ccall malloc(32::Int)::Ptr{Nothing}
1 Like
9

This is even less Julia-like :smile:

10

There is also Libc.malloc(32)

1 Like
11

Wow, looks much better :smiley:

1 Like