# Arrays, MPI, and broadcasting

**URL:** <https://discourse.julialang.org/t/arrays-mpi-and-broadcasting/45098>\
**Category:** Julia at Scale\
**Tags:** question\
**Created:** [August 17, 2020, 10:02am UTC](https://discourse.julialang.org/t/arrays-mpi-and-broadcasting/45098 "2020-08-17T10:02:12Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ranocha](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ranocha/32/35588_2.png) [@ranocha](https://discourse.julialang.org/u/ranocha)\
**Post date:** [August 17, 2020, 10:02am UTC](https://discourse.julialang.org/t/arrays-mpi-and-broadcasting/45098/1 "2020-08-17T10:02:12Z")

</div>

I would like to use some kind of `AbstractArray` based on MPI such that

- broadcasting can be used for things like `+`, `*`, `muladd`, which are necessary for simple explicit Runge-Kutta methods (e.g. from [OrdinaryDiffEq.jl](https://github.com/SciML/OrdinaryDiffEq.jl))
- custom functions can be used on the local part of the arrays (as usual in MPI-C/Fortran applications)

It’s okay to distribute the arrays along a single dimension across multiple CPUs.

Which approach would you recommend? I know there are some related implementations like

- [MPIStateArrays.jl in ClimateMachine.jl](https://github.com/CliMA/ClimateMachine.jl/blob/master/src/Arrays/MPIStateArrays.jl) Seems to be in active development but tied to ClimateMachine.jl
- [MPIArrays.jl](https://github.com/barche/MPIArrays.jl): Seems to be a proof-of-concept of some functionality (not including broadcasting)

---

<div class="post-metadata">

**Author:** ![jipolanco](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jipolanco/32/12129_2.png) [@jipolanco](https://discourse.julialang.org/u/jipolanco)\
**Post date:** [August 17, 2020, 10:21am UTC](https://discourse.julialang.org/t/arrays-mpi-and-broadcasting/45098/2 "2020-08-17T10:21:01Z")

</div>

You can also look at [PencilArrays](https://jipolanco.github.io/PencilFFTs.jl/dev/PencilArrays/) for yet another approach of arrays that can be distributed along one or more dimensions using MPI.

PencilArrays are currently part of the [PencilFFTs](https://github.com/jipolanco/PencilFFTs.jl) package, but can be used independently of the parallel FFT functionality. I’m planning to split them into a separate package soon.

---

<div class="post-metadata">

**Author:** ![ranocha](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ranocha/32/35588_2.png) [@ranocha](https://discourse.julialang.org/u/ranocha)\
**Post date:** [August 17, 2020, 10:30am UTC](https://discourse.julialang.org/t/arrays-mpi-and-broadcasting/45098/3 "2020-08-17T10:30:22Z")

</div>

Looks also interesting, thanks! Does PencilArrays.jl support broadcasting and calling custom MPI kernels? If so, dou you have an example for the latter?

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [August 17, 2020, 11:20am UTC](https://discourse.julialang.org/t/arrays-mpi-and-broadcasting/45098/4 "2020-08-17T11:20:32Z")

</div>

The PencilArrays is cool  
At the lower level, [`MPITopology`](https://jipolanco.github.io/PencilFFTs.jl/dev/MPITopology/#PencilFFTs.Pencils.MPITopologies.MPITopology) uses [`MPI_Cart_create`](https://www.mpich.org/static/docs/latest/www3/MPI_Cart_create.html) to define a Cartesian MPI communicator.

I guess it is nothing new really, but you can submit a job to a cluster and use information from the environment variables of the job (number of nodes, number of CPU cores per node etc.) to set up these communicators.

Not really related, but I have set up ‘bladesets’ on HPC clusters where the jobs ‘prefer’ to run on nodes close to each other in the topology.

---

<div class="post-metadata">

**Author:** ![jipolanco](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jipolanco/32/12129_2.png) [@jipolanco](https://discourse.julialang.org/u/jipolanco)\
**Post date:** [August 17, 2020, 1:42pm UTC](https://discourse.julialang.org/t/arrays-mpi-and-broadcasting/45098/5 "2020-08-17T13:42:57Z")

</div>

I’m not sure if broadcasting fully works, but that would be really easy to add if it doesn’t.

As for custom MPI kernels, yes it is possible, and it works pretty much the same way as in a C or Fortran MPI application. Indexing a `PencilArray`, either with linear or Cartesian indices, always yields a value in the local part of the array.

For some examples you can look [at the docs](https://jipolanco.github.io/PencilFFTs.jl/dev/examples/gradient/), where different approaches for iterating over arrays are compared.

---

<div class="post-metadata">

**Author:** ![ranocha](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ranocha/32/35588_2.png) [@ranocha](https://discourse.julialang.org/u/ranocha)\
**Post date:** [August 17, 2020, 2:03pm UTC](https://discourse.julialang.org/t/arrays-mpi-and-broadcasting/45098/6 "2020-08-17T14:03:46Z")

</div>

Great, thank you all! I’ll dig deeper into `PencilArray`s and file an issue or something like that if I get into trouble with broadcasting.