# \[ANN\] MPIMapReduce.jl: A simplified mapreduce using MPI

**URL:** <https://discourse.julialang.org/t/ann-mpimapreduce-jl-a-simplified-mapreduce-using-mpi/58635>\
**Category:** Package Announcements\
**Tags:** mpi, distributed\
**Created:** [April 5, 2021, 6:35pm UTC](https://discourse.julialang.org/t/ann-mpimapreduce-jl-a-simplified-mapreduce-using-mpi/58635 "2021-04-05T18:35:38Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jishnub](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jishnub/32/33620_2.png) [@jishnub](https://discourse.julialang.org/u/jishnub)\
**Post date:** [April 5, 2021, 6:35pm UTC](https://discourse.julialang.org/t/ann-mpimapreduce-jl-a-simplified-mapreduce-using-mpi/58635/1 "2021-04-05T18:35:39Z")

</div>

MPI is the state-of-the-art in multiprocessing on HPC clusters, however the syntax of MPI may be a bit unfamiliar for beginners. [MPI.jl](https://github.com/JuliaParallel/MPI.jl) provides Julia wrappers for MPI that make the process much simpler. This package provides wrappers around MPI.jl that make the process even simpler, and perhaps puts it in a form that is more familiar to Julia users.

A sample script to evaluate a parallel map-reduce:

```julia
using MPIMapReduce
using MPI
MPI.Init()

y1 = pmapreduce(x -> ones(2) * x, +, 1:5)

if MPI.Comm_rank(MPI.COMM_WORLD) == 0
    show(stdout, MIME"text/plain"(), y1)
    println()
end

```

This produces the output

```julia
2-element Array{Float64,1}:
 15.0
 15.0

```

Note that the reduction is applied elementwise, so this is equivalent to `mapreduce(x -> ones(2)*x, (x,y) -> x .+ y, 1:5)` in Julia.

The package also exports another function `pmapgatherv`, that peforms a map followed by a concatenation. The sample script

```julia
using MPIMapReduce
using MPI
MPI.Init()

y = pmapgatherv(x -> ones(2) * x^2, hcat, 1:5)

if MPI.Comm_rank(MPI.COMM_WORLD) == 0
    show(stdout, MIME"text/plain"(), y)
    println()
end

```

produces the output

```julia
2×5 Array{Float64,2}:
 1.0 4.0 9.0 16.0 25.0
 1.0 4.0 9.0 16.0 25.0

```

In this function the reduction is not performed elementwise, so this is actually equivalent to Julia’s `mapreduce`. The naming reflects the fact that the function uses MPI’s gatherv function under the hood.

This is pretty new and I have not tested this extensively, but basic tests appear to work. Issues and PR are welcome!

Link to package: [GitHub - jishnub/MPIMapReduce.jl: An MPI-based distributed map-reduce function for Julia](https://github.com/jishnub/MPIMapReduce.jl)