# ANN: MPI.jl v0.10.0: new build process and CUDA-aware support
**URL:** https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601
**Category:** Julia at Scale
**Created:** [August 16, 2019, 4:18am UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601 "2019-08-16T04:18:08Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)
#### Post date: [August 16, 2019, 4:18am UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/1 "2019-08-16T04:18:08Z")
I have just tagged a [new version of MPI.jl](https://github.com/JuliaParallel/MPI.jl/releases/tag/v0.10.0). Though the user-facing interface is largely the same, there has been extensive work underneath to internally use the C API (instead of the Fortran one). As a result, the build process is much simpler (it no longer requires CMake or a Fortran compiler). Additionally, it also directly supports CUDA-aware MPI libraries, allowing CuArrays to be passed directly as buffers (thanks to Seyoon Ko).
I would greatly appreciate if people are able to try it out, especially with different clusters and MPI implementations.
---
### Author:  [@johnh](https://discourse.julialang.org/u/johnh)
#### Post date: [August 16, 2019, 5:06am UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/2 "2019-08-16T05:06:56Z")
Is it worth sharing this on the OpenMPI mailing list? I would guess you work closely with those guys anyway.
---
### Author:  [@johnh](https://discourse.julialang.org/u/johnh)
#### Post date: [August 16, 2019, 5:09am UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/3 "2019-08-16T05:09:22Z")
I just noticed this is in Julia at Scale!
Picture me doing a happy dance. Or maybe my Miata doing doughnuts (see avatar image).
---
### Author:  [@kose-y](https://discourse.julialang.org/u/kose-y)
#### Post date: [September 3, 2019, 3:20am UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/4 "2019-09-03T03:20:00Z")
Thank you @simonbyrne! Here are some code examples using the new version of MPI.jl.
[https://github.com/kose-y/Julia-MPI-CuArray](https://github.com/kose-y/Julia-MPI-CuArray)
---
### Author:  [@mcreel](https://discourse.julialang.org/u/mcreel)
#### Post date: [September 4, 2019, 12:46am UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/5 "2019-09-04T00:46:16Z")
Basic send Send, Recv! works fine on an ordinary Linux cluster with OpenMPI. Thanks!
---
### Author:  [@rveltz](https://discourse.julialang.org/u/rveltz)
#### Post date: [September 9, 2019, 4:35pm UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/6 "2019-09-09T16:35:24Z")
Nice! Is there a way to partition a big `CuArray` into `p` parts like the _slab_ decomposition?
---
### Author:  [@kose-y](https://discourse.julialang.org/u/kose-y)
#### Post date: [September 10, 2019, 12:10am UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/7 "2019-09-10T00:10:53Z")
I’m working on it based on barche/MPIArrays.jl. I think it can be released some time this year.
---
### Author:  [@rveltz](https://discourse.julialang.org/u/rveltz)
#### Post date: [September 10, 2019, 5:55am UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/8 "2019-09-10T05:55:22Z")
> [@kose-y](#):
>
> barche/MPIArrays.jl
You may be interested in the python [lib](https://github.com/mpi4py/mpi4py-fft).
---
### Author:  [@kose-y](https://discourse.julialang.org/u/kose-y)
#### Post date: [September 10, 2019, 7:36am UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/9 "2019-09-10T07:36:22Z")
@rveltz This looks interesting. Thanks for the reference!
---
### Author:  [@johnh](https://discourse.julialang.org/u/johnh)
#### Post date: [September 13, 2019, 10:11am UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/10 "2019-09-13T10:11:48Z")
I have access to a DGX-1 GPU system. I’m not sure how much time I can get on it.
If there are any tests which could be run for this package I could give it a try.
---
### Author:  [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)
#### Post date: [September 13, 2019, 4:11pm UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/11 "2019-09-13T16:11:45Z")
Do you know if it has a CUDA-aware MPI? If so, would be good to run the test suite with `JULIA_PROJECT` set to `[pkgdir]/test/cudaenv`.
---
### Author:  [@samo](https://discourse.julialang.org/u/samo)
#### Post date: [December 3, 2019, 5:53pm UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/12 "2019-12-03T17:53:47Z")
@kose-y and @simonbyrne, a big thanks for making CUDA-aware MPI available to the Julia community! Unfortunately, when I tried CUDA-aware MPI with MPI.jl on two different systems, it failed in both cases. Could you have a look at [this post](https://discourse.julialang.org/t/error-segfault-in-basic-test-of-cuda-aware-mpi/31755) where I reported the errors? It would be fantastic if I got it to work before the AGU conference this weekend… 🙂
---
### Author:  [@rveltz](https://discourse.julialang.org/u/rveltz)
#### Post date: [May 31, 2020, 8:06pm UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/13 "2020-05-31T20:06:57Z")
Hi,
Is there any news about Distributed Arrays with MPI, or mixing CuArrays with MPI?
---
### Author:  [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)
#### Post date: [June 1, 2020, 3:25pm UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/14 "2020-06-01T15:25:18Z")
You will need to build and link against a CUDA-aware MPI implementation, but other than that, CuArrays should work with MPI.jl:
[https://juliaparallel.github.io/MPI.jl/stable/usage/#CUDA-aware-MPI-support-1](https://juliaparallel.github.io/MPI.jl/stable/usage/#CUDA-aware-MPI-support-1)
There is a proof-of-concept package of distributed arrays:
> **[GitHub - barche/MPIArrays.jl: Distributed arrays based on MPI onesided...](https://github.com/barche/MPIArrays.jl)**
>
> Distributed arrays based on MPI onesided communication - GitHub - barche/MPIArrays.jl: Distributed arrays based on MPI onesided communication
but other than that, no. It really depends on what sort of functionality you will want, e.g. we’ve built our own to provide support for ghost elements ([https://github.com/CliMA/ClimateMachine.jl/blob/master/src/Arrays/MPIStateArrays.jl](https://github.com/CliMA/ClimateMachine.jl/blob/master/src/Arrays/MPIStateArrays.jl)), but it makes a lot of assumptions about data layout, etc.
---
### Author:  [@rveltz](https://discourse.julialang.org/u/rveltz)
#### Post date: [June 2, 2020, 8:31am UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/15 "2020-06-02T08:31:00Z")
I see. Ideally, I want to perform ffts on multiGPU…
---
### Author:  [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)
#### Post date: [June 2, 2020, 6:34pm UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/16 "2020-06-02T18:34:24Z")
There is some discussion here: [PencilFFTs on GPUs? · Issue #3 · jipolanco/PencilFFTs.jl · GitHub](https://github.com/jipolanco/PencilFFTs.jl/issues/3)
---
### Author:  [@rveltz](https://discourse.julialang.org/u/rveltz)
#### Post date: [June 2, 2020, 7:11pm UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/17 "2020-06-02T19:11:02Z")
yes! That is exactly what I am looking for
---
### Author:  [@jipolanco](https://discourse.julialang.org/u/jipolanco)
#### Post date: [June 2, 2020, 7:35pm UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/18 "2020-06-02T19:35:12Z")
As mentioned in the issue, it would be great to add GPU support to PencilFFTs. The MPI-heavy part of the code was recently refactored, and hopefully, extending things to work with CuArrays should not take too much work.
I personally don’t have any experience with CUDA-aware MPI, and I don’t have access to multi-GPU systems (that I’m aware of), so any help with this is most welcome!
---
### Author:  [@kose-y](https://discourse.julialang.org/u/kose-y)
#### Post date: [October 28, 2020, 1:34pm UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/19 "2020-10-28T13:34:36Z")
I have just released it here.
[https://github.com/kose-y/DistStat.jl](https://github.com/kose-y/DistStat.jl)
---
### Author:  [@rveltz](https://discourse.julialang.org/u/rveltz)
#### Post date: [October 28, 2020, 4:59pm UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601/20 "2020-10-28T16:59:51Z")
This is really cool stuff. I am wondering if you should not rename it `MPIArraysV2.jl` 😅
It would be a shame to have this work “duplicated”. I see there is already `PencilArrays` as well.
[Next page](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601.md?page=2)