# \[ANN\] PencilFFTs: parallel FFTs of MPI-distributed arrays
**URL:** https://discourse.julialang.org/t/ann-pencilffts-parallel-ffts-of-mpi-distributed-arrays/33191
**Category:** Package Announcements
**Tags:** fftw, hpc, parallel, cluster, mpi
**Created:** [January 10, 2020, 12:37pm UTC](https://discourse.julialang.org/t/ann-pencilffts-parallel-ffts-of-mpi-distributed-arrays/33191 "2020-01-10T12:37:07Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@jipolanco](https://discourse.julialang.org/u/jipolanco)
#### Post date: [January 10, 2020, 12:37pm UTC](https://discourse.julialang.org/t/ann-pencilffts-parallel-ffts-of-mpi-distributed-arrays/33191/1 "2020-01-10T12:37:07Z")
I’m happy to announce [PencilFFTs.jl](https://github.com/jipolanco/PencilFFTs.jl), a Julia package for performing parallel fast Fourier transforms using MPI.
This package is an attempt to bring parallel FFTs to Julia. It is well adapted for running on thousands of cores in CPU computing clusters (support for GPUs may arrive in the future!). The package uses a similar strategy as “standard” C/C++/Fortran codes that are commonly used in research. Namely, it decomposes a multidimensional array along one or more dimensions, so that each MPI process handles one of the resulting portions of the array. This is illustrated by the image below, which shows the 2D decomposition of a 3D array over 12 MPI processes. This particular configuration is sometimes called “pencil” decomposition, hence the name of this package.
Given a decomposition, each process can perform serial FFTs along the non-decomposed dimensions. Then, to transform along the remaining dimensions, one has to redistribute (or _transpose_) data among MPI processes as shown in the image. The method is described in a bit more detail in the [package documentation](https://jipolanco.github.io/PencilFFTs.jl/dev/).
This kind of approach is commonly used in [spectral methods](https://en.wikipedia.org/wiki/Spectral_method) that may use Fourier or Chebyshev expansions to decompose the solution of the governing equations. Applications in physics include solvers for the [Navier–Stokes equations](https://en.wikipedia.org/wiki/Navier%E2%80%93Stokes_equations) to describe fluid turbulence in periodic or wall-bounded domains, and for the [Gross–Pitaevskii equation](https://en.wikipedia.org/wiki/Gross%E2%80%93Pitaevskii_equation) describing Bose–Einstein condensates. See also this [very relevant discussion](https://discourse.julialang.org/t/implementing-parallel-fluids-code-in-julia-parallel-fftw-mpi-jl/2689) here on Discourse from a few years ago.
As shown in the [benchmarks](https://jipolanco.github.io/PencilFFTs.jl/dev/benchmarks/), the performance and scalability of PencilFFTs are close to those displayed by the [P3DFFT](https://www.p3dfft.net/) library written in Fortran, which is probably the most popular library providing FFTs with pencil decomposition. Further performance improvements are possible and will likely arrive in the near future.
The implementation of PencilFFTs is generic in the sense that it can work efficiently with geometries of arbitrary dimension. Moreover, one can choose among all transforms supported by [FFTW.jl](https://github.com/JuliaMath/FFTW.jl), including complex-to-complex, real-to-complex and real-to-real transforms. It’s also possible to combine different transforms along each dimension.
The parallel FFT functionality is implemented on top of a [`Pencils`](https://jipolanco.github.io/PencilFFTs.jl/dev/Pencils/) module that can also be used by itself, e.g. if one only wants to conveniently deal with MPI-distributed arrays without needing FFTs. I’m considering making this module a separate package. Further functionality that I’d like to associate to this module include exchange of ghost cells between neighbouring MPI processes, and parallel I/O using [HDF5.jl](https://github.com/JuliaIO/HDF5.jl).
More details are available in [the documentation](https://jipolanco.github.io/PencilFFTs.jl/dev/), including a [tutorial](https://jipolanco.github.io/PencilFFTs.jl/dev/tutorial/) and a few more [examples](https://jipolanco.github.io/PencilFFTs.jl/dev/examples/gradient/).
---
### Author:  [@John\_Gibson](https://discourse.julialang.org/u/John_Gibson)
#### Post date: [January 10, 2020, 12:43pm UTC](https://discourse.julialang.org/t/ann-pencilffts-parallel-ffts-of-mpi-distributed-arrays/33191/2 "2020-01-10T12:43:39Z")
❤ ❤ ❤ ❤
That’ is fantastic news. I can’t wait to try it out!
---
### Author:  [@antoine-levitt](https://discourse.julialang.org/u/antoine-levitt)
#### Post date: [January 10, 2020, 2:03pm UTC](https://discourse.julialang.org/t/ann-pencilffts-parallel-ffts-of-mpi-distributed-arrays/33191/3 "2020-01-10T14:03:38Z")
That’s awesome, thanks for doing this! I might have an use for it in the near future.
- Can you comment on the difference between this approach and the one used in the MPI mode of FFTW? (sorry if this is very naive)
- Although having a pure julia solution is of course the best solution, I guess linking to a binary P3DFFT is also an option? Any advantages for users in doing the former?
- It looks like you’ve been able to run this on a large number of nodes. How has your experience been doing that?
---
### Author:  [@jipolanco](https://discourse.julialang.org/u/jipolanco)
#### Post date: [January 10, 2020, 2:34pm UTC](https://discourse.julialang.org/t/ann-pencilffts-parallel-ffts-of-mpi-distributed-arrays/33191/4 "2020-01-10T14:34:52Z")
Great! I hope it can be useful.
> Can you comment on the difference between this approach and the one used in the MPI mode of FFTW? (sorry if this is very naive)
The difference is that the MPI FFTW routines can only perform 1D decomposition, which means that you’re more limited in the number of MPI processes that you can use. For instance, if you have 512^3 grid points, you can use up to 512 MPI processes. Of course, in this case you can combine threads and MPI if you want to go beyond. This is what many people do and it works pretty well.
> Although having a pure julia solution is of course the best solution, I guess linking to a binary P3DFFT is also an option? Any advantages for users in doing the former?
I guess you could do that, but I have no idea how that would play with MPI.jl, which currently wraps the MPI C API. Also, the functionality of the Fortran P3DFFT is more limited, and for example it cannot do complex-to-complex transforms. I can’t say a lot about other libraries, only that I tried the more recent C++ version of P3DFFT, which has more functionality, and I didn’t have a very good experience. Actually my first benchmarks were done against that version…
> It looks like you’ve been able to run this on a large number of nodes. How has your experience been doing that?
I had the [same issues](https://discourse.julialang.org/t/precompilation-error-using-hpc/17094) that other people [have encountered](https://discourse.julialang.org/t/run-a-julia-application-at-large-scale-on-thousands-of-nodes/23873), related to the fact that when you launch Julia with MPI, all processes precompile the same code, and there are race conditions and some other issues. As suggested in one of those threads, I ended up running Julia with `--compiled-modules=no` to workaround the issue. It’s not ideal, but it works. You can also take a look at the [SLURM submitting script](https://github.com/jipolanco/PencilFFTs.jl/blob/master/benchmarks/clusters/idris.jean_zay/submit_slurm.sh) that I used for the benchmarks.
---
### Author:  [@ranocha](https://discourse.julialang.org/u/ranocha)
#### Post date: [December 7, 2021, 7:09pm UTC](https://discourse.julialang.org/t/ann-pencilffts-parallel-ffts-of-mpi-distributed-arrays/33191/5 "2021-12-07T19:09:08Z")
@jipolanco Nice! Do you have an academic publication coming with it?
---
### Author:  [@jipolanco](https://discourse.julialang.org/u/jipolanco)
#### Post date: [December 7, 2021, 9:02pm UTC](https://discourse.julialang.org/t/ann-pencilffts-parallel-ffts-of-mpi-distributed-arrays/33191/6 "2021-12-07T21:02:12Z")
Thanks for your interest! Not yet, I have focused on other things for now. But it’s in my plans to start doing some physics with this package.
In any case the approach is nothing new, and there are similar libraries in other languages. After working on PencilFFTs.jl I found out about [PFFT](https://github.com/mpip/pfft), a C library which offers many of the same functionalities including arbitrary dimensions. There is [a paper](https://doi.org/10.1137/120885887) describing the library. There is also [P3DFFT](https://p3dfft.readthedocs.io/en/latest/home.html) ([paper](https://doi.org/10.1137/11082748X)), which is quite popular in my field, and which is used in the PencilFFTs benchmarks.
---
### Author:  [@ranocha](https://discourse.julialang.org/u/ranocha)
#### Post date: [December 8, 2021, 8:04am UTC](https://discourse.julialang.org/t/ann-pencilffts-parallel-ffts-of-mpi-distributed-arrays/33191/7 "2021-12-08T08:04:27Z")
Thanks! Let me rephrase my question a bit: Is there a way to cite your effort and results (quite nice performance scaling using Julia) in an academic context?
---
### Author:  [@jipolanco](https://discourse.julialang.org/u/jipolanco)
#### Post date: [December 8, 2021, 10:09am UTC](https://discourse.julialang.org/t/ann-pencilffts-parallel-ffts-of-mpi-distributed-arrays/33191/8 "2021-12-08T10:09:50Z")
You’re right, it would be a good idea to have a citable paper. For now there’s just a citable entry on [Zenodo](https://zenodo.org/record/5336489). I will consider submitting a paper to JOSS within the next few months.