# \[ANN\] Announcing GridapDistributed.jl - FEM on parallel supercomputers
**URL:** https://discourse.julialang.org/t/ann-announcing-gridapdistributed-jl-fem-on-parallel-supercomputers/76138
**Category:** General Usage
**Tags:** package, announcement, pde, fem
**Created:** [February 10, 2022, 2:42am UTC](https://discourse.julialang.org/t/ann-announcing-gridapdistributed-jl-fem-on-parallel-supercomputers/76138 "2022-02-10T02:42:06Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@amartinhuertas](https://discourse.julialang.org/u/amartinhuertas)
#### Post date: [February 10, 2022, 2:42am UTC](https://discourse.julialang.org/t/ann-announcing-gridapdistributed-jl-fem-on-parallel-supercomputers/76138/1 "2022-02-10T02:42:06Z")
We (@santiagobadia, @amartinhuertas, and @fverdugo) are pleased to announce [`GridapDistributed.jl`](https://github.com/gridap/GridapDistributed.jl), a distributed-memory extension of [`Gridap.jl`](https://github.com/gridap/Gridap.jl). It provides a massively parallel, generic toolbox written in Julia for the large-scale numerical approximation of partial differential equations (PDEs). `GridapDistributed.jl` extends its sequential counterpart package [`Gridap.jl`](https://github.com/gridap/Gridap.jl) and shares its design principles and goals (see, e.g., this [Julia discourse post](https://discourse.julialang.org/t/ann-gridap-jl-a-feature-rich-finite-element-ecosystem-100-in-julia/42824) for more details).
## Why `GridapDistributed.jl`?
It provides a very compact, user-friendly, and mathematically-supported syntax suitable for writing Finite Element solvers for PDEs, with the bonus of being able to efficiently tackle large-scale problems on state-of-the-art supercomputers.
## Satellite packages
`GridapDistributed.jl` can be combined with its satellite packages to achieve high performance and scalability, and applicability in real-world application problems:
- [`GridapP4est.jl`](https://github.com/gridap/GridapP4est.jl), for scalable mesh generation using the [p4est](https://www.p4est.org/) mesh engine.
- [`GridapGmsh.jl`](https://github.com/gridap/GridapGmsh.jl), for handling unstructured distributed meshes loaded from secondary storage.
- [`GridapPETSc.jl`](https://github.com/gridap/GridapPETSc.jl), which provides access to the full suite of linear and nonlinear solvers in the [PETSc](https://petsc.org/release/) package.
## Example code
The following snippet illustrates how a Poisson problem can be solved with `GridapDistributed.jl` in very few lines of code.
```julia
using Gridap
using GridapDistributed
using PartitionedArrays
using GridapPETSc
# Function to be executed on each subdomain/MPI task
function main(parts)
# Conjugate Gradients iterative solver preconditioned
# with algebraic multigrid (as provided by PETSc)
options = "-ksp_type cg -pc_type gamg -ksp_monitor"
GridapPETSc.with(args=split(options)) do
domain = (0,1,0,1)
# Split the box into a 4x4 Cartesian-like quadrilateral mesh
mesh_partition = (4,4)
model = CartesianDiscreteModel(parts,domain,mesh_partition)
order = 2
u((x,y)) = (x+y)^order
f(x) = -Δ(u,x)
reffe = ReferenceFE(lagrangian,Float64,order)
V = TestFESpace(model,reffe,dirichlet_tags="boundary")
U = TrialFESpace(u,V)
Ω = Triangulation(model)
dΩ = Measure(Ω,2*order)
a(u,v) = ∫( ∇(v)⋅∇(u) )dΩ
l(v) = ∫( v*f )dΩ
op = AffineFEOperator(a,l,U,V)
solver = PETScLinearSolver()
uh = solve(solver,op)
writevtk(Ω,"results",cellfields=["uh"=>uh,"grad_uh"=>∇(uh)])
end
end
# Lay out subdomains/MPI tasks into 2x2 Cartesian subdomain/MPI task mesh
partition = (2,2)
# Trigger the main function on each subdomain/MPI task
prun(main, mpi, partition)
```
The code example leverages `GridapPETSc.jl` to solve the distributed linear system resulting from discretization. The example uses a Cartesian mesh generator built-in in `GridapDistributed.jl` to mesh a box. However, one may very easily modify it to use [`GridapGmsh.jl`](https://github.com/gridap/GridapGmsh.jl) or [`GridapP4est.jl`](https://github.com/gridap/GridapP4est.jl) for meshing more complex domains. See this [tutorial](https://gridap.github.io/Tutorials/dev/pages/t016_poisson_distributed/) for more details.
## Performance and scalability
The figures below report remarkable strong (left) and weak (scalability) of the example program above when applied to solve a 3D Poisson problem on a real-world supercomputer (Gadi at NCI, Australia).
|  |  |
| --- | --- |
| Strong scalability | Weak scalability |
## How to start ?
If you are further interested in the project, visit the [Gridap.jl](https://github.com/gridap/Gridap.jl) and [GridapDistributed.jl](https://github.com/gridap/GridapDistributed.jl) repositories.
If you want to start learning how to solve PDEs with the Gridap package ecosystem, then visit our [Tutorials](https://github.com/gridap/Tutorials) repository.
---
### Author:  [@eneiva](https://discourse.julialang.org/u/eneiva)
#### Post date: [February 11, 2022, 2:08pm UTC](https://discourse.julialang.org/t/ann-announcing-gridapdistributed-jl-fem-on-parallel-supercomputers/76138/2 "2022-02-11T14:08:31Z")
Thanks for sharing your work! May I know the global (resp., local) problem sizes in the strong (resp., weak) scaling analysis?
Looking forward to using GridapDistributed.jl!
---
### Author:  [@fverdugo](https://discourse.julialang.org/u/fverdugo)
#### Post date: [February 11, 2022, 3:23pm UTC](https://discourse.julialang.org/t/ann-announcing-gridapdistributed-jl-fem-on-parallel-supercomputers/76138/3 "2022-02-11T15:23:57Z")
If I remember well, it was about 0.5B DOFs for the largest global problem in the weak scaling. But @amartinhuertas can tell perhaps better.
---
### Author:  [@amartinhuertas](https://discourse.julialang.org/u/amartinhuertas)
#### Post date: [February 12, 2022, 1:05am UTC](https://discourse.julialang.org/t/ann-announcing-gridapdistributed-jl-fem-on-parallel-supercomputers/76138/4 "2022-02-12T01:05:34Z")
Hi @eneiva, thanks for your interest in the project!
For details regarding the experiment, see [https://github.com/openjournals/joss-papers/blob/joss.04085/joss.04085/10.21105.joss.04085.pdf](https://github.com/openjournals/joss-papers/blob/joss.04085/joss.04085/10.21105.joss.04085.pdf)
---
### Author:  [@amartinhuertas](https://discourse.julialang.org/u/amartinhuertas)
#### Post date: [February 14, 2022, 11:50am UTC](https://discourse.julialang.org/t/ann-announcing-gridapdistributed-jl-fem-on-parallel-supercomputers/76138/5 "2022-02-14T11:50:45Z")
The link is now broken. Try [https://github.com/openjournals/joss-papers/blob/joss.04157/joss.04157/10.21105.joss.04157.pdf](https://github.com/openjournals/joss-papers/blob/joss.04157/joss.04157/10.21105.joss.04157.pdf) instead.
---
### Author:  [@eneiva](https://discourse.julialang.org/u/eneiva)
#### Post date: [February 14, 2022, 12:09pm UTC](https://discourse.julialang.org/t/ann-announcing-gridapdistributed-jl-fem-on-parallel-supercomputers/76138/6 "2022-02-14T12:09:30Z")
This one works! Thanks a lot, @amartinhuertas!
---
### Author:  [@amartin](https://discourse.julialang.org/u/amartin)
#### Post date: [June 29, 2022, 3:35am UTC](https://discourse.julialang.org/t/ann-announcing-gridapdistributed-jl-fem-on-parallel-supercomputers/76138/7 "2022-06-29T03:35:40Z")
Just a heads up on this post. GridapDistributed.jl paper at JOSS is out [Journal of Open Source Software: GridapDistributed: a massively parallel finite element toolbox in Julia](https://joss.theoj.org/papers/10.21105/joss.04157) !!!