[ANN] Announcing GridapDistributed.jl - FEM on parallel supercomputers

We (@santiagobadia, @amartinhuertas, and @fverdugo) are pleased to announce GridapDistributed.jl, a distributed-memory extension of Gridap.jl. It provides a massively parallel, generic toolbox written in Julia for the large-scale numerical approximation of partial differential equations (PDEs). GridapDistributed.jl extends its sequential counterpart package Gridap.jl and shares its design principles and goals (see, e.g., this Julia discourse post for more details).

Why GridapDistributed.jl?

It provides a very compact, user-friendly, and mathematically-supported syntax suitable for writing Finite Element solvers for PDEs, with the bonus of being able to efficiently tackle large-scale problems on state-of-the-art supercomputers.

Satellite packages

GridapDistributed.jl can be combined with its satellite packages to achieve high performance and scalability, and applicability in real-world application problems:

  • GridapP4est.jl, for scalable mesh generation using the p4est mesh engine.
  • GridapGmsh.jl, for handling unstructured distributed meshes loaded from secondary storage.
  • GridapPETSc.jl, which provides access to the full suite of linear and nonlinear solvers in the PETSc package.

Example code

The following snippet illustrates how a Poisson problem can be solved with GridapDistributed.jl in very few lines of code.

using Gridap
using GridapDistributed
using PartitionedArrays
using GridapPETSc

# Function to be executed on each subdomain/MPI task
function main(parts)
  # Conjugate Gradients iterative solver preconditioned 
  # with algebraic multigrid (as provided by PETSc)
  options = "-ksp_type cg -pc_type gamg -ksp_monitor"
  GridapPETSc.with(args=split(options)) do
    domain = (0,1,0,1)
    # Split the box into a 4x4 Cartesian-like quadrilateral mesh
    mesh_partition = (4,4)
    model = CartesianDiscreteModel(parts,domain,mesh_partition)
    order = 2
    u((x,y)) = (x+y)^order
    f(x) = -Δ(u,x)
    reffe = ReferenceFE(lagrangian,Float64,order)
    V = TestFESpace(model,reffe,dirichlet_tags="boundary")
    U = TrialFESpace(u,V)
    Ω = Triangulation(model)
    dΩ = Measure(Ω,2*order)
    a(u,v) = ∫( ∇(v)⋅∇(u) )dΩ
    l(v) = ∫( v*f )dΩ
    op = AffineFEOperator(a,l,U,V)
    solver = PETScLinearSolver()
    uh = solve(solver,op)
    writevtk(Ω,"results",cellfields=["uh"=>uh,"grad_uh"=>∇(uh)])
  end
end

# Lay out subdomains/MPI tasks into 2x2 Cartesian subdomain/MPI task mesh
partition = (2,2)
# Trigger the main function on each subdomain/MPI task
prun(main, mpi, partition)

The code example leverages GridapPETSc.jl to solve the distributed linear system resulting from discretization. The example uses a Cartesian mesh generator built-in in GridapDistributed.jl to mesh a box. However, one may very easily modify it to use GridapGmsh.jl or GridapP4est.jl for meshing more complex domains. See this tutorial for more details.

Performance and scalability

The figures below report remarkable strong (left) and weak (scalability) of the example program above when applied to solve a 3D Poisson problem on a real-world supercomputer (Gadi at NCI, Australia).

Strong scalability Weak scalability

How to start ?

If you are further interested in the project, visit the Gridap.jl and GridapDistributed.jl repositories.

If you want to start learning how to solve PDEs with the Gridap package ecosystem, then visit our Tutorials repository.

30 Likes

Thanks for sharing your work! May I know the global (resp., local) problem sizes in the strong (resp., weak) scaling analysis?

Looking forward to using GridapDistributed.jl!

2 Likes

If I remember well, it was about 0.5B DOFs for the largest global problem in the weak scaling. But @amartinhuertas can tell perhaps better.

Hi @eneiva, thanks for your interest in the project!

For details regarding the experiment, see https://github.com/openjournals/joss-papers/blob/joss.04085/joss.04085/10.21105.joss.04085.pdf

The link is now broken. Try https://github.com/openjournals/joss-papers/blob/joss.04157/joss.04157/10.21105.joss.04157.pdf instead.

1 Like

This one works! Thanks a lot, @amartinhuertas!

Just a heads up on this post. GridapDistributed.jl paper at JOSS is out Journal of Open Source Software: GridapDistributed: a massively parallel finite element toolbox in Julia !!!

4 Likes