I have found some papers pointing out the slowness problem of nn solvers. Thanks.
Would you please recommend several references that (potentially) optimize nn solvers with FEM (numerical) techniques? For example, I am wondering whether optimizing linear equation solving (the “Distributed Dense, Structured, and Sparse Linear Algebra” tool) is still essential for nn solvers. After reading the 18.337J/6.338J lectures on solving PDEs, my understanding is that even for numerical PDEs, most likely we need to solve linear equations only for stiff PDEs. As for nn solvers, although some nn solvers (PINN, Learning data driven discretizations for partial differential equations) discretize the PDEs, looks like they do not build or solve linear equations.
Another question related to nn PDE solvers, you mentioned some numerical solvers in the universal differential equation post.
There is this property of ODEs called stiffness, and when it comes into play, the simple Runge-Kutta method or Adams-Bashforth-Moulton methods are no longer stable enough to accurately solve the equations. Thus when looking at the universal partial differential equations, we had to make use of a set of ODE solvers which have package implementations in Julia and Fortran.
In my understanding, NeuralNetDiffEq.jl and DiffEqFlux.jl provide differentiable PDE solvers while DifferentialEquations.jl provides conventional numerical solvers (Runge-Kutta method, Adams-Bashforth-Moulton, etc). Do you mean that we can / should also differentiate the numerical solvers?