# Parallel computing and GPU support in neuralPDE.jl package

**URL:** <https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976>\
**Category:** New to Julia\
**Tags:** package, performance, neural-network\
**Created:** [October 15, 2023, 12:13pm UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976 "2023-10-15T12:13:08Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![Aakhash\_Sundaresan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aakhash_sundaresan/32/202478_2.png) [@Aakhash\_Sundaresan](https://discourse.julialang.org/u/Aakhash_Sundaresan)\
**Post date:** [October 19, 2023, 12:59pm UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/21 "2023-10-19T12:59:17Z")

</div>

I’m going through the documentation that you had shared…but I’m confused about how to convert the documentation given for the Lotka Voltera model to suite my application Chris…Could you please elaborate on this one and guide me please?

---

<div class="post-metadata">

**Author:** ![Aakhash\_Sundaresan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aakhash_sundaresan/32/202478_2.png) [@Aakhash\_Sundaresan](https://discourse.julialang.org/u/Aakhash_Sundaresan)\
**Post date:** [October 19, 2023, 1:51pm UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/22 "2023-10-19T13:51:52Z")

</div>

Chris…At least can you hint me on how I can add the pde system and the data to the code that is given in the documentation for UDE’s…(the documentation in the link that you shared)…please…

---

<div class="post-metadata">

**Author:** ![Aakhash\_Sundaresan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aakhash_sundaresan/32/202478_2.png) [@Aakhash\_Sundaresan](https://discourse.julialang.org/u/Aakhash_Sundaresan)\
**Post date:** [October 19, 2023, 9:58pm UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/23 "2023-10-19T21:58:34Z")

</div>

```julia
# Define the hybrid model
function ude_dynamics!(du, u, p, t, p_true)
    û = U(u, p, _st)[1] # Network prediction
    du[1] = p_true[1] * u[1] + û[1]
    du[2] = -p_true[4] * u[2] + û[2]
end

# Closure with the known parameter
nn_dynamics!(du, u, p, t) = ude_dynamics!(du, u, p, t, p_)
# Define the problem
prob_nn = ODEProblem(nn_dynamics!, Xₙ[:, 1], tspan, p)

```

How would I translate this idea for my PDE system of equations which involve 1 continuity, 2 momentum and an energy equation (2D system) along with data Chris?

---

<div class="post-metadata">

**Author:** ![Aakhash\_Sundaresan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aakhash_sundaresan/32/202478_2.png) [@Aakhash\_Sundaresan](https://discourse.julialang.org/u/Aakhash_Sundaresan)\
**Post date:** [October 20, 2023, 10:21am UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/24 "2023-10-20T10:21:59Z")

</div>

Hi Chris…Can you just explain me how I can do the UDE formulation of the RANS equations with energy on a PDE discretization please?..  
I have attached the screenshot containing the equations that I’m using to solve along with the BC’s…

 ![Screenshot 2023-10-20 at 9.23.35 pm](https://global.discourse-cdn.com/julialang/original/3X/1/e/1e62a274ff3d1170fe6138f7749c6424b07c347c.png)

---

<div class="post-metadata">

**Author:** ![Aakhash\_Sundaresan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aakhash_sundaresan/32/202478_2.png) [@Aakhash\_Sundaresan](https://discourse.julialang.org/u/Aakhash_Sundaresan)\
**Post date:** [October 21, 2023, 12:25am UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/25 "2023-10-21T00:25:25Z")

</div>

Can you atleast divert me to some example which exists in the documentation that is more closely aligned to my problem specification? That is solving the PDE using a UDE forumation…I’m confused in the implementation of the same…Thanks…

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [October 22, 2023, 5:14pm UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/26 "2023-10-22T17:14:43Z")

</div>

Just semi-discretize the UDE, like is done in the papers. See things like:

- [Partial Differential Equation (PDE) Constrained Optimization · SciMLSensitivity.jl](https://docs.sciml.ai/SciMLSensitivity/dev/examples/pde/pde_constrained/)
- [https://github.com/ChrisRackauckas/universal\_differential\_equations/blob/master/FisherKPP/Fisher-KPP-CNN-Small.jl](https://github.com/ChrisRackauckas/universal_differential_equations/blob/master/FisherKPP/Fisher-KPP-CNN-Small.jl)
- [https://arxiv.org/abs/2303.13555](https://arxiv.org/abs/2303.13555)

---

<div class="post-metadata">

**Author:** ![Aakhash\_Sundaresan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aakhash_sundaresan/32/202478_2.png) [@Aakhash\_Sundaresan](https://discourse.julialang.org/u/Aakhash_Sundaresan)\
**Post date:** [November 6, 2023, 12:17am UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/27 "2023-11-06T00:17:33Z")

</div>

@ChrisRackauckas Can you just give me an example of how to do a UDE formulation on a set of PDE’s and how to incorporate the neural networks for extrapolating accurately beyond the experimental data that I have…?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [November 6, 2023, 12:21am UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/28 "2023-11-06T00:21:25Z")

</div>

See those links above. Or see:

> <https://github.com/viniviena/ude_chromatography/tree/master/UDE_paper_chromatography>
>
> //github.com/viniviena/ude\_chromatography/tree/master/UDE\_paper\_chromatography

which is the repo for [[2303.13555] Efficient hybrid modeling and sorption model discovery for non-linear advection-diffusion-sorption systems: A systematic scientific machine learning approach](https://arxiv.org/abs/2303.13555)

---

<div class="post-metadata">

**Author:** ![Aakhash\_Sundaresan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aakhash_sundaresan/32/202478_2.png) [@Aakhash\_Sundaresan](https://discourse.julialang.org/u/Aakhash_Sundaresan)\
**Post date:** [November 6, 2023, 12:38am UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/29 "2023-11-06T00:38:40Z")

</div>

@ChrisRackauckas I was going through the documentation chris, but I feel it too overwhelming to grasp the program construct and the logic…Could you please help me with any simpler example that is available or any tutorials which can explain in a step by step manner how to scientific machine learning for extrapolation in the non-existant data region…Please?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [November 6, 2023, 12:42am UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/30 "2023-11-06T00:42:03Z")

</div>

[Automatically Discover Missing Physics by Embedding Machine Learning into Differential Equations · Overview of Julia's SciML](https://docs.sciml.ai/Overview/stable/showcase/missing_physics/) is a step by step tutorial, or follow the chromatography repo which does exactly that on a PDE system.

---

<div class="post-metadata">

**Author:** ![Aakhash\_Sundaresan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aakhash_sundaresan/32/202478_2.png) [@Aakhash\_Sundaresan](https://discourse.julialang.org/u/Aakhash_Sundaresan)\
**Post date:** [November 6, 2023, 1:41am UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/31 "2023-11-06T01:41:49Z")

</div>

@ChrisRackauckas So the paper in the Chromatography repo uses the finite element PDE discretization scheme right? Can I just use the second order accurate differences for all the terms in my equations?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [November 6, 2023, 1:56am UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/32 "2023-11-06T01:56:08Z")

</div>

You can use whatever discretization you want.

---

<div class="post-metadata">

**Author:** ![Aakhash\_Sundaresan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aakhash_sundaresan/32/202478_2.png) [@Aakhash\_Sundaresan](https://discourse.julialang.org/u/Aakhash_Sundaresan)\
**Post date:** [November 6, 2023, 2:44am UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/33 "2023-11-06T02:44:11Z")

</div>

@ChrisRackauckas Why are only selected terms in the PDE replaced with neural networks as universal approximators and what is the rationale behind choosing which term in the PDE to be replaced with a neural network?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [November 6, 2023, 2:45am UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/34 "2023-11-06T02:45:09Z")

</div>

> [@Aakhash\_Sundaresan](#):
>
> what is the rationale behind choosing which term in the PDE to be replaced with a neural network?

Those are the pieces that you don’t know.

> [@Aakhash\_Sundaresan](#):
>
> Why are only selected terms in the PDE replaced with neural networks as universal approximators

Because you know things.

Please see the videos and read the papers.

---

<div class="post-metadata">

**Author:** ![Aakhash\_Sundaresan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aakhash_sundaresan/32/202478_2.png) [@Aakhash\_Sundaresan](https://discourse.julialang.org/u/Aakhash_Sundaresan)\
**Post date:** [November 15, 2023, 9:06am UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/35 "2023-11-15T09:06:50Z")

</div>

@ChrisRackauckas Is there any way that we can parallelize the training of multiple networks in the neuralPDE framework?..If so how can we do it to achieve maximum performance?

---

<div class="post-metadata">

**Author:** ![Aakhash\_Sundaresan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aakhash_sundaresan/32/202478_2.png) [@Aakhash\_Sundaresan](https://discourse.julialang.org/u/Aakhash_Sundaresan)\
**Post date:** [November 16, 2023, 7:14am UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/36 "2023-11-16T07:14:21Z")

</div>

@ChrisRackauckas Is it that for GPU training, only the initial parameters and experimental data (if available) needs to be put on the GPU right? We need not put the neural network chain on to the GPU?

---

<div class="post-metadata">

**Author:** ![Aakhash\_Sundaresan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aakhash_sundaresan/32/202478_2.png) [@Aakhash\_Sundaresan](https://discourse.julialang.org/u/Aakhash_Sundaresan)\
**Post date:** [November 16, 2023, 8:12am UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/37 "2023-11-16T08:12:16Z")

</div>

@ChrisRackauckas If I have additional loss as the data loss, how do I program it for computing it on the GPU in the NeuralPDE package?

---

<div class="post-metadata">

**Author:** ![Aakhash\_Sundaresan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aakhash_sundaresan/32/202478_2.png) [@Aakhash\_Sundaresan](https://discourse.julialang.org/u/Aakhash_Sundaresan)\
**Post date:** [November 16, 2023, 8:16am UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/38 "2023-11-16T08:16:40Z")

</div>

In the NeuralPDE documentation usage of GPU’s is explained only wth incorporation of Physics. It is not entirely clear if there is additional data loss term to be included in the training, how to do so?. I think if I put only the initial parameters onto the GPU, it throws an Error saying that “LoadError: GPU compilation of MethodInstance for…failed!”.

---

<div class="post-metadata">

**Author:** ![Aakhash\_Sundaresan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aakhash_sundaresan/32/202478_2.png) [@Aakhash\_Sundaresan](https://discourse.julialang.org/u/Aakhash_Sundaresan)\
**Post date:** [November 17, 2023, 2:59am UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/39 "2023-11-17T02:59:05Z")

</div>

@ChrisRackauckas I would like to convey to you that the problem use case outlined in ude\_chromatography repo does’nt use system of equations (involves only pde equation and boundary equation) and also, the way the data that is being fed to the neural network is little confusing to translate to my use case Chris. Can you please help me as I’m not able to solve my problem efficiently using the NeuralPDE efficiently and have to rely on alternate techniques to solve the problem currently at hand…Also, I was looking at using the MethodOfLines.jl to solve the problem. But, how do I incorporate the experimental data into that. Also does it scale it well for more than 2 system of pde equations and boundary conditions? The documentation does’nt explain the application of MethodOfLines for system of PDE’s. Is it possible to shed some light on this one please ?

---

<div class="post-metadata">

**Author:** ![Aakhash\_Sundaresan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aakhash_sundaresan/32/202478_2.png) [@Aakhash\_Sundaresan](https://discourse.julialang.org/u/Aakhash_Sundaresan)\
**Post date:** [November 17, 2023, 4:07am UTC](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976/40 "2023-11-17T04:07:28Z")

</div>

┌ Warning: Using `gpu` inside performance critical code will cause massive slowdowns due to type inference failure. Please update your code to use `gpu_device` API.  
└ @ Lux C:\Users\Sunda.julia\packages\Lux\hlo4t\src\deprecated.jl:32  
┌ Warning: No functional GPU backend found! Defaulting to CPU.  
│  
│ 1. If no GPU is available, nothing needs to be done.  
│ 2. If GPU is available, load the corresponding trigger package.  
└ @ LuxDeviceUtils C:\Users\Sunda.julia\packages\LuxDeviceUtils\rMeCf\src\LuxDeviceUtils.jl:158

Here is the issue when I use the exact code in the documentation and run it on my NVIDIA GPU, and its massively slow, slower than CPU for the case of a example problem outlined in the documentaiton for “Training NeuralPDE on the GPU”.

I request you if you can kindly look into the issue and prompt me a solution to this Chris…

[Previous page](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976.md?page=1)

[Next page](https://discourse.julialang.org/t/parallel-computing-and-gpu-support-in-neuralpde-jl-package/104976.md?page=3)