# Strange behaviour in the NeuralPDE framework when implementing PINN

**URL:** <https://discourse.julialang.org/t/strange-behaviour-in-the-neuralpde-framework-when-implementing-pinn/106507>\
**Category:** General Usage\
**Tags:** question\
**Created:** [November 21, 2023, 5:27am UTC](https://discourse.julialang.org/t/strange-behaviour-in-the-neuralpde-framework-when-implementing-pinn/106507 "2023-11-21T05:27:18Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Aakhash\_Sundaresan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aakhash_sundaresan/32/202478_2.png) [@Aakhash\_Sundaresan](https://discourse.julialang.org/u/Aakhash_Sundaresan)\
**Post date:** [November 21, 2023, 5:27am UTC](https://discourse.julialang.org/t/strange-behaviour-in-the-neuralpde-framework-when-implementing-pinn/106507/1 "2023-11-21T05:27:18Z")

</div>

When I try to train the Neural Network with the LBFGS or BFGS Optimization algorithm, I see that the training speed keeps fluctuating with time. Suddenly the iterations seems to have frozen and no updates are visible and again few iterations are faster. Why is this the case with LBFGS or BFGS optimizers. When training with ADAM, the iterations are consistent but for some reason the data loss doesnt go down after a fixed number of iterations. Then, increasing the network architecture is of no use and hinders perfromance…Can someone help me with the issue and advice me on how to solve a system of PDE’s like the RANS equations with energy equation and the BC’s in addition to experimental data using NeuralPDE effiiciently…?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [November 21, 2023, 10:50am UTC](https://discourse.julialang.org/t/strange-behaviour-in-the-neuralpde-framework-when-implementing-pinn/106507/2 "2023-11-21T10:50:01Z")

</div>

> [@Aakhash\_Sundaresan](#):
>
> Why is this the case with LBFGS or BFGS optimizers.

That’s due to the linesearch rejecting some steps.

> [@Aakhash\_Sundaresan](#):
>
> Can someone help me with the issue and advice me on how to solve a system of PDE’s like the RANS equations with energy equation

Add the energy equation as a constraint via a BC.

---

<div class="post-metadata">

**Author:** ![Aakhash\_Sundaresan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aakhash_sundaresan/32/202478_2.png) [@Aakhash\_Sundaresan](https://discourse.julialang.org/u/Aakhash_Sundaresan)\
**Post date:** [November 21, 2023, 12:50pm UTC](https://discourse.julialang.org/t/strange-behaviour-in-the-neuralpde-framework-when-implementing-pinn/106507/3 "2023-11-21T12:50:08Z")

</div>

But physically, how can I add the equation as a boundary condition and what is the basis of doing so? WHy would I have to shift the energy equation to the boundary condition list? Would this improve the performance of the training @ChrisRackauckas ? Also, if I have to add them, how do I exactly do that? Just cut paste the energy equation defined as eqs into the bcs array? Your help would be really appreciated and acknowledged duely in this regard…

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [November 21, 2023, 1:06pm UTC](https://discourse.julialang.org/t/strange-behaviour-in-the-neuralpde-framework-when-implementing-pinn/106507/4 "2023-11-21T13:06:21Z")

</div>

> [@Aakhash\_Sundaresan](#):
>
> But physically, how can I add the equation as a boundary condition and what is the basis of doing so?

BCs are generally just algebraic equations.

---

<div class="post-metadata">

**Author:** ![Aakhash\_Sundaresan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aakhash_sundaresan/32/202478_2.png) [@Aakhash\_Sundaresan](https://discourse.julialang.org/u/Aakhash_Sundaresan)\
**Post date:** [November 21, 2023, 1:17pm UTC](https://discourse.julialang.org/t/strange-behaviour-in-the-neuralpde-framework-when-implementing-pinn/106507/5 "2023-11-21T13:17:58Z")

</div>

So instead of this…

```julia
# Constructing the PDE System symbolically to solve the Physics Informed Neural Network (PINN's) in a SciML ecosystem
# Define the parameters and the variables in the PDE
@parameters X, Y
@variables U(..), V(..), u′u′_mean(..), u′v′_mean(..), v′v′_mean(..), Θ(..), u′θ′_mean(..), v′θ′_mean(..), P(..)

# Define the derivatives to be used in defining the PDE system
DXX = Differential(X)^2
DX = Differential(X)
DYY = Differential(Y)^2
DY = Differential(Y)

# Define the Reynolds and Prandtl number here
const Re::T = 1880
const Pr::T = 0.72
const k::T = 0.03 # Thermal condutivity of air at a particular temperature
const Dₕ::T = 0.025 # Hydraulic diameter of the square channel
const Q̇::T = 13 # This is the power input to the flat plate
const ṁ::T = 25 # LPM units
const convFactor::T = 0.00002042 # Conversion factor for converting LPM to (kg/s)
const Cₚ::T = 1006 # Specfic heat at contant pressure
const A::T = 0.1*0.02 # This is the area of the flat plate used for heating
const ΔT::T = Q̇/(ṁ*convFactor*Cₚ)
#q̅ = -(((Q̇/A)*Dₕ)/(k*ΔT))

eqs = [DX(U(X,Y)) + DY(V(X,Y)) ~ 0,
       U(X,Y)*DX(U(X,Y)) + V(X,Y)*DY(U(X,Y)) + DX(u′u′_mean(X,Y)) + DY(u′v′_mean(X,Y)) + DX(P(X,Y)) - (1/Re)*(DXX(U(X,Y)) + DYY(U(X,Y))) ~ 0,
       U(X,Y)*DX(V(X,Y)) + V(X,Y)*DY(V(X,Y)) + DX(u′v′_mean(X,Y)) + DY(v′v′_mean(X,Y)) + DY(P(X,Y)) - (1/Re)*(DXX(V(X,Y)) + DYY(V(X,Y))) ~ 0,
       U(X,Y)*DX(Θ(X,Y)) + V(X,Y)*DY(Θ(X,Y)) + DX(u′θ′_mean(X,Y)) + DY(v′θ′_mean(X,Y)) - (1/(Re*Pr))*(DXX(Θ(X,Y)) + DYY(Θ(X,Y))) ~ 0]

# Removed the boundary condition for the constant wall heat flux which is impractical for the experimental data  
bcs = [U(X,0) ~ 0, DX(U(X,0)) ~ 0, V(X,0) ~ 0, DY(V(X,0)) ~ 0, u′u′_mean(X,0) ~ 0, u′v′_mean(X,0) ~ 0, v′v′_mean(X,0) ~ 0, u′θ′_mean(X,0) ~ 0, v′θ′_mean(X,0) ~ 0, DYY(Θ(X,0)) ~ 0]
#bcs=[0~0]
# Define the space domain for symbolic processing
domains = [X ∈ Interval(0,3.25),
           Y ∈ Interval(0,0.5)]

```

Just remove the energy from the eqs list and put it in bc list?

---

<div class="post-metadata">

**Author:** ![Aakhash\_Sundaresan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aakhash_sundaresan/32/202478_2.png) [@Aakhash\_Sundaresan](https://discourse.julialang.org/u/Aakhash_Sundaresan)\
**Post date:** [November 22, 2023, 4:11am UTC](https://discourse.julialang.org/t/strange-behaviour-in-the-neuralpde-framework-when-implementing-pinn/106507/6 "2023-11-22T04:11:43Z")

</div>

@ChrisRackauckas Can you comment on the above question please?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [November 22, 2023, 6:37am UTC](https://discourse.julialang.org/t/strange-behaviour-in-the-neuralpde-framework-when-implementing-pinn/106507/7 "2023-11-22T06:37:22Z")

</div>

For a PINN it actually doesn’t matter too much in which spot it goes, unless you put it into constrained mode.