# Neural Networks combined with Diffeq

**URL:** <https://discourse.julialang.org/t/neural-networks-combined-with-diffeq/58271>\
**Category:** New to Julia\
**Tags:** sciml\
**Created:** [March 31, 2021, 2:41am UTC](https://discourse.julialang.org/t/neural-networks-combined-with-diffeq/58271 "2021-03-31T02:41:22Z")\
**Posts on this page:** 7\
**Page:** 2

<div class="post-metadata">

**Author:** ![Allan\_Baker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/allan_baker/32/42645_2.png) [@Allan\_Baker](https://discourse.julialang.org/u/Allan_Baker)\
**Post date:** [April 2, 2021, 4:44pm UTC](https://discourse.julialang.org/t/neural-networks-combined-with-diffeq/58271/21 "2021-04-02T16:44:23Z")

</div>

I can call the loss function and manually change values in the vector of network parameters and I get different losses, but the gradient fails.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 2, 2021, 9:37pm UTC](https://discourse.julialang.org/t/neural-networks-combined-with-diffeq/58271/22 "2021-04-02T21:37:03Z")

</div>

This all means nothing without knowing the code that is causing it.

---

<div class="post-metadata">

**Author:** ![Allan\_Baker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/allan_baker/32/42645_2.png) [@Allan\_Baker](https://discourse.julialang.org/u/Allan_Baker)\
**Post date:** [April 5, 2021, 4:00pm UTC](https://discourse.julialang.org/t/neural-networks-combined-with-diffeq/58271/23 "2021-04-05T16:00:17Z")

</div>

I know. I’m trying to think how to construct a simpler example.

I started with a diffeq based three-dof which has parameters that setup the details, similar to how the trebuche example problem might work. So it started out complicated. Some of those parameters I want to optimize, but I really want a neural network to output those parameters and set them prior to the solve through time.

I’m still looking for a good example that hilites this workflow, which I would think would be very common, but I don’t really find anything, but there is a lot of material out there so I probably missed it.

---

<div class="post-metadata">

**Author:** ![triceert](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/triceert/32/21896_2.png) [@triceert](https://discourse.julialang.org/u/triceert)\
**Post date:** [May 6, 2021, 1:23pm UTC](https://discourse.julialang.org/t/neural-networks-combined-with-diffeq/58271/24 "2021-05-06T13:23:41Z")

</div>

I think it has to do with preallocations for e.g. mul! where the left hand side is of the wrong type.  
I tried to strip down an MWE which leads to the same error:

```julia
using BandedMatrices
using DiffEqFlux
using OrdinaryDiffEq
using DiffEqSensitivity
using Flux
using GalacticOptim
using LinearAlgebra
using Zygote

#band stuff

function Dband(n,l,u,Δx,D)
  A = BandedMatrix{Float64}(undef, (n,n), (l,u))
  A[band(0)] .= -2*1/Δx^2
  A[band(1)] .= 1/Δx^2
  A[band(-1)] .= 1/Δx^2
  A[1,1:2] .= 0
  A[n,n] = -1/Δx^2
  A[n,n-1] = 1/Δx^2
  A = A.*D
  return A
end

function Vband(n,l,u,Δx,v)
  A = BandedMatrix{Float64}(undef, (n,n), (l,u))
  A[band(0)] .= 0
  A[band(1)] .= -1/(2*Δx)
  A[band(-1)] .= 1/(2*Δx)
  A[1,1:2] .= 0
  A[n,n] = -1/(2*Δx)
  A[n,n-1] = 1/(2*Δx)
  A = A.*v
  return A
end

function build_bands(npoints,L,v,D)
  Δx = L/(npoints-1)
  n = npoints
  VB = Vband(n,1,1,Δx,v)
  DB = Dband(n,1,1,Δx,D)
  return DB.+VB
end

RHS = FastChain(FastDense(2,9,tanh,bias = false),FastDense(9,1,bias = false))

function eqsys(du,u,θ,t,n,Φ,FB,FBc)
  c = @view u[1:n]
  dcdt = @view du[1:n]
  dqdt = @view du[n+1:2*n]
  dqdt .= vec(RHS(reshape(u,n,:)',θ))
  mul!(FBc,FB,c)
  @. dcdt = FBc - Φ*dqdt
  dqdt[1] = 0
  dcdt[1] = 0
end

#closure with preallocation
function fun(p,u0,n,L,v,D,Φ,save_times)
  FB = build_bands(n,L,v,D)
  FBc = zeros(n)
  ode_closure = ODEFunction((du,u,p,t) -> eqsys(du,u,p,t,n,Φ,FB,FBc))
  prob = ODEProblem(ode_closure,u0,tspan,p)
  solve(prob,KenCarp4(autodiff=false),saveat = save_times,sensealg = QuadratureAdjoint(autojacvec=ReverseDiffVJP(true)))
end

#callback & loss
cb = function (p,l)
  println(l)
  if l < 1e-3 #1e-3 original
      return true
  else
      return false
  end
end

function loss(p,u0,n,L,v,D,Φ,save_times,data)
  sol = fun(p,u0,n,L,v,D,Φ,save_times)
  diff_vec = vec(sol[n,:].-data)
  loss = sqrt(sum(abs2,diff_vec))
  return loss
end

#some toy paras (will probably never fit)

n = 20
L = 10
Epsilon = 0.36
Φ = zeros(n)
Φ .= (1-Epsilon)/Epsilon

c_in = 1
save_times = collect(0:0.2*pi:20)
data = save_times./20

u0 = zeros(2*n)
u0[1] = c_in

v = 2.8
D = 3.5e-7

tᵢ = save_times[1]
tₑ = save_times[end]
tspan = (tᵢ,tₑ)

#initialize & optimize
p = Float64.(initial_params(RHS)).*1e-3
optprob = OptimizationFunction((p,x) -> loss(p,u0,n,L,v,D,Φ,save_times,data), GalacticOptim.AutoZygote())
prob = GalacticOptim.OptimizationProblem(optprob, p)
res = GalacticOptim.solve(prob,ADAM(1e-2),cb = cb,maxiters = 100)

```

with

```julia
      Status `~/.julia/environments/v1.6/Project.toml`
  [c52e3926] Atom v0.12.31
  [aae01518] BandedMatrices v0.16.8
  [6e4b80f9] BenchmarkTools v0.7.0
  [a134a8b2] BlackBoxOptim v0.5.0
  [2b5f629d] DiffEqBase v6.60.1
  [459566f4] DiffEqCallbacks v2.16.1
  [aae7a2af] DiffEqFlux v1.36.1
  [41bf760c] DiffEqSensitivity v6.44.1
  [587475ba] Flux v0.12.1
  [28b8d3ca] GR v0.57.4
  [a75be94c] GalacticOptim v1.1.0
  [0f8b85d8] JSON3 v1.8.1
  [e5e0dc1b] Juno v0.8.4
  [7f56f5a3] LSODA v0.7.0
  [961ee093] ModelingToolkit v5.16.0
  [429524aa] Optim v1.3.0
  [1dea7af3] OrdinaryDiffEq v5.53.0
  [91a5bcdd] Plots v1.13.2
  [c46f51b8] ProfileView v0.6.9
  [37e2e3b7] ReverseDiff v1.8.0
  [684fba80] SparsityDetection v0.3.4
  [856f2bd8] StructTypes v1.7.2
  [c3572dad] Sundials v4.4.2
  [e88e6eb3] Zygote v0.6.10
  [9abbd945] Profile
  [2f01184e] SparseArrays

```

As far as I see it the issue was introduced with [fix of #76 in ReverseDiff.jl](https://github.com/JuliaDiff/ReverseDiff.jl/issues/76) in ReverseDiff. 1.7 as it does not throw the error if 1.6 is tagged.

Does it need a similar handling to ` DiffEqBase.get_tmp` for reverse diff tracked values? If yes, what would be a fix to facilitate both, e.g. pre-allocations that facilitate autodiff through the solver and fix this issue at the same time.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 6, 2021, 1:57pm UTC](https://discourse.julialang.org/t/neural-networks-combined-with-diffeq/58271/25 "2021-05-06T13:57:59Z")

</div>

ReverseDiff v1.7 has a few issues, so we actually don’t allow it right now.

---

<div class="post-metadata">

**Author:** ![triceert](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/triceert/32/21896_2.png) [@triceert](https://discourse.julialang.org/u/triceert)\
**Post date:** [May 6, 2021, 3:36pm UTC](https://discourse.julialang.org/t/neural-networks-combined-with-diffeq/58271/26 "2021-05-06T15:36:18Z")

</div>

So will DiffEqSensitivity compat be even reduced further to 1.6? Or did you actually mean you don’t support 1.8? Because with` ]add ReverseDiff @ 1.7` the error is still there.

What I tried was to `FBc = eltype(p).(FBc)` in `fun` but in the `eqsys` `typeof(FBc)` still return an `Array{Float64}` despite `p` being a ReverseDiff Array in the reverse mode call. Of course just preallocating an `AbstractArray` probably makes the problem go away - this will also kill speed however. So I’m little bit lost on how to fix this here.

Edit: Stacktrace:

```julia
ERROR: LoadError: ArgumentError: Converting an instance of ReverseDiff.TrackedReal{Float64, Float64, Nothing} to Float64 is not defined. Please use `ReverseDiff.value` instead.
Stacktrace:
  [1] convert(#unused#::Type{Float64}, t::ReverseDiff.TrackedReal{Float64, Float64, Nothing})
    @ ReverseDiff ~/.julia/packages/ReverseDiff/iHmB4/src/tracked.jl:261
  [2] setindex!(A::Vector{Float64}, x::ReverseDiff.TrackedReal{Float64, Float64, Nothing}, i1::Int64)
    @ Base ./array.jl:839
  [3] materialize!
    @ ~/.julia/packages/BandedMatrices/sSIkN/src/generic/matmul.jl:109 [inlined]
  [4] muladd!(α::ReverseDiff.TrackedReal{Float64, Float64, Nothing}, A::BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}, B::SubArray{ReverseDiff.TrackedReal{Float64, Float64, ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}}, 1, ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}, Tuple{UnitRange{Int64}}, true}, β::ReverseDiff.TrackedReal{Float64, Float64, Nothing}, C::Vector{Float64})
    @ ArrayLayouts ~/.julia/packages/ArrayLayouts/M5SFk/src/muladd.jl:68
  [5] copyto!
    @ ~/.julia/packages/ArrayLayouts/M5SFk/src/muladd.jl:75 [inlined]
  [6] copyto!
    @ ~/.julia/packages/ArrayLayouts/M5SFk/src/mul.jl:110 [inlined]
  [7] mul!(dest::Vector{Float64}, A::BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}, B::SubArray{ReverseDiff.TrackedReal{Float64, Float64, ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}}, 1, ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}, Tuple{UnitRange{Int64}}, true})
    @ ArrayLayouts ~/.julia/packages/ArrayLayouts/M5SFk/src/mul.jl:111
  [8] mul!
    @ ~/.julia/packages/ArrayLayouts/M5SFk/src/mul.jl:160 [inlined]
  [9] eqsys(du::Vector{ReverseDiff.TrackedReal{Float64, Float64, ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}}}, u::ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}, θ::ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}, t::ReverseDiff.TrackedReal{Float64, Float64, ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}}, n::Int64, Φ::Vector{Float64}, FB::BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}, FBc::Vector{Float64})
    @ Main ~/Algorithms/ML/CC_speedtest.jl:54
 [10] #7
    @ ~/Algorithms/ML/CC_speedtest.jl:64 [inlined]
 [11] ODEFunction
    @ ~/.julia/packages/SciMLBase/EFFG1/src/scimlfunctions.jl:334 [inlined]
 [12] (::DiffEqSensitivity.var"#89#100"{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}})(u::ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}, p::ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}, t::ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}})
    @ DiffEqSensitivity ~/.julia/packages/DiffEqSensitivity/lnBE2/src/adjoint_common.jl:140
 [13] ReverseDiff.GradientTape(f::Function, input::Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, cfg::ReverseDiff.GradientConfig{Tuple{ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}, ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}, ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}}})
    @ ReverseDiff ~/.julia/packages/ReverseDiff/iHmB4/src/api/tape.jl:207
 [14] ReverseDiff.GradientTape(f::Function, input::Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}})
    @ ReverseDiff ~/.julia/packages/ReverseDiff/iHmB4/src/api/tape.jl:204
 [15] adjointdiffcache(g::Function, sensealg::QuadratureAdjoint{0, true, Val{:central}, ReverseDiffVJP{true}}, discrete::Bool, sol::ODESolution{Float64, 2, Vector{Vector{Float64}}, Nothing, Nothing, Vector{Float64}, Vector{Vector{Vector{Float64}}}, ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, Vector{Float64}, ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, KenCarp4{0, false, DefaultLinSolve, NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, DataType}, OrdinaryDiffEq.InterpolationData{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Vector{Vector{Float64}}, Vector{Float64}, Vector{Vector{Vector{Float64}}}, OrdinaryDiffEq.KenCarp4Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.NLSolver{NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, true, Vector{Float64}, Float64, Nothing, Float64, OrdinaryDiffEq.NLNewtonCache{Vector{Float64}, Float64, Float64, Vector{Float64}, Matrix{Float64}, Matrix{Float64}, SciMLBase.UJacobianWrapper{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Float64, Vector{Float64}}, FiniteDiff.JacobianCache{Vector{Float64}, Vector{Float64}, Vector{Float64}, UnitRange{Int64}, Nothing, Val{:forward}(), Float64}, DefaultLinSolve}}, OrdinaryDiffEq.KenCarp4Tableau{Float64, Float64}, Nothing}}, DiffEqBase.DEStats}, dg::Nothing, f::ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}; quad::Bool, noiseterm::Bool)
    @ DiffEqSensitivity ~/.julia/packages/DiffEqSensitivity/lnBE2/src/adjoint_common.jl:138
 [16] DiffEqSensitivity.ODEQuadratureAdjointSensitivityFunction(g::Function, sensealg::QuadratureAdjoint{0, true, Val{:central}, ReverseDiffVJP{true}}, discrete::Bool, sol::ODESolution{Float64, 2, Vector{Vector{Float64}}, Nothing, Nothing, Vector{Float64}, Vector{Vector{Vector{Float64}}}, ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, Vector{Float64}, ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, KenCarp4{0, false, DefaultLinSolve, NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, DataType}, OrdinaryDiffEq.InterpolationData{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Vector{Vector{Float64}}, Vector{Float64}, Vector{Vector{Vector{Float64}}}, OrdinaryDiffEq.KenCarp4Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.NLSolver{NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, true, Vector{Float64}, Float64, Nothing, Float64, OrdinaryDiffEq.NLNewtonCache{Vector{Float64}, Float64, Float64, Vector{Float64}, Matrix{Float64}, Matrix{Float64}, SciMLBase.UJacobianWrapper{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Float64, Vector{Float64}}, FiniteDiff.JacobianCache{Vector{Float64}, Vector{Float64}, Vector{Float64}, UnitRange{Int64}, Nothing, Val{:forward}(), Float64}, DefaultLinSolve}}, OrdinaryDiffEq.KenCarp4Tableau{Float64, Float64}, Nothing}}, DiffEqBase.DEStats}, dg::Nothing)
    @ DiffEqSensitivity ~/.julia/packages/DiffEqSensitivity/lnBE2/src/quadrature_adjoint.jl:12
 [17] ODEAdjointProblem(sol::ODESolution{Float64, 2, Vector{Vector{Float64}}, Nothing, Nothing, Vector{Float64}, Vector{Vector{Vector{Float64}}}, ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, Vector{Float64}, ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, KenCarp4{0, false, DefaultLinSolve, NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, DataType}, OrdinaryDiffEq.InterpolationData{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Vector{Vector{Float64}}, Vector{Float64}, Vector{Vector{Vector{Float64}}}, OrdinaryDiffEq.KenCarp4Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.NLSolver{NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, true, Vector{Float64}, Float64, Nothing, Float64, OrdinaryDiffEq.NLNewtonCache{Vector{Float64}, Float64, Float64, Vector{Float64}, Matrix{Float64}, Matrix{Float64}, SciMLBase.UJacobianWrapper{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Float64, Vector{Float64}}, FiniteDiff.JacobianCache{Vector{Float64}, Vector{Float64}, Vector{Float64}, UnitRange{Int64}, Nothing, Val{:forward}(), Float64}, DefaultLinSolve}}, OrdinaryDiffEq.KenCarp4Tableau{Float64, Float64}, Nothing}}, DiffEqBase.DEStats}, sensealg::QuadratureAdjoint{0, true, Val{:central}, ReverseDiffVJP{true}}, g::Function, t::Vector{Float64}, dg::Nothing, callback::Nothing)
    @ DiffEqSensitivity ~/.julia/packages/DiffEqSensitivity/lnBE2/src/quadrature_adjoint.jl:60
 [18] _adjoint_sensitivities(sol::ODESolution{Float64, 2, Vector{Vector{Float64}}, Nothing, Nothing, Vector{Float64}, Vector{Vector{Vector{Float64}}}, ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, Vector{Float64}, ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, KenCarp4{0, false, DefaultLinSolve, NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, DataType}, OrdinaryDiffEq.InterpolationData{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Vector{Vector{Float64}}, Vector{Float64}, Vector{Vector{Vector{Float64}}}, OrdinaryDiffEq.KenCarp4Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.NLSolver{NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, true, Vector{Float64}, Float64, Nothing, Float64, OrdinaryDiffEq.NLNewtonCache{Vector{Float64}, Float64, Float64, Vector{Float64}, Matrix{Float64}, Matrix{Float64}, SciMLBase.UJacobianWrapper{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Float64, Vector{Float64}}, FiniteDiff.JacobianCache{Vector{Float64}, Vector{Float64}, Vector{Float64}, UnitRange{Int64}, Nothing, Val{:forward}(), Float64}, DefaultLinSolve}}, OrdinaryDiffEq.KenCarp4Tableau{Float64, Float64}, Nothing}}, DiffEqBase.DEStats}, sensealg::QuadratureAdjoint{0, true, Val{:central}, ReverseDiffVJP{true}}, alg::KenCarp4{0, false, DefaultLinSolve, NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, DataType}, g::DiffEqSensitivity.var"#df#168"{Matrix{Float64}, Vector{Float64}, Colon}, t::Vector{Float64}, dg::Nothing; abstol::Float64, reltol::Float64, callback::Nothing, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ DiffEqSensitivity ~/.julia/packages/DiffEqSensitivity/lnBE2/src/quadrature_adjoint.jl:192
 [19] adjoint_sensitivities(::ODESolution{Float64, 2, Vector{Vector{Float64}}, Nothing, Nothing, Vector{Float64}, Vector{Vector{Vector{Float64}}}, ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, Vector{Float64}, ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, KenCarp4{0, false, DefaultLinSolve, NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, DataType}, OrdinaryDiffEq.InterpolationData{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Vector{Vector{Float64}}, Vector{Float64}, Vector{Vector{Vector{Float64}}}, OrdinaryDiffEq.KenCarp4Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.NLSolver{NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, true, Vector{Float64}, Float64, Nothing, Float64, OrdinaryDiffEq.NLNewtonCache{Vector{Float64}, Float64, Float64, Vector{Float64}, Matrix{Float64}, Matrix{Float64}, SciMLBase.UJacobianWrapper{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Float64, Vector{Float64}}, FiniteDiff.JacobianCache{Vector{Float64}, Vector{Float64}, Vector{Float64}, UnitRange{Int64}, Nothing, Val{:forward}(), Float64}, DefaultLinSolve}}, OrdinaryDiffEq.KenCarp4Tableau{Float64, Float64}, Nothing}}, DiffEqBase.DEStats}, ::KenCarp4{0, false, DefaultLinSolve, NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, DataType}, ::Vararg{Any, N} where N; sensealg::QuadratureAdjoint{0, true, Val{:central}, ReverseDiffVJP{true}}, kwargs::Base.Iterators.Pairs{Symbol, Nothing, Tuple{Symbol}, NamedTuple{(:callback,), Tuple{Nothing}}})
    @ DiffEqSensitivity ~/.julia/packages/DiffEqSensitivity/lnBE2/src/sensitivity_interface.jl:6
 [20] (::DiffEqSensitivity.var"#adjoint_sensitivity_backpass#167"{Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, KenCarp4{0, false, DefaultLinSolve, NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, DataType}, QuadratureAdjoint{0, true, Val{:central}, ReverseDiffVJP{true}}, Vector{Float64}, Vector{Float64}, Tuple{}, Colon, NamedTuple{(), Tuple{}}})(Δ::Matrix{Float64})
    @ DiffEqSensitivity ~/.julia/packages/DiffEqSensitivity/lnBE2/src/concrete_solve.jl:179
 [21] #101#back
    @ ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:65 [inlined]
 [22] #180
    @ ~/.julia/packages/Zygote/6HN9x/src/lib/lib.jl:194 [inlined]
 [23] (::Zygote.var"#1689#back#182"{Zygote.var"#180#181"{Tuple{NTuple{6, Nothing}, Tuple{Nothing}}, DiffEqBase.var"#101#back#74"{DiffEqSensitivity.var"#adjoint_sensitivity_backpass#167"{Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, KenCarp4{0, false, DefaultLinSolve, NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, DataType}, QuadratureAdjoint{0, true, Val{:central}, ReverseDiffVJP{true}}, Vector{Float64}, Vector{Float64}, Tuple{}, Colon, NamedTuple{(), Tuple{}}}}}})(Δ::Matrix{Float64})
    @ Zygote ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
 [24] Pullback
    @ ~/.julia/packages/DiffEqBase/GmecW/src/solve.jl:70 [inlined]
 [25] (::typeof(∂(#solve#57)))(Δ::Matrix{Float64})
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface2.jl:0
 [26] (::Zygote.var"#180#181"{Tuple{NTuple{6, Nothing}, Tuple{Nothing}}, typeof(∂(#solve#57))})(Δ::Matrix{Float64})
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/lib/lib.jl:194
 [27] (::Zygote.var"#1689#back#182"{Zygote.var"#180#181"{Tuple{NTuple{6, Nothing}, Tuple{Nothing}}, typeof(∂(#solve#57))}})(Δ::Matrix{Float64})
    @ Zygote ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
 [28] Pullback
    @ ~/.julia/packages/DiffEqBase/GmecW/src/solve.jl:68 [inlined]
 [29] (::typeof(∂(solve##kw)))(Δ::Matrix{Float64})
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface2.jl:0
 [30] Pullback
    @ ~/Algorithms/ML/CC_speedtest.jl:66 [inlined]
 [31] (::typeof(∂(fun)))(Δ::Matrix{Float64})
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface2.jl:0
 [32] Pullback
    @ ~/Algorithms/ML/CC_speedtest.jl:81 [inlined]
 [33] (::typeof(∂(loss)))(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface2.jl:0
 [34] Pullback
    @ ~/Algorithms/ML/CC_speedtest.jl:112 [inlined]
 [35] (::typeof(∂(#11)))(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface2.jl:0
 [36] #180
    @ ~/.julia/packages/Zygote/6HN9x/src/lib/lib.jl:194 [inlined]
 [37] #1689#back
    @ ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59 [inlined]
 [38] Pullback
    @ ~/.julia/packages/SciMLBase/EFFG1/src/problems/basic_problems.jl:107 [inlined]
 [39] #180
    @ ~/.julia/packages/Zygote/6HN9x/src/lib/lib.jl:194 [inlined]
 [40] (::Zygote.var"#1689#back#182"{Zygote.var"#180#181"{Tuple{Tuple{Nothing, Nothing}, Int64}, typeof(∂(OptimizationFunction{true, GalacticOptim.AutoZygote, var"#11#12", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}(var"#11#12"(), GalacticOptim.AutoZygote(), nothing, nothing, nothing, nothing, nothing, nothing)))}})(Δ::Float64)
    @ Zygote ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
 [41] Pullback
    @ ~/.julia/packages/GalacticOptim/JnLwV/src/solve.jl:94 [inlined]
 [42] (::typeof(∂(λ)))(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface2.jl:0
 [43] (::Zygote.var"#69#70"{Params, typeof(∂(λ)), Zygote.Context})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface.jl:252
 [44] gradient(f::Function, args::Params)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface.jl:59
 [45] __solve(prob::OptimizationProblem{true, OptimizationFunction{true, GalacticOptim.AutoZygote, var"#11#12", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, SciMLBase.NullParameters, Nothing, Nothing, Nothing, Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, opt::ADAM, data::Base.Iterators.Cycle{Tuple{GalacticOptim.NullData}}; maxiters::Int64, cb::Function, progress::Bool, save_best::Bool, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ GalacticOptim ~/.julia/packages/GalacticOptim/JnLwV/src/solve.jl:93
 [46] #solve#468
    @ ~/.julia/packages/SciMLBase/EFFG1/src/solve.jl:3 [inlined]
 [47] top-level scope
    @ ~/Algorithms/ML/CC_speedtest.jl:114
 [48] eval
    @ ./boot.jl:360 [inlined]
 [49] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
    @ Base ./loading.jl:1094
in expression starting at ~/Algorithms/ML/CC_speedtest.jl:114

```

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 7, 2021, 8:18pm UTC](https://discourse.julialang.org/t/neural-networks-combined-with-diffeq/58271/27 "2021-05-07T20:18:07Z")

</div>

Open an issue.

[Previous page](https://discourse.julialang.org/t/neural-networks-combined-with-diffeq/58271.md?page=1)