Neural Networks combined with Diffeq

I can call the loss function and manually change values in the vector of network parameters and I get different losses, but the gradient fails.

This all means nothing without knowing the code that is causing it.

I know. I’m trying to think how to construct a simpler example.

I started with a diffeq based three-dof which has parameters that setup the details, similar to how the trebuche example problem might work. So it started out complicated. Some of those parameters I want to optimize, but I really want a neural network to output those parameters and set them prior to the solve through time.

I’m still looking for a good example that hilites this workflow, which I would think would be very common, but I don’t really find anything, but there is a lot of material out there so I probably missed it.

I think it has to do with preallocations for e.g. mul! where the left hand side is of the wrong type.
I tried to strip down an MWE which leads to the same error:

using BandedMatrices
using DiffEqFlux
using OrdinaryDiffEq
using DiffEqSensitivity
using Flux
using GalacticOptim
using LinearAlgebra
using Zygote

#band stuff

function Dband(n,l,u,Δx,D)
  A = BandedMatrix{Float64}(undef, (n,n), (l,u))
  A[band(0)] .= -2*1/Δx^2
  A[band(1)] .= 1/Δx^2
  A[band(-1)] .= 1/Δx^2
  A[1,1:2] .= 0
  A[n,n] = -1/Δx^2
  A[n,n-1] = 1/Δx^2
  A = A.*D
  return A
end

function Vband(n,l,u,Δx,v)
  A = BandedMatrix{Float64}(undef, (n,n), (l,u))
  A[band(0)] .= 0
  A[band(1)] .= -1/(2*Δx)
  A[band(-1)] .= 1/(2*Δx)
  A[1,1:2] .= 0
  A[n,n] = -1/(2*Δx)
  A[n,n-1] = 1/(2*Δx)
  A = A.*v
  return A
end


function build_bands(npoints,L,v,D)
  Δx = L/(npoints-1)
  n = npoints
  VB = Vband(n,1,1,Δx,v)
  DB = Dband(n,1,1,Δx,D)
  return DB.+VB
end

RHS =  FastChain(FastDense(2,9,tanh,bias = false),FastDense(9,1,bias = false))

function eqsys(du,u,θ,t,n,Φ,FB,FBc)
  c =  @view u[1:n]
  dcdt =  @view du[1:n]
  dqdt =  @view du[n+1:2*n]
  dqdt .= vec(RHS(reshape(u,n,:)',θ))
  mul!(FBc,FB,c)
  @. dcdt = FBc - Φ*dqdt
  dqdt[1] = 0
  dcdt[1] = 0
end

#closure with preallocation
function fun(p,u0,n,L,v,D,Φ,save_times)
  FB = build_bands(n,L,v,D)
  FBc = zeros(n)
  ode_closure = ODEFunction((du,u,p,t) -> eqsys(du,u,p,t,n,Φ,FB,FBc))
  prob = ODEProblem(ode_closure,u0,tspan,p)
  solve(prob,KenCarp4(autodiff=false),saveat = save_times,sensealg = QuadratureAdjoint(autojacvec=ReverseDiffVJP(true)))
end


#callback & loss
cb = function (p,l)
  println(l)
  if l < 1e-3 #1e-3 original
      return true
  else
      return false
  end
end

function loss(p,u0,n,L,v,D,Φ,save_times,data)
  sol = fun(p,u0,n,L,v,D,Φ,save_times)
  diff_vec = vec(sol[n,:].-data)
  loss = sqrt(sum(abs2,diff_vec))
  return loss
end

#some toy paras (will probably never fit)

n = 20
L = 10
Epsilon = 0.36
Φ = zeros(n)
Φ .= (1-Epsilon)/Epsilon

c_in = 1
save_times  = collect(0:0.2*pi:20)
data = save_times./20


u0 = zeros(2*n)
u0[1] = c_in

v = 2.8
D = 3.5e-7

tᵢ = save_times[1]
tₑ = save_times[end]
tspan = (tᵢ,tₑ)

#initialize & optimize
p = Float64.(initial_params(RHS)).*1e-3
optprob = OptimizationFunction((p,x) -> loss(p,u0,n,L,v,D,Φ,save_times,data), GalacticOptim.AutoZygote())
prob = GalacticOptim.OptimizationProblem(optprob, p)
res = GalacticOptim.solve(prob,ADAM(1e-2),cb = cb,maxiters = 100)

with

      Status `~/.julia/environments/v1.6/Project.toml`
  [c52e3926] Atom v0.12.31
  [aae01518] BandedMatrices v0.16.8
  [6e4b80f9] BenchmarkTools v0.7.0
  [a134a8b2] BlackBoxOptim v0.5.0
  [2b5f629d] DiffEqBase v6.60.1
  [459566f4] DiffEqCallbacks v2.16.1
  [aae7a2af] DiffEqFlux v1.36.1
  [41bf760c] DiffEqSensitivity v6.44.1
  [587475ba] Flux v0.12.1
  [28b8d3ca] GR v0.57.4
  [a75be94c] GalacticOptim v1.1.0
  [0f8b85d8] JSON3 v1.8.1
  [e5e0dc1b] Juno v0.8.4
  [7f56f5a3] LSODA v0.7.0
  [961ee093] ModelingToolkit v5.16.0
  [429524aa] Optim v1.3.0
  [1dea7af3] OrdinaryDiffEq v5.53.0
  [91a5bcdd] Plots v1.13.2
  [c46f51b8] ProfileView v0.6.9
  [37e2e3b7] ReverseDiff v1.8.0
  [684fba80] SparsityDetection v0.3.4
  [856f2bd8] StructTypes v1.7.2
  [c3572dad] Sundials v4.4.2
  [e88e6eb3] Zygote v0.6.10
  [9abbd945] Profile
  [2f01184e] SparseArrays

As far as I see it the issue was introduced with fix of #76 in ReverseDiff.jl in ReverseDiff. 1.7 as it does not throw the error if 1.6 is tagged.

Does it need a similar handling to DiffEqBase.get_tmp for reverse diff tracked values? If yes, what would be a fix to facilitate both, e.g. pre-allocations that facilitate autodiff through the solver and fix this issue at the same time.

ReverseDiff v1.7 has a few issues, so we actually don’t allow it right now.

So will DiffEqSensitivity compat be even reduced further to 1.6? Or did you actually mean you don’t support 1.8? Because with ]add ReverseDiff @ 1.7 the error is still there.

What I tried was to FBc = eltype(p).(FBc) in fun but in the eqsys typeof(FBc) still return an Array{Float64} despite p being a ReverseDiff Array in the reverse mode call. Of course just preallocating an AbstractArray probably makes the problem go away - this will also kill speed however. So I’m little bit lost on how to fix this here.

Edit: Stacktrace:

ERROR: LoadError: ArgumentError: Converting an instance of ReverseDiff.TrackedReal{Float64, Float64, Nothing} to Float64 is not defined. Please use `ReverseDiff.value` instead.
Stacktrace:
  [1] convert(#unused#::Type{Float64}, t::ReverseDiff.TrackedReal{Float64, Float64, Nothing})
    @ ReverseDiff ~/.julia/packages/ReverseDiff/iHmB4/src/tracked.jl:261
  [2] setindex!(A::Vector{Float64}, x::ReverseDiff.TrackedReal{Float64, Float64, Nothing}, i1::Int64)
    @ Base ./array.jl:839
  [3] materialize!
    @ ~/.julia/packages/BandedMatrices/sSIkN/src/generic/matmul.jl:109 [inlined]
  [4] muladd!(α::ReverseDiff.TrackedReal{Float64, Float64, Nothing}, A::BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}, B::SubArray{ReverseDiff.TrackedReal{Float64, Float64, ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}}, 1, ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}, Tuple{UnitRange{Int64}}, true}, β::ReverseDiff.TrackedReal{Float64, Float64, Nothing}, C::Vector{Float64})
    @ ArrayLayouts ~/.julia/packages/ArrayLayouts/M5SFk/src/muladd.jl:68
  [5] copyto!
    @ ~/.julia/packages/ArrayLayouts/M5SFk/src/muladd.jl:75 [inlined]
  [6] copyto!
    @ ~/.julia/packages/ArrayLayouts/M5SFk/src/mul.jl:110 [inlined]
  [7] mul!(dest::Vector{Float64}, A::BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}, B::SubArray{ReverseDiff.TrackedReal{Float64, Float64, ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}}, 1, ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}, Tuple{UnitRange{Int64}}, true})
    @ ArrayLayouts ~/.julia/packages/ArrayLayouts/M5SFk/src/mul.jl:111
  [8] mul!
    @ ~/.julia/packages/ArrayLayouts/M5SFk/src/mul.jl:160 [inlined]
  [9] eqsys(du::Vector{ReverseDiff.TrackedReal{Float64, Float64, ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}}}, u::ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}, θ::ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}, t::ReverseDiff.TrackedReal{Float64, Float64, ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}}, n::Int64, Φ::Vector{Float64}, FB::BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}, FBc::Vector{Float64})
    @ Main ~/Algorithms/ML/CC_speedtest.jl:54
 [10] #7
    @ ~/Algorithms/ML/CC_speedtest.jl:64 [inlined]
 [11] ODEFunction
    @ ~/.julia/packages/SciMLBase/EFFG1/src/scimlfunctions.jl:334 [inlined]
 [12] (::DiffEqSensitivity.var"#89#100"{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}})(u::ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}, p::ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}, t::ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}})
    @ DiffEqSensitivity ~/.julia/packages/DiffEqSensitivity/lnBE2/src/adjoint_common.jl:140
 [13] ReverseDiff.GradientTape(f::Function, input::Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, cfg::ReverseDiff.GradientConfig{Tuple{ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}, ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}, ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}}})
    @ ReverseDiff ~/.julia/packages/ReverseDiff/iHmB4/src/api/tape.jl:207
 [14] ReverseDiff.GradientTape(f::Function, input::Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}})
    @ ReverseDiff ~/.julia/packages/ReverseDiff/iHmB4/src/api/tape.jl:204
 [15] adjointdiffcache(g::Function, sensealg::QuadratureAdjoint{0, true, Val{:central}, ReverseDiffVJP{true}}, discrete::Bool, sol::ODESolution{Float64, 2, Vector{Vector{Float64}}, Nothing, Nothing, Vector{Float64}, Vector{Vector{Vector{Float64}}}, ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, Vector{Float64}, ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, KenCarp4{0, false, DefaultLinSolve, NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, DataType}, OrdinaryDiffEq.InterpolationData{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Vector{Vector{Float64}}, Vector{Float64}, Vector{Vector{Vector{Float64}}}, OrdinaryDiffEq.KenCarp4Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.NLSolver{NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, true, Vector{Float64}, Float64, Nothing, Float64, OrdinaryDiffEq.NLNewtonCache{Vector{Float64}, Float64, Float64, Vector{Float64}, Matrix{Float64}, Matrix{Float64}, SciMLBase.UJacobianWrapper{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Float64, Vector{Float64}}, FiniteDiff.JacobianCache{Vector{Float64}, Vector{Float64}, Vector{Float64}, UnitRange{Int64}, Nothing, Val{:forward}(), Float64}, DefaultLinSolve}}, OrdinaryDiffEq.KenCarp4Tableau{Float64, Float64}, Nothing}}, DiffEqBase.DEStats}, dg::Nothing, f::ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}; quad::Bool, noiseterm::Bool)
    @ DiffEqSensitivity ~/.julia/packages/DiffEqSensitivity/lnBE2/src/adjoint_common.jl:138
 [16] DiffEqSensitivity.ODEQuadratureAdjointSensitivityFunction(g::Function, sensealg::QuadratureAdjoint{0, true, Val{:central}, ReverseDiffVJP{true}}, discrete::Bool, sol::ODESolution{Float64, 2, Vector{Vector{Float64}}, Nothing, Nothing, Vector{Float64}, Vector{Vector{Vector{Float64}}}, ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, Vector{Float64}, ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, KenCarp4{0, false, DefaultLinSolve, NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, DataType}, OrdinaryDiffEq.InterpolationData{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Vector{Vector{Float64}}, Vector{Float64}, Vector{Vector{Vector{Float64}}}, OrdinaryDiffEq.KenCarp4Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.NLSolver{NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, true, Vector{Float64}, Float64, Nothing, Float64, OrdinaryDiffEq.NLNewtonCache{Vector{Float64}, Float64, Float64, Vector{Float64}, Matrix{Float64}, Matrix{Float64}, SciMLBase.UJacobianWrapper{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Float64, Vector{Float64}}, FiniteDiff.JacobianCache{Vector{Float64}, Vector{Float64}, Vector{Float64}, UnitRange{Int64}, Nothing, Val{:forward}(), Float64}, DefaultLinSolve}}, OrdinaryDiffEq.KenCarp4Tableau{Float64, Float64}, Nothing}}, DiffEqBase.DEStats}, dg::Nothing)
    @ DiffEqSensitivity ~/.julia/packages/DiffEqSensitivity/lnBE2/src/quadrature_adjoint.jl:12
 [17] ODEAdjointProblem(sol::ODESolution{Float64, 2, Vector{Vector{Float64}}, Nothing, Nothing, Vector{Float64}, Vector{Vector{Vector{Float64}}}, ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, Vector{Float64}, ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, KenCarp4{0, false, DefaultLinSolve, NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, DataType}, OrdinaryDiffEq.InterpolationData{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Vector{Vector{Float64}}, Vector{Float64}, Vector{Vector{Vector{Float64}}}, OrdinaryDiffEq.KenCarp4Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.NLSolver{NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, true, Vector{Float64}, Float64, Nothing, Float64, OrdinaryDiffEq.NLNewtonCache{Vector{Float64}, Float64, Float64, Vector{Float64}, Matrix{Float64}, Matrix{Float64}, SciMLBase.UJacobianWrapper{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Float64, Vector{Float64}}, FiniteDiff.JacobianCache{Vector{Float64}, Vector{Float64}, Vector{Float64}, UnitRange{Int64}, Nothing, Val{:forward}(), Float64}, DefaultLinSolve}}, OrdinaryDiffEq.KenCarp4Tableau{Float64, Float64}, Nothing}}, DiffEqBase.DEStats}, sensealg::QuadratureAdjoint{0, true, Val{:central}, ReverseDiffVJP{true}}, g::Function, t::Vector{Float64}, dg::Nothing, callback::Nothing)
    @ DiffEqSensitivity ~/.julia/packages/DiffEqSensitivity/lnBE2/src/quadrature_adjoint.jl:60
 [18] _adjoint_sensitivities(sol::ODESolution{Float64, 2, Vector{Vector{Float64}}, Nothing, Nothing, Vector{Float64}, Vector{Vector{Vector{Float64}}}, ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, Vector{Float64}, ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, KenCarp4{0, false, DefaultLinSolve, NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, DataType}, OrdinaryDiffEq.InterpolationData{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Vector{Vector{Float64}}, Vector{Float64}, Vector{Vector{Vector{Float64}}}, OrdinaryDiffEq.KenCarp4Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.NLSolver{NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, true, Vector{Float64}, Float64, Nothing, Float64, OrdinaryDiffEq.NLNewtonCache{Vector{Float64}, Float64, Float64, Vector{Float64}, Matrix{Float64}, Matrix{Float64}, SciMLBase.UJacobianWrapper{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Float64, Vector{Float64}}, FiniteDiff.JacobianCache{Vector{Float64}, Vector{Float64}, Vector{Float64}, UnitRange{Int64}, Nothing, Val{:forward}(), Float64}, DefaultLinSolve}}, OrdinaryDiffEq.KenCarp4Tableau{Float64, Float64}, Nothing}}, DiffEqBase.DEStats}, sensealg::QuadratureAdjoint{0, true, Val{:central}, ReverseDiffVJP{true}}, alg::KenCarp4{0, false, DefaultLinSolve, NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, DataType}, g::DiffEqSensitivity.var"#df#168"{Matrix{Float64}, Vector{Float64}, Colon}, t::Vector{Float64}, dg::Nothing; abstol::Float64, reltol::Float64, callback::Nothing, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ DiffEqSensitivity ~/.julia/packages/DiffEqSensitivity/lnBE2/src/quadrature_adjoint.jl:192
 [19] adjoint_sensitivities(::ODESolution{Float64, 2, Vector{Vector{Float64}}, Nothing, Nothing, Vector{Float64}, Vector{Vector{Vector{Float64}}}, ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, Vector{Float64}, ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, KenCarp4{0, false, DefaultLinSolve, NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, DataType}, OrdinaryDiffEq.InterpolationData{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Vector{Vector{Float64}}, Vector{Float64}, Vector{Vector{Vector{Float64}}}, OrdinaryDiffEq.KenCarp4Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, OrdinaryDiffEq.NLSolver{NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, true, Vector{Float64}, Float64, Nothing, Float64, OrdinaryDiffEq.NLNewtonCache{Vector{Float64}, Float64, Float64, Vector{Float64}, Matrix{Float64}, Matrix{Float64}, SciMLBase.UJacobianWrapper{ODEFunction{true, var"#7#8"{Int64, Vector{Float64}, Vector{Float64}, BandedMatrix{Float64, Matrix{Float64}, Base.OneTo{Int64}}}, UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing}, Float64, Vector{Float64}}, FiniteDiff.JacobianCache{Vector{Float64}, Vector{Float64}, Vector{Float64}, UnitRange{Int64}, Nothing, Val{:forward}(), Float64}, DefaultLinSolve}}, OrdinaryDiffEq.KenCarp4Tableau{Float64, Float64}, Nothing}}, DiffEqBase.DEStats}, ::KenCarp4{0, false, DefaultLinSolve, NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, DataType}, ::Vararg{Any, N} where N; sensealg::QuadratureAdjoint{0, true, Val{:central}, ReverseDiffVJP{true}}, kwargs::Base.Iterators.Pairs{Symbol, Nothing, Tuple{Symbol}, NamedTuple{(:callback,), Tuple{Nothing}}})
    @ DiffEqSensitivity ~/.julia/packages/DiffEqSensitivity/lnBE2/src/sensitivity_interface.jl:6
 [20] (::DiffEqSensitivity.var"#adjoint_sensitivity_backpass#167"{Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, KenCarp4{0, false, DefaultLinSolve, NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, DataType}, QuadratureAdjoint{0, true, Val{:central}, ReverseDiffVJP{true}}, Vector{Float64}, Vector{Float64}, Tuple{}, Colon, NamedTuple{(), Tuple{}}})(Δ::Matrix{Float64})
    @ DiffEqSensitivity ~/.julia/packages/DiffEqSensitivity/lnBE2/src/concrete_solve.jl:179
 [21] #101#back
    @ ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:65 [inlined]
 [22] #180
    @ ~/.julia/packages/Zygote/6HN9x/src/lib/lib.jl:194 [inlined]
 [23] (::Zygote.var"#1689#back#182"{Zygote.var"#180#181"{Tuple{NTuple{6, Nothing}, Tuple{Nothing}}, DiffEqBase.var"#101#back#74"{DiffEqSensitivity.var"#adjoint_sensitivity_backpass#167"{Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, KenCarp4{0, false, DefaultLinSolve, NLNewton{Rational{Int64}, Rational{Int64}, Rational{Int64}}, DataType}, QuadratureAdjoint{0, true, Val{:central}, ReverseDiffVJP{true}}, Vector{Float64}, Vector{Float64}, Tuple{}, Colon, NamedTuple{(), Tuple{}}}}}})(Δ::Matrix{Float64})
    @ Zygote ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
 [24] Pullback
    @ ~/.julia/packages/DiffEqBase/GmecW/src/solve.jl:70 [inlined]
 [25] (::typeof(∂(#solve#57)))(Δ::Matrix{Float64})
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface2.jl:0
 [26] (::Zygote.var"#180#181"{Tuple{NTuple{6, Nothing}, Tuple{Nothing}}, typeof(∂(#solve#57))})(Δ::Matrix{Float64})
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/lib/lib.jl:194
 [27] (::Zygote.var"#1689#back#182"{Zygote.var"#180#181"{Tuple{NTuple{6, Nothing}, Tuple{Nothing}}, typeof(∂(#solve#57))}})(Δ::Matrix{Float64})
    @ Zygote ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
 [28] Pullback
    @ ~/.julia/packages/DiffEqBase/GmecW/src/solve.jl:68 [inlined]
 [29] (::typeof(∂(solve##kw)))(Δ::Matrix{Float64})
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface2.jl:0
 [30] Pullback
    @ ~/Algorithms/ML/CC_speedtest.jl:66 [inlined]
 [31] (::typeof(∂(fun)))(Δ::Matrix{Float64})
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface2.jl:0
 [32] Pullback
    @ ~/Algorithms/ML/CC_speedtest.jl:81 [inlined]
 [33] (::typeof(∂(loss)))(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface2.jl:0
 [34] Pullback
    @ ~/Algorithms/ML/CC_speedtest.jl:112 [inlined]
 [35] (::typeof(∂(#11)))(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface2.jl:0
 [36] #180
    @ ~/.julia/packages/Zygote/6HN9x/src/lib/lib.jl:194 [inlined]
 [37] #1689#back
    @ ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59 [inlined]
 [38] Pullback
    @ ~/.julia/packages/SciMLBase/EFFG1/src/problems/basic_problems.jl:107 [inlined]
 [39] #180
    @ ~/.julia/packages/Zygote/6HN9x/src/lib/lib.jl:194 [inlined]
 [40] (::Zygote.var"#1689#back#182"{Zygote.var"#180#181"{Tuple{Tuple{Nothing, Nothing}, Int64}, typeof(∂(OptimizationFunction{true, GalacticOptim.AutoZygote, var"#11#12", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}(var"#11#12"(), GalacticOptim.AutoZygote(), nothing, nothing, nothing, nothing, nothing, nothing)))}})(Δ::Float64)
    @ Zygote ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
 [41] Pullback
    @ ~/.julia/packages/GalacticOptim/JnLwV/src/solve.jl:94 [inlined]
 [42] (::typeof(∂(λ)))(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface2.jl:0
 [43] (::Zygote.var"#69#70"{Params, typeof(∂(λ)), Zygote.Context})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface.jl:252
 [44] gradient(f::Function, args::Params)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface.jl:59
 [45] __solve(prob::OptimizationProblem{true, OptimizationFunction{true, GalacticOptim.AutoZygote, var"#11#12", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, SciMLBase.NullParameters, Nothing, Nothing, Nothing, Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, opt::ADAM, data::Base.Iterators.Cycle{Tuple{GalacticOptim.NullData}}; maxiters::Int64, cb::Function, progress::Bool, save_best::Bool, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ GalacticOptim ~/.julia/packages/GalacticOptim/JnLwV/src/solve.jl:93
 [46] #solve#468
    @ ~/.julia/packages/SciMLBase/EFFG1/src/solve.jl:3 [inlined]
 [47] top-level scope
    @ ~/Algorithms/ML/CC_speedtest.jl:114
 [48] eval
    @ ./boot.jl:360 [inlined]
 [49] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
    @ Base ./loading.jl:1094
in expression starting at ~/Algorithms/ML/CC_speedtest.jl:114

Open an issue.