sciML_train func running but not optimizing

New to Julia
, , ,
1

Hi all,

My executions of DiffEqFlux.sciml_train are not optimizing but yet are iterating for the full amount without error. By not optimizing, I mean that for whatever random initial parameters I feed for a fitting example, the loss function does not change at each iteration and the final minimizers are the initial parameters. This remains true despite changing the learning rate for Adam, the number of iterations, the solver entirely (I also tried BFGS and SOSRI) or sensealg=ForwardDiffSensitivity().

I am using the standard loss(abs2, predicted - observed) and based on the initializations I randomize with I see significant variations in the loss which makes me think for some initial conditions some improvement should be possible. Previously, I have optimized identical systems with higher dimensions and terrible data with matlab’s fmincon().

All this leads me to believe I am making some novice error which is likely given I am new to julia and DiffEqFlux. Any help would be much appreciated. Note, I did just update everything, restarted atom and found the same problem.

First, I generate the model problem,

##
using Plots, Random
using DifferentialEquations, Flux, Optim, DiffEqFlux, DiffEqSensitivity

## Define GLV and generate synthetic data from defined parameters

tsteps = 24
n=3

function gLV!(dx,x,p,t)

    # Weird ReverseDiff (AD) bug
    if (typeof(p) != Array{Float64,1}) #&& (typeof(p) != SciMLBase.NullParameters)
        p=p.value
    end

    # unpack params
    n = length(x)
    mu = p[1:n]
    A = Array{Float64,2}(undef,3,3)
    for i=1:n; A[:,i] = p[n*i + 1: n*(i+1)]; end

    dx .= x.*(mu + A*x)

    if any(x .> 100.0) # no cc catch
        dx .= 0
    end
end

condition(u,t,integrator) = any(u .< 1e-8) #|| any(u .> 10)
function affect!(integrator)
    integrator.u[integrator.u .< 1e-8] .= 0
end
cb = ContinuousCallback(condition,affect!)

mu_true = [0.4; 0.8; 0.2].*0.1
A_true = [-1.5 0.5 -0.2; -0.7 -1.3 0.1; 0.8 -0.2 -0.9]
p_true = [mu_true;  A_true[:]]

x0 = [0.1; 0.1; 0.1].*0.1
tspan = (0.0, 144.0)

prob = ODEProblem(gLV!, x0, tspan)
sol = DifferentialEquations.solve(prob, Tsit5(), p=p_true, saveat=tsteps, callback=cb)
syndata = Array(sol) + 0.002*randn(size(Array(sol)))
plot(sol, alpha=0.3)
Plots.scatter!(sol.t, syndata')
title!("Definite")

After making the model problem and generating some noisy data with the true parameters, I then initialize random parameters, define a loss function and call sciml_train.

## Random Initialize

mu_init = 0.1*rand(n)
A_init = -1*rand(n,n)
p_init = [mu_init;  A_init[:]]

## Optimization and Fitting

i=0

function loss(p_curr)
    global i
    i+=1
    sol = DifferentialEquations.solve(prob, Tsit5(), p=p_curr, saveat=tsteps, callback=cb)
    loss = sum(abs2, Array(sol) - syndata)
    print("\n Loss of ",loss, " on iteration ",i)
    return loss, sol
end

result_ode = DiffEqFlux.sciml_train(loss, p_init,
                                    ADAM(0.1),
                                    maxiters=50)

print("\n")
print("\n p_init:", round.(p_init,digits=4))
print("\n p_optm:", round.(result_ode.minimizer,digits=4))

which outputs for a given initialization one score repetitively and then prints the same two sets of parameters, like so:

0.013182825077950067 on iteration 1
 Loss of 0.013182825077950069 on iteration 2
 Loss of 0.013182825077950067 on iteration 3
 Loss of 0.013182825077950069 on iteration 4
 Loss of 0.013182825077950067 on iteration 5
 Loss of 0.013182825077950069 on iteration 6
 Loss of 0.013182825077950067 on iteration 7
 Loss of 0.013182825077950069 on iteration 8
 Loss of 0.013182825077950067 on iteration 9
 Loss of 0.013182825077950069 on iteration 10
 Loss of 0.013182825077950067 on iteration 11
 Loss of 0.013182825077950069 on iteration 12
 Loss of 0.013182825077950067 on iteration 13
 Loss of 0.013182825077950069 on iteration 14
 Loss of 0.013182825077950067 on iteration 15
 Loss of 0.013182825077950069 on iteration 16
 Loss of 0.013182825077950067 on iteration 17
 Loss of 0.013182825077950069 on iteration 18
 Loss of 0.013182825077950067 on iteration 19
 Loss of 0.013182825077950069 on iteration 20
 Loss of 0.013182825077950067 on iteration 21
 Loss of 0.013182825077950069 on iteration 22
 Loss of 0.013182825077950067 on iteration 23
 Loss of 0.013182825077950069 on iteration 24
 Loss of 0.013182825077950067 on iteration 25
 Loss of 0.013182825077950069 on iteration 26
 Loss of 0.013182825077950067 on iteration 27
 Loss of 0.013182825077950069 on iteration 28
 Loss of 0.013182825077950067 on iteration 29
 Loss of 0.013182825077950069 on iteration 30
 Loss of 0.013182825077950067 on iteration 31
 Loss of 0.013182825077950069 on iteration 32
 Loss of 0.013182825077950067 on iteration 33
 Loss of 0.013182825077950069 on iteration 34
 Loss of 0.013182825077950067 on iteration 35
 Loss of 0.013182825077950069 on iteration 36
 Loss of 0.013182825077950067 on iteration 37
 Loss of 0.013182825077950069 on iteration 38
 Loss of 0.013182825077950067 on iteration 39
 Loss of 0.013182825077950069 on iteration 40
 Loss of 0.013182825077950067 on iteration 41
 Loss of 0.013182825077950069 on iteration 42
 Loss of 0.013182825077950067 on iteration 43
 Loss of 0.013182825077950069 on iteration 44
 Loss of 0.013182825077950067 on iteration 45
 Loss of 0.013182825077950069 on iteration 46
 Loss of 0.013182825077950067 on iteration 47
 Loss of 0.013182825077950069 on iteration 48
 Loss of 0.013182825077950067 on iteration 49
 Loss of 0.013182825077950069 on iteration 50
 Loss of 0.013182825077950067 on iteration 51
 Loss of 0.013182825077950069 on iteration 52
 Loss of 0.013182825077950067 on iteration 53
 Loss of 0.013182825077950069 on iteration 54
 Loss of 0.013182825077950067 on iteration 55
 Loss of 0.013182825077950069 on iteration 56
 Loss of 0.013182825077950067 on iteration 57
 Loss of 0.013182825077950069 on iteration 58
 Loss of 0.013182825077950067 on iteration 59
 Loss of 0.013182825077950069 on iteration 60
 Loss of 0.013182825077950067 on iteration 61
 Loss of 0.013182825077950069 on iteration 62
 Loss of 0.013182825077950067 on iteration 63
 Loss of 0.013182825077950069 on iteration 64
 Loss of 0.013182825077950067 on iteration 65
 Loss of 0.013182825077950069 on iteration 66
 Loss of 0.013182825077950067 on iteration 67
 Loss of 0.013182825077950069 on iteration 68
 Loss of 0.013182825077950067 on iteration 69
 Loss of 0.013182825077950069 on iteration 70
 Loss of 0.013182825077950067 on iteration 71
 Loss of 0.013182825077950069 on iteration 72
 Loss of 0.013182825077950067 on iteration 73
 Loss of 0.013182825077950069 on iteration 74
 Loss of 0.013182825077950067 on iteration 75
 Loss of 0.013182825077950069 on iteration 76
 Loss of 0.013182825077950067 on iteration 77
 Loss of 0.013182825077950069 on iteration 78
 Loss of 0.013182825077950067 on iteration 79
 Loss of 0.013182825077950069 on iteration 80
 Loss of 0.013182825077950067 on iteration 81
 Loss of 0.013182825077950069 on iteration 82
 Loss of 0.013182825077950067 on iteration 83
 Loss of 0.013182825077950069 on iteration 84
 Loss of 0.013182825077950067 on iteration 85
 Loss of 0.013182825077950069 on iteration 86
 Loss of 0.013182825077950067 on iteration 87
 Loss of 0.013182825077950069 on iteration 88
 Loss of 0.013182825077950067 on iteration 89
 Loss of 0.013182825077950069 on iteration 90
 Loss of 0.013182825077950067 on iteration 91
 Loss of 0.013182825077950069 on iteration 92
 Loss of 0.013182825077950067 on iteration 93
 Loss of 0.013182825077950069 on iteration 94
 Loss of 0.013182825077950067 on iteration 95
 Loss of 0.013182825077950069 on iteration 96
 Loss of 0.013182825077950067 on iteration 97
 Loss of 0.013182825077950069 on iteration 98
 Loss of 0.013182825077950067 on iteration 99
 Loss of 0.013182825077950069 on iteration 100

p_init:[0.0621, 0.0467, 0.0351, -0.7881, -0.9748, -0.8404, -0.6985, -0.387, -0.8855, -0.6415, -0.5578, -0.2624]
 p_optm:[0.0621, 0.0467, 0.0351, -0.7881, -0.9748, -0.8404, -0.6985, -0.387, -0.8855, -0.6415, -0.5578, -0.2624]
2

Double post https://github.com/SciML/DiffEqFlux.jl/issues/528

What happens if you change to sensealg=ReverseDiffAdjoint()?

3

Same output and loss score

4

This of course needs to be removed since that will make all gradients zero.

5

Ah well the reason I incorporated that in the first place was because without it I receive this error which halts the execution before the first iteration:

ERROR: ArgumentError: Converting an instance of ReverseDiff.TrackedReal{Float64,Float64,ReverseDiff.TrackedArray{Float64,Float64,1,Array{Float64,1},Array{Float64,1}}} to Float64 is not defined. Please use `ReverseDiff.value` instead.

with full stacktrace

Stacktrace:
 [1] convert(::Type{Float64}, ::ReverseDiff.TrackedReal{Float64,Float64,ReverseDiff.TrackedArray{Float64,Float64,1,Array{Float64,1},Array{Float64,1}}}) at /Users/willsharpless/.julia/packages/ReverseDiff/iHmB4/src/tracked.jl:261
 [2] setindex! at ./array.jl:849 [inlined]
 [3] macro expansion at ./multidimensional.jl:802 [inlined]
 [4] macro expansion at ./cartesian.jl:64 [inlined]
 [5] macro expansion at ./multidimensional.jl:797 [inlined]
 [6] _unsafe_setindex! at ./multidimensional.jl:789 [inlined]
 [7] _setindex! at ./multidimensional.jl:785 [inlined]
 [8] setindex! at ./abstractarray.jl:1153 [inlined]
 [9] gLV!(::Array{ReverseDiff.TrackedReal{Float64,Float64,ReverseDiff.TrackedArray{Float64,Float64,1,Array{Float64,1},Array{Float64,1}}},1}, ::ReverseDiff.TrackedArray{Float64,Float64,1,Array{Float64,1},Array{Float64,1}}, ::ReverseDiff.TrackedArray{Float64,Float64,1,Array{Float64,1},Array{Float64,1}}, ::ReverseDiff.TrackedReal{Float64,Float64,ReverseDiff.TrackedArray{Float64,Float64,1,Array{Float64,1},Array{Float64,1}}}) at /Users/willsharpless/Documents/Julia/PostBin_Analysis_v2.jl:24
 [10] ODEFunction at /Users/willsharpless/.julia/packages/SciMLBase/WHspn/src/scimlfunctions.jl:334 [inlined]
 [11] (::DiffEqSensitivity.var"#89#100"{ODEFunction{true,typeof(gLV!),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,typeof(SciMLBase.DEFAULT_OBSERVED),Nothing}})(::ReverseDiff.TrackedArray{Float64,Float64,1,Array{Float64,1},Array{Float64,1}}, ::ReverseDiff.TrackedArray{Float64,Float64,1,Array{Float64,1},Array{Float64,1}}, ::ReverseDiff.TrackedArray{Float64,Float64,1,Array{Float64,1},Array{Float64,1}}) at /Users/willsharpless/.julia/packages/DiffEqSensitivity/Yn6oc/src/adjoint_common.jl:140
 [12] ReverseDiff.GradientTape(::Function, ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1}}, ::ReverseDiff.GradientConfig{Tuple{ReverseDiff.TrackedArray{Float64,Float64,1,Array{Float64,1},Array{Float64,1}},ReverseDiff.TrackedArray{Float64,Float64,1,Array{Float64,1},Array{Float64,1}},ReverseDiff.TrackedArray{Float64,Float64,1,Array{Float64,1},Array{Float64,1}}}}) at /Users/willsharpless/.julia/packages/ReverseDiff/iHmB4/src/api/tape.jl:207
 [13] ReverseDiff.GradientTape(::Function, ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1}}) at /Users/willsharpless/.julia/packages/ReverseDiff/iHmB4/src/api/tape.jl:204
 [14] adjointdiffcache(::Function, ::InterpolatingAdjoint{0,true,Val{:central},Bool,Bool}, ::Bool, ::ODESolution{Float64,2,Array{Array{Float64,1},1},Nothing,Nothing,Array{Float64,1},Array{Array{Array{Float64,1},1},1},ODEProblem{Array{Float64,1},Tuple{Float64,Float64},true,Array{Float64,1},ODEFunction{true,typeof(gLV!),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,typeof(SciMLBase.DEFAULT_OBSERVED),Nothing},Base.Iterators.Pairs{Symbol,CallbackSet{Tuple{ContinuousCallback{typeof(condition),DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64}},Tuple{}},Tuple{Symbol},NamedTuple{(:callback,),Tuple{CallbackSet{Tuple{ContinuousCallback{typeof(condition),DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64}},Tuple{}}}}},SciMLBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{true,typeof(gLV!),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,typeof(SciMLBase.DEFAULT_OBSERVED),Nothing},Array{Array{Float64,1},1},Array{Float64,1},Array{Array{Array{Float64,1},1},1},OrdinaryDiffEq.Tsit5Cache{Array{Float64,1},Array{Float64,1},Array{Float64,1},OrdinaryDiffEq.Tsit5ConstantCache{Float64,Float64}}},DiffEqBase.DEStats}, ::Nothing, ::ODEFunction{true,typeof(gLV!),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,typeof(SciMLBase.DEFAULT_OBSERVED),Nothing}; quad::Bool, noiseterm::Bool) at /Users/willsharpless/.julia/packages/DiffEqSensitivity/Yn6oc/src/adjoint_common.jl:138
 [15] DiffEqSensitivity.ODEInterpolatingAdjointSensitivityFunction(::Function, ::InterpolatingAdjoint{0,true,Val{:central},Bool,Bool}, ::Bool, ::ODESolution{Float64,2,Array{Array{Float64,1},1},Nothing,Nothing,Array{Float64,1},Array{Array{Array{Float64,1},1},1},ODEProblem{Array{Float64,1},Tuple{Float64,Float64},true,Array{Float64,1},ODEFunction{true,typeof(gLV!),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,typeof(SciMLBase.DEFAULT_OBSERVED),Nothing},Base.Iterators.Pairs{Symbol,CallbackSet{Tuple{ContinuousCallback{typeof(condition),DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64}},Tuple{}},Tuple{Symbol},NamedTuple{(:callback,),Tuple{CallbackSet{Tuple{ContinuousCallback{typeof(condition),DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64}},Tuple{}}}}},SciMLBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{true,typeof(gLV!),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,typeof(SciMLBase.DEFAULT_OBSERVED),Nothing},Array{Array{Float64,1},1},Array{Float64,1},Array{Array{Array{Float64,1},1},1},OrdinaryDiffEq.Tsit5Cache{Array{Float64,1},Array{Float64,1},Array{Float64,1},OrdinaryDiffEq.Tsit5ConstantCache{Float64,Float64}}},DiffEqBase.DEStats}, ::Nothing, ::Function, ::Array{Float64,1}, ::NamedTuple{(:reltol, :abstol),Tuple{Float64,Float64}}, ::Nothing; noiseterm::Bool) at /Users/willsharpless/.julia/packages/DiffEqSensitivity/Yn6oc/src/interpolating_adjoint.jl:65
 [16] DiffEqSensitivity.ODEInterpolatingAdjointSensitivityFunction(::Function, ::InterpolatingAdjoint{0,true,Val{:central},Bool,Bool}, ::Bool, ::ODESolution{Float64,2,Array{Array{Float64,1},1},Nothing,Nothing,Array{Float64,1},Array{Array{Array{Float64,1},1},1},ODEProblem{Array{Float64,1},Tuple{Float64,Float64},true,Array{Float64,1},ODEFunction{true,typeof(gLV!),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,typeof(SciMLBase.DEFAULT_OBSERVED),Nothing},Base.Iterators.Pairs{Symbol,CallbackSet{Tuple{ContinuousCallback{typeof(condition),DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64}},Tuple{}},Tuple{Symbol},NamedTuple{(:callback,),Tuple{CallbackSet{Tuple{ContinuousCallback{typeof(condition),DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64}},Tuple{}}}}},SciMLBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{true,typeof(gLV!),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,typeof(SciMLBase.DEFAULT_OBSERVED),Nothing},Array{Array{Float64,1},1},Array{Float64,1},Array{Array{Array{Float64,1},1},1},OrdinaryDiffEq.Tsit5Cache{Array{Float64,1},Array{Float64,1},Array{Float64,1},OrdinaryDiffEq.Tsit5ConstantCache{Float64,Float64}}},DiffEqBase.DEStats}, ::Nothing, ::Function, ::Array{Float64,1}, ::NamedTuple{(:reltol, :abstol),Tuple{Float64,Float64}}, ::Nothing) at /Users/willsharpless/.julia/packages/DiffEqSensitivity/Yn6oc/src/interpolating_adjoint.jl:23
 [17] ODEAdjointProblem(::ODESolution{Float64,2,Array{Array{Float64,1},1},Nothing,Nothing,Array{Float64,1},Array{Array{Array{Float64,1},1},1},ODEProblem{Array{Float64,1},Tuple{Float64,Float64},true,Array{Float64,1},ODEFunction{true,typeof(gLV!),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,typeof(SciMLBase.DEFAULT_OBSERVED),Nothing},Base.Iterators.Pairs{Symbol,CallbackSet{Tuple{ContinuousCallback{typeof(condition),DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64}},Tuple{}},Tuple{Symbol},NamedTuple{(:callback,),Tuple{CallbackSet{Tuple{ContinuousCallback{typeof(condition),DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64}},Tuple{}}}}},SciMLBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{true,typeof(gLV!),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,typeof(SciMLBase.DEFAULT_OBSERVED),Nothing},Array{Array{Float64,1},1},Array{Float64,1},Array{Array{Array{Float64,1},1},1},OrdinaryDiffEq.Tsit5Cache{Array{Float64,1},Array{Float64,1},Array{Float64,1},OrdinaryDiffEq.Tsit5ConstantCache{Float64,Float64}}},DiffEqBase.DEStats}, ::InterpolatingAdjoint{0,true,Val{:central},Bool,Bool}, ::DiffEqSensitivity.var"#df#168"{Array{Float64,2},Array{Float64,1},Colon}, ::StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}, ::Nothing; checkpoints::Array{Float64,1}, callback::CallbackSet{Tuple{ContinuousCallback{typeof(condition),DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64}},Tuple{}}, reltol::Float64, abstol::Float64, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /Users/willsharpless/.julia/packages/DiffEqSensitivity/Yn6oc/src/interpolating_adjoint.jl:236
 [18] _adjoint_sensitivities(::ODESolution{Float64,2,Array{Array{Float64,1},1},Nothing,Nothing,Array{Float64,1},Array{Array{Array{Float64,1},1},1},ODEProblem{Array{Float64,1},Tuple{Float64,Float64},true,Array{Float64,1},ODEFunction{true,typeof(gLV!),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,typeof(SciMLBase.DEFAULT_OBSERVED),Nothing},Base.Iterators.Pairs{Symbol,CallbackSet{Tuple{ContinuousCallback{typeof(condition),DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64}},Tuple{}},Tuple{Symbol},NamedTuple{(:callback,),Tuple{CallbackSet{Tuple{ContinuousCallback{typeof(condition),DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64}},Tuple{}}}}},SciMLBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{true,typeof(gLV!),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,typeof(SciMLBase.DEFAULT_OBSERVED),Nothing},Array{Array{Float64,1},1},Array{Float64,1},Array{Array{Array{Float64,1},1},1},OrdinaryDiffEq.Tsit5Cache{Array{Float64,1},Array{Float64,1},Array{Float64,1},OrdinaryDiffEq.Tsit5ConstantCache{Float64,Float64}}},DiffEqBase.DEStats}, ::InterpolatingAdjoint{0,true,Val{:central},Bool,Bool}, ::Tsit5, ::DiffEqSensitivity.var"#df#168"{Array{Float64,2},Array{Float64,1},Colon}, ::StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}, ::Nothing; abstol::Float64, reltol::Float64, checkpoints::Array{Float64,1}, corfunc_analytical::Nothing, callback::CallbackSet{Tuple{ContinuousCallback{typeof(condition),DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64}},Tuple{}}, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /Users/willsharpless/.julia/packages/DiffEqSensitivity/Yn6oc/src/sensitivity_interface.jl:17
 [19] adjoint_sensitivities(::ODESolution{Float64,2,Array{Array{Float64,1},1},Nothing,Nothing,Array{Float64,1},Array{Array{Array{Float64,1},1},1},ODEProblem{Array{Float64,1},Tuple{Float64,Float64},true,Array{Float64,1},ODEFunction{true,typeof(gLV!),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,typeof(SciMLBase.DEFAULT_OBSERVED),Nothing},Base.Iterators.Pairs{Symbol,CallbackSet{Tuple{ContinuousCallback{typeof(condition),DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64}},Tuple{}},Tuple{Symbol},NamedTuple{(:callback,),Tuple{CallbackSet{Tuple{ContinuousCallback{typeof(condition),DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64}},Tuple{}}}}},SciMLBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{true,typeof(gLV!),LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,typeof(SciMLBase.DEFAULT_OBSERVED),Nothing},Array{Array{Float64,1},1},Array{Float64,1},Array{Array{Array{Float64,1},1},1},OrdinaryDiffEq.Tsit5Cache{Array{Float64,1},Array{Float64,1},Array{Float64,1},OrdinaryDiffEq.Tsit5ConstantCache{Float64,Float64}}},DiffEqBase.DEStats}, ::Tsit5, ::Vararg{Any,N} where N; sensealg::InterpolatingAdjoint{0,true,Val{:central},Bool,Bool}, kwargs::Base.Iterators.Pairs{Symbol,CallbackSet{Tuple{ContinuousCallback{typeof(condition),DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64}},Tuple{}},Tuple{Symbol},NamedTuple{(:callback,),Tuple{CallbackSet{Tuple{ContinuousCallback{typeof(condition),DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},DiffEqSensitivity.TrackedAffect{Float64,typeof(affect!)},typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64}},Tuple{}}}}}) at /Users/willsharpless/.julia/packages/DiffEqSensitivity/Yn6oc/src/sensitivity_interface.jl:6
 [20] (::DiffEqSensitivity.var"#adjoint_sensitivity_backpass#167"{Base.Iterators.Pairs{Symbol,ContinuousCallback{typeof(condition),typeof(affect!),typeof(affect!),typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64},Tuple{Symbol},NamedTuple{(:callback,),Tuple{ContinuousCallback{typeof(condition),typeof(affect!),typeof(affect!),typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64}}}},Tsit5,InterpolatingAdjoint{0,true,Val{:central},Bool,Bool},Array{Float64,1},Array{Float64,1},Tuple{},NamedTuple{(),Tuple{}},Colon})(::Array{Float64,2}) at /Users/willsharpless/.julia/packages/DiffEqSensitivity/Yn6oc/src/concrete_solve.jl:179
 [21] #263#back at /Users/willsharpless/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:65 [inlined]
 [22] #178 at /Users/willsharpless/.julia/packages/Zygote/RxTZu/src/lib/lib.jl:194 [inlined]
 [23] (::Zygote.var"#1698#back#180"{Zygote.var"#178#179"{DiffEqBase.var"#263#back#74"{DiffEqSensitivity.var"#adjoint_sensitivity_backpass#167"{Base.Iterators.Pairs{Symbol,ContinuousCallback{typeof(condition),typeof(affect!),typeof(affect!),typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64},Tuple{Symbol},NamedTuple{(:callback,),Tuple{ContinuousCallback{typeof(condition),typeof(affect!),typeof(affect!),typeof(DiffEqBase.INITIALIZE_DEFAULT),typeof(DiffEqBase.FINALIZE_DEFAULT),Float64,Int64,Nothing,Int64}}}},Tsit5,InterpolatingAdjoint{0,true,Val{:central},Bool,Bool},Array{Float64,1},Array{Float64,1},Tuple{},NamedTuple{(),Tuple{}},Colon}},Tuple{NTuple{6,Nothing},Tuple{Nothing}}}})(::Array{Float64,2}) at /Users/willsharpless/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
 [24] #solve#57 at /Users/willsharpless/.julia/packages/DiffEqBase/qntkj/src/solve.jl:70 [inlined]
 [25] (::typeof(∂(#solve#57)))(::Array{Float64,2}) at /Users/willsharpless/.julia/packages/Zygote/RxTZu/src/compiler/interface2.jl:0
 [26] (::Zygote.var"#178#179"{typeof(∂(#solve#57)),Tuple{NTuple{6,Nothing},Tuple{Nothing}}})(::Array{Float64,2}) at /Users/willsharpless/.julia/packages/Zygote/RxTZu/src/lib/lib.jl:194
 [27] (::Zygote.var"#1698#back#180"{Zygote.var"#178#179"{typeof(∂(#solve#57)),Tuple{NTuple{6,Nothing},Tuple{Nothing}}}})(::Array{Float64,2}) at /Users/willsharpless/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
 [28] (::typeof(∂(solve##kw)))(::Array{Float64,2}) at /Users/willsharpless/.julia/packages/Zygote/RxTZu/src/compiler/interface2.jl:0
 [29] loss at /Users/willsharpless/Documents/Julia/PostBin_Analysis_v2.jl:71 [inlined]
 [30] (::typeof(∂(loss)))(::Tuple{Float64,Nothing}) at /Users/willsharpless/.julia/packages/Zygote/RxTZu/src/compiler/interface2.jl:0
 [31] #69 at /Users/willsharpless/.julia/packages/DiffEqFlux/alPQ3/src/train.jl:3 [inlined]
 [32] #178 at /Users/willsharpless/.julia/packages/Zygote/RxTZu/src/lib/lib.jl:194 [inlined]
 [33] #1698#back at /Users/willsharpless/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59 [inlined]
 [34] OptimizationFunction at /Users/willsharpless/.julia/packages/SciMLBase/WHspn/src/problems/basic_problems.jl:107 [inlined]
 [35] #178 at /Users/willsharpless/.julia/packages/Zygote/RxTZu/src/lib/lib.jl:194 [inlined]
 [36] #1698#back at /Users/willsharpless/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59 [inlined]
 [37] OptimizationFunction at /Users/willsharpless/.julia/packages/SciMLBase/WHspn/src/problems/basic_problems.jl:107 [inlined]
 [38] #178 at /Users/willsharpless/.julia/packages/Zygote/RxTZu/src/lib/lib.jl:194 [inlined]
 [39] (::Zygote.var"#1698#back#180"{Zygote.var"#178#179"{typeof(∂(λ)),Tuple{Tuple{Nothing,Nothing},Int64}}})(::Tuple{Float64,Nothing}) at /Users/willsharpless/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
 [40] #8 at /Users/willsharpless/.julia/packages/GalacticOptim/JnLwV/src/solve.jl:94 [inlined]
 [41] (::typeof(∂(λ)))(::Float64) at /Users/willsharpless/.julia/packages/Zygote/RxTZu/src/compiler/interface2.jl:0
 [42] (::Zygote.var"#69#70"{Zygote.Params,Zygote.Context,typeof(∂(λ))})(::Float64) at /Users/willsharpless/.julia/packages/Zygote/RxTZu/src/compiler/interface.jl:252
 [43] gradient(::Function, ::Zygote.Params) at /Users/willsharpless/.julia/packages/Zygote/RxTZu/src/compiler/interface.jl:59
 [44] __solve(::OptimizationProblem{false,OptimizationFunction{false,GalacticOptim.AutoZygote,OptimizationFunction{true,GalacticOptim.AutoZygote,DiffEqFlux.var"#69#70"{typeof(loss)},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},GalacticOptim.var"#146#156"{GalacticOptim.var"#145#155"{OptimizationFunction{true,GalacticOptim.AutoZygote,DiffEqFlux.var"#69#70"{typeof(loss)},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Nothing}},GalacticOptim.var"#149#159"{GalacticOptim.var"#145#155"{OptimizationFunction{true,GalacticOptim.AutoZygote,DiffEqFlux.var"#69#70"{typeof(loss)},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Nothing}},GalacticOptim.var"#154#164",Nothing,Nothing,Nothing},Array{Float64,1},SciMLBase.NullParameters,Nothing,Nothing,Nothing,Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:sensealg, :maxiters),Tuple{ReverseDiffAdjoint,Int64}}}}, ::ADAM, ::Base.Iterators.Cycle{Tuple{GalacticOptim.NullData}}; maxiters::Int64, cb::Function, progress::Bool, save_best::Bool, kwargs::Base.Iterators.Pairs{Symbol,ReverseDiffAdjoint,Tuple{Symbol},NamedTuple{(:sensealg,),Tuple{ReverseDiffAdjoint}}}) at /Users/willsharpless/.julia/packages/GalacticOptim/JnLwV/src/solve.jl:93
 [45] #solve#450 at /Users/willsharpless/.julia/packages/SciMLBase/WHspn/src/solve.jl:3 [inlined]
 [46] sciml_train(::typeof(loss), ::Array{Float64,1}, ::ADAM, ::GalacticOptim.AutoZygote; lower_bounds::Nothing, upper_bounds::Nothing, kwargs::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:sensealg, :maxiters),Tuple{ReverseDiffAdjoint,Int64}}}) at /Users/willsharpless/.julia/packages/DiffEqFlux/alPQ3/src/train.jl:6
 [47] top-level scope at none:1
6

The reason is because this is incorrect during differentiation: you aren’t using Float64 values all of the time. Instead you’re looking for A = Array{eltype(u),2}(undef,3,3).

2 Likes