Optimization.jl, DataInterpolations.jl and Gradients

General Usage
, ,
1

Hello - This is sort of a follow-up to this post here which didn’t receive attention. Below I provide a more explicit example of what I’m trying to do with interpolating and gradients during optimization. I understand this is a stilly example because I could explicitly use a least squares spline, but in my actual problem there are extra steps to construct the full model before comparing with the data, so a simple least squares spline would not work, thus why I would like this functionality.

In this example, I try both AutoForwardDiff() and AutoZyote().

using Optimization, Optimisers, OptimizationOptimisers, OptimizationFlux, Zygote, ForwardDiff
using DataInterpolations, Distributions

function gauss(x, a, μ, σ)
    return @. a * exp(-0.5 * ((x - μ) / σ)^2)
end

function rmsloss(x, y)
    r = x .- y
    return sqrt(sum(r.^2) / length(r))
end

# Data on arbitrary grid
x = sort(rand(Uniform(-10, 10), 100))
ptrue = [1.0, 1.2, 2.5]
data = gauss.(x, ptrue...)

# Objective
function obj(pars, extra)
    y = DataInterpolations.CubicSpline(pars, extra[1]).(extra[2])
    return rmsloss(extra[3], y)
end
objfunc = OptimizationFunction(obj, Optimization.AutoForwardDiff())
#objfunc = OptimizationFunction(obj, Optimization.AutoZygote())


# Initial params
n_knots = 50
knots_x = collect(range(-10, stop=10, length=n_knots))
p0 = ones(n_knots)

# Extra params
extra = (knots_x, x, data)

# Opt problem
prob = OptimizationProblem(objfunc, p0, extra)

# Solve
result = solve(prob, ADAM(0.001, (0.9, 0.999)), maxiters=1000)

And I receive the following errors:

Zygote:

ERROR: LoadError: ArgumentError: new: too few arguments (expected 4)
Stacktrace:
  [1] __new__
    @ ~/.julia/packages/Zygote/SuKWp/src/tools/builtins.jl:9 [inlined]
  [2] adjoint
    @ ~/.julia/packages/Zygote/SuKWp/src/lib/lib.jl:296 [inlined]
  [3] _pullback
    @ ~/.julia/packages/ZygoteRules/AIbCs/src/adjoint.jl:65 [inlined]
  [4] _pullback
    @ /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/LinearAlgebra/src/tridiag.jl:498 [inlined]
  [5] _pullback(::Zygote.Context{false}, ::Type{LinearAlgebra.Tridiagonal{Float64, Vector{Float64}}}, ::Vector{Float64}, ::Vector{Float64}, ::Vector{Float64})
    @ Zygote ~/.julia/packages/Zygote/SuKWp/src/compiler/interface2.jl:0
  [6] _pullback
    @ /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/LinearAlgebra/src/tridiag.jl:533 [inlined]
  [7] _pullback
    @ ~/.julia/packages/DataInterpolations/ivHqg/src/interpolation_caches.jl:160 [inlined]
  [8] _pullback(::Zygote.Context{false}, ::Type{CubicSpline}, ::Vector{Float64}, ::Vector{Float64})
    @ Zygote ~/.julia/packages/Zygote/SuKWp/src/compiler/interface2.jl:0
  [9] _pullback
    @ ~/Desktop/test_interp1d2.jl:20 [inlined]
 [10] _pullback(::Zygote.Context{false}, ::typeof(obj), ::Vector{Float64}, ::Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}})
    @ Zygote ~/.julia/packages/Zygote/SuKWp/src/compiler/interface2.jl:0
 [11] _apply(::Function, ::Vararg{Any})
    @ Core ./boot.jl:838
 [12] adjoint
    @ ~/.julia/packages/Zygote/SuKWp/src/lib/lib.jl:203 [inlined]
 [13] _pullback
    @ ~/.julia/packages/ZygoteRules/AIbCs/src/adjoint.jl:65 [inlined]
 [14] _pullback
    @ ~/.julia/packages/SciMLBase/ys6dl/src/scimlfunctions.jl:3596 [inlined]
 [15] _pullback(::Zygote.Context{false}, ::OptimizationFunction{true, Optimization.AutoZygote, typeof(obj), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, ::Vector{Float64}, ::Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}})
    @ Zygote ~/.julia/packages/Zygote/SuKWp/src/compiler/interface2.jl:0
 [16] _apply(::Function, ::Vararg{Any})
    @ Core ./boot.jl:838
 [17] adjoint
    @ ~/.julia/packages/Zygote/SuKWp/src/lib/lib.jl:203 [inlined]
 [18] _pullback
    @ ~/.julia/packages/ZygoteRules/AIbCs/src/adjoint.jl:65 [inlined]
 [19] _pullback
    @ ~/.julia/packages/Optimization/RHDsr/src/function/zygote.jl:31 [inlined]
 [20] _pullback(ctx::Zygote.Context{false}, f::Optimization.var"#138#147"{OptimizationFunction{true, Optimization.AutoZygote, typeof(obj), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}}, args::Vector{Float64})
    @ Zygote ~/.julia/packages/Zygote/SuKWp/src/compiler/interface2.jl:0
 [21] _apply(::Function, ::Vararg{Any})
    @ Core ./boot.jl:838
 [22] adjoint
    @ ~/.julia/packages/Zygote/SuKWp/src/lib/lib.jl:203 [inlined]
 [23] _pullback
    @ ~/.julia/packages/ZygoteRules/AIbCs/src/adjoint.jl:65 [inlined]
 [24] _pullback
    @ ~/.julia/packages/Optimization/RHDsr/src/function/zygote.jl:35 [inlined]
 [25] _pullback(ctx::Zygote.Context{false}, f::Optimization.var"#140#149"{Tuple{}, Optimization.var"#138#147"{OptimizationFunction{true, Optimization.AutoZygote, typeof(obj), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}}}, args::Vector{Float64})
    @ Zygote ~/.julia/packages/Zygote/SuKWp/src/compiler/interface2.jl:0
 [26] pullback(f::Function, cx::Zygote.Context{false}, args::Vector{Float64})
    @ Zygote ~/.julia/packages/Zygote/SuKWp/src/compiler/interface.jl:44
 [27] pullback
    @ ~/.julia/packages/Zygote/SuKWp/src/compiler/interface.jl:42 [inlined]
 [28] gradient(f::Function, args::Vector{Float64})
    @ Zygote ~/.julia/packages/Zygote/SuKWp/src/compiler/interface.jl:96
 [29] (::Optimization.var"#139#148"{Optimization.var"#138#147"{OptimizationFunction{true, Optimization.AutoZygote, typeof(obj), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}}})(::Vector{Float64}, ::Vector{Float64})
    @ Optimization ~/.julia/packages/Optimization/RHDsr/src/function/zygote.jl:33
 [30] macro expansion
    @ ~/.julia/packages/OptimizationOptimisers/FWIuf/src/OptimizationOptimisers.jl:31 [inlined]
 [31] macro expansion
    @ ~/.julia/packages/Optimization/RHDsr/src/utils.jl:37 [inlined]
 [32] __solve(prob::OptimizationProblem{true, OptimizationFunction{true, Optimization.AutoZygote, typeof(obj), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, opt::Optimisers.Adam{Float64}, data::Base.Iterators.Cycle{Tuple{Optimization.NullData}}; maxiters::Int64, callback::Function, progress::Bool, save_best::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ OptimizationOptimisers ~/.julia/packages/OptimizationOptimisers/FWIuf/src/OptimizationOptimisers.jl:30
 [33] __solve (repeats 2 times)
    @ ~/.julia/packages/OptimizationOptimisers/FWIuf/src/OptimizationOptimisers.jl:7 [inlined]
 [34] #solve#544
    @ ~/.julia/packages/SciMLBase/ys6dl/src/solve.jl:85 [inlined]
 [35] top-level scope
    @ ~/Desktop/test_interp1d2.jl:37
 [36] include(fname::String)
    @ Base.MainInclude ./client.jl:478
 [37] top-level scope
    @ REPL[2]:1
in expression starting at /Users/cale/Desktop/test_interp1d2.jl:37

julia>

ForwardDiff:

ERROR: LoadError: MethodError: no method matching Float64(::ForwardDiff.Dual{ForwardDiff.Tag{Optimization.var"#70#87"{OptimizationFunction{true, Optimization.AutoForwardDiff{nothing}, typeof(obj), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 12})

Closest candidates are:
  (::Type{T})(::Real, ::RoundingMode) where T<:AbstractFloat
   @ Base rounding.jl:207
  (::Type{T})(::T) where T<:Number
   @ Core boot.jl:792
  (::Type{T})(::AbstractChar) where T<:Union{AbstractChar, Number}
   @ Base char.jl:50
  ...

Stacktrace:
  [1] convert(#unused#::Type{Float64}, x::ForwardDiff.Dual{ForwardDiff.Tag{Optimization.var"#70#87"{OptimizationFunction{true, Optimization.AutoForwardDiff{nothing}, typeof(obj), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 12})
    @ Base ./number.jl:7
  [2] setindex!(A::Vector{Float64}, x::ForwardDiff.Dual{ForwardDiff.Tag{Optimization.var"#70#87"{OptimizationFunction{true, Optimization.AutoForwardDiff{nothing}, typeof(obj), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 12}, i1::Int64)
    @ Base ./array.jl:969
  [3] _unsafe_copyto!(dest::Vector{Float64}, doffs::Int64, src::Vector{Real}, soffs::Int64, n::Int64)
    @ Base ./array.jl:250
  [4] unsafe_copyto!
    @ ./array.jl:304 [inlined]
  [5] _copyto_impl!
    @ ./array.jl:327 [inlined]
  [6] copyto!
    @ ./array.jl:314 [inlined]
  [7] copyto!
    @ ./array.jl:339 [inlined]
  [8] \(F::LinearAlgebra.LU{Float64, LinearAlgebra.Tridiagonal{Float64, Vector{Float64}}, Vector{Int64}}, B::Vector{Real})
    @ LinearAlgebra /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/LinearAlgebra/src/LinearAlgebra.jl:533
  [9] \(A::LinearAlgebra.Tridiagonal{Float64, Vector{Float64}}, B::Vector{Real})
    @ LinearAlgebra /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/LinearAlgebra/src/generic.jl:1115
 [10] CubicSpline(u::Vector{ForwardDiff.Dual{ForwardDiff.Tag{Optimization.var"#70#87"{OptimizationFunction{true, Optimization.AutoForwardDiff{nothing}, typeof(obj), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 12}}, t::Vector{Float64})
    @ DataInterpolations ~/.julia/packages/DataInterpolations/ivHqg/src/interpolation_caches.jl:162
 [11] obj(pars::Vector{ForwardDiff.Dual{ForwardDiff.Tag{Optimization.var"#70#87"{OptimizationFunction{true, Optimization.AutoForwardDiff{nothing}, typeof(obj), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 12}}, extra::Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}})
    @ Main ~/Desktop/test_interp1d2.jl:20
 [12] (::Optimization.var"#70#87"{OptimizationFunction{true, Optimization.AutoForwardDiff{nothing}, typeof(obj), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}})(::Vector{ForwardDiff.Dual{ForwardDiff.Tag{Optimization.var"#70#87"{OptimizationFunction{true, Optimization.AutoForwardDiff{nothing}, typeof(obj), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 12}})
    @ Optimization ~/.julia/packages/Optimization/RHDsr/src/function/forwarddiff.jl:47
 [13] #72
    @ ~/.julia/packages/Optimization/RHDsr/src/function/forwarddiff.jl:51 [inlined]
 [14] chunk_mode_gradient!(result::Vector{Float64}, f::Optimization.var"#72#89"{Tuple{}, Optimization.var"#70#87"{OptimizationFunction{true, Optimization.AutoForwardDiff{nothing}, typeof(obj), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}}}, x::Vector{Float64}, cfg::ForwardDiff.GradientConfig{ForwardDiff.Tag{Optimization.var"#70#87"{OptimizationFunction{true, Optimization.AutoForwardDiff{nothing}, typeof(obj), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 12, Vector{ForwardDiff.Dual{ForwardDiff.Tag{Optimization.var"#70#87"{OptimizationFunction{true, Optimization.AutoForwardDiff{nothing}, typeof(obj), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 12}}})
    @ ForwardDiff ~/.julia/packages/ForwardDiff/vXysl/src/gradient.jl:123
 [15] gradient!
    @ ~/.julia/packages/ForwardDiff/vXysl/src/gradient.jl:39 [inlined]
 [16] (::Optimization.var"#71#88"{ForwardDiff.GradientConfig{ForwardDiff.Tag{Optimization.var"#70#87"{OptimizationFunction{true, Optimization.AutoForwardDiff{nothing}, typeof(obj), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 12, Vector{ForwardDiff.Dual{ForwardDiff.Tag{Optimization.var"#70#87"{OptimizationFunction{true, Optimization.AutoForwardDiff{nothing}, typeof(obj), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 12}}}, Optimization.var"#70#87"{OptimizationFunction{true, Optimization.AutoForwardDiff{nothing}, typeof(obj), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}}})(::Vector{Float64}, ::Vector{Float64})
    @ Optimization ~/.julia/packages/Optimization/RHDsr/src/function/forwarddiff.jl:51
 [17] macro expansion
    @ ~/.julia/packages/OptimizationOptimisers/FWIuf/src/OptimizationOptimisers.jl:31 [inlined]
 [18] macro expansion
    @ ~/.julia/packages/Optimization/RHDsr/src/utils.jl:37 [inlined]
 [19] __solve(prob::OptimizationProblem{true, OptimizationFunction{true, Optimization.AutoForwardDiff{nothing}, typeof(obj), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, opt::Optimisers.Adam{Float64}, data::Base.Iterators.Cycle{Tuple{Optimization.NullData}}; maxiters::Int64, callback::Function, progress::Bool, save_best::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ OptimizationOptimisers ~/.julia/packages/OptimizationOptimisers/FWIuf/src/OptimizationOptimisers.jl:30
 [20] __solve (repeats 2 times)
    @ ~/.julia/packages/OptimizationOptimisers/FWIuf/src/OptimizationOptimisers.jl:7 [inlined]
 [21] #solve#544
    @ ~/.julia/packages/SciMLBase/ys6dl/src/solve.jl:85 [inlined]
 [22] top-level scope
    @ ~/Desktop/test_interp1d2.jl:39
 [23] include(fname::String)
    @ Base.MainInclude ./client.jl:478
 [24] top-level scope
    @ REPL[2]:1
in expression starting at /Users/cale/Desktop/test_interp1d2.jl:39

julia>

Package st:

Status `~/.julia/environments/v1.9/Project.toml`
  [a0f608ac] AffineInvariantMCMC v1.0.2
  [7d9fca2a] Arpack v0.5.4
  [5c4adb95] AstroAngles v0.1.3
  [c7932e45] AstroLib v0.4.2
  [c61b5328] AstroTime v0.7.0
  [6e4b80f9] BenchmarkTools v1.3.2
  [a5944310] CMPFit v0.3.5
  [336ed68f] CSV v0.10.9
  [49dc2e85] Calculus v0.5.1
  [5ae59095] Colors v0.12.10
  [861a8166] Combinatorics v1.0.2
  [717857b8] DSP v0.7.8
  [a93c6f00] DataFrames v1.5.0
  [82cc6244] DataInterpolations v4.0.0
  [864edb3b] DataStructures v0.18.13
  [8bb1440f] DelimitedFiles v1.9.1
⌃ [39dd38d3] Dierckx v0.5.2
⌃ [2b5f629d] DiffEqBase v6.119.0
  [31c24e10] Distributions v0.25.87
⌃ [7da242da] Enzyme v0.10.18
  [525bcba6] FITSIO v0.17.0
  [9aa1b823] FastClosures v0.3.2
  [d8961e24] FindPeaks1D v0.1.8
  [59287772] Formatting v0.4.2
  [f6369f11] ForwardDiff v0.10.35
⌃ [c27321d9] Glob v1.3.0
  [5903a43b] Infiltrator v1.6.3
  [842dd82b] InlineStrings v1.4.0
  [42fd0dbc] IterativeSolvers v0.9.2
⌃ [033835bb] JLD2 v0.4.30
  [b964fa9f] LaTeXStrings v1.3.0
  [fc60dff9] LombScargle v1.0.3
⌃ [bdcacae8] LoopVectorization v0.12.150
  [2fda8390] LsqFit v0.13.0
⌃ [23992714] MAT v0.10.3
  [6f286f6a] MultivariateStats v0.10.1
  [15e1cf62] NPZ v0.4.3
⌃ [b946abbf] NaNStatistics v0.6.24
  [86f7a689] NamedArrays v0.9.8
  [429524aa] Optim v1.7.5
  [87e2bd06] OptimBase v2.0.2
  [3bd65402] Optimisers v0.2.18
  [7f7a1694] Optimization v3.13.1
  [253f991c] OptimizationFlux v0.1.4
⌃ [36348300] OptimizationOptimJL v0.1.5
  [42dfb2eb] OptimizationOptimisers v0.1.2
  [2412ca09] ParameterHandling v0.4.6
  [18e31ff7] Peaks v0.4.3
  [fbb45041] Pickle v0.3.2
  [f0f68f2c] PlotlyJS v0.18.10
⌃ [ca7969ec] PlotlyLight v0.6.1
⌃ [f27b6e38] Polynomials v3.2.4
  [92933f4c] ProgressMeter v1.7.2
  [438e738f] PyCall v1.95.1
⌃ [d330b81b] PyPlot v2.11.0
  [6099a3de] PythonCall v0.9.12
  [1c8f45e8] RVModelingToolkit v0.1.0 `~/Development/RVModelingToolkit`
  [189a3867] Reexport v1.2.2
⌃ [295af30f] Revise v3.5.1
  [fdea26ae] SIMD v3.4.4
  [c4bf5708] SavitzkyGolay v0.6.2
⌃ [fc659fc5] SkyCoords v1.1.0
  [102930c3] SmoothingSplines v0.3.1
  [66db9d55] SnoopPrecompile v1.0.3
  [276daf66] SpecialFunctions v2.2.0
  [928aab9d] SpecialMatrices v3.0.0
⌃ [a25cea48] SpecialPolynomials v0.4.4
  [2913bbd2] StatsBase v0.33.21
  [bd369af6] Tables v1.10.1
  [1986cc42] Unitful v1.13.1
  [6112ee07] UnitfulAstro v1.2.0
  [e88e6eb3] Zygote v0.6.60
  [ade2ca70] Dates
  [8ba89e20] Distributed
  [37e2e46d] LinearAlgebra
  [2f01184e] SparseArrays
2

See if changing to y = DataInterpolations.CubicSpline(pars, eltype(pars).(extra[1])).(extra[2]) fixes the ForwardDiff one. It might be missing a type promotion within the spline construction and thus require the two elements are the same type (which would be easy to fix).

As for the Zygote one, try running the following before using any Zygote in a fresh session:

ZygoteRules.@adjoint ZygoteRules.literal_getproperty(A::Tridiagonal, ::Val{:dl}) = A.dl, y -> Tridiagonal(dl, zeros(length(d)), zeros(length(du)),)
ZygoteRules.@adjoint ZygoteRules.literal_getproperty(A::Tridiagonal, ::Val{:d}) = A.d, y -> Tridiagonal(zeros(length(dl)), d, zeros(length(du)),)
ZygoteRules.@adjoint ZygoteRules.literal_getproperty(A::Tridiagonal, ::Val{:du}) = A.dl, y -> Tridiagonal(zeros(length(dl)), zeros(length(d), du),)
ZygoteRules.@adjoint Tridiagonal(dl, d, du) = Tridiagonal(dl, d, du), p̄ -> (diag(p̄[2:end, 1:end-1]), diag(p̄), diag(p̄[1:end-1, 2:end]))
3

Thank you Chris! This worked for all of my use cases. I also tried LinearInterpolation and AkimaInterpolation with success.

Interestingly enough, I found that ForwardDiff was ~170x faster vs. Zygote (1 sec vs 170 sec) for my example above.

4

Not surprising as your optimization has far less than 100 variables (where the cutoff point for reverse mode vs forward mode tends to occur).

So the Zygote rules overloads were required? If so, I’ll report this and get those integrated into Zygote. Seems like it’s just missing a few basic rules for Tridiagonal and that would be the cause.

5

Yes, the Zygote rules worked for LinearInterpolation and CubicSpline but does fail on AkimaInterpolation (I don’t need it, but probably just needs another rule?). I didn’t test others. Casting with eltype worked across the board using ForwardDiff!