Thank you for all your suggestions and help.
I tried gelu activation function. But the same issue persists. So I reduced the weights again as mentioned by multiplying the initial weights with 1e-2
. Then the issue went away. The weights were updating nicely. So starting with smaller weights helped in the case of gelu activation function too.
Is there any explanation as to why it occurs to the gelu activation function also?
I encountered another issue too. I applied the code to actual data set. I used Gelu activation function inializing by multiplying it with 1e-3
like the following
optprob = Optimization.OptimizationProblem(optf,Float32(1e-3).*ComponentVector(_para))
I did optimiazation with ADAM for 700 iterations. It was successful. Then I tried BFGS . The following is the loss function and definition of optimization problem for it
# Defining loss function for BFGS
function total_loss(θ)
total_error = 0
data_points = [
(prob1, t1, T1),
(prob2, t2, T2),
(prob3, t3, T3),
(prob4, t4, T4),
(prob5, t5, T5),
(prob6, t6, T6)
]
for (prob,t,T) in data_points
_prob = remake(prob,p=θ)
_sol = Array(solve(_prob,Tsit5(),saveat = t,sensealg = QuadratureAdjoint(autojacvec = ReverseDiffVJP(true))))
# Print dimensions of T and _sol[2,:]
println("Dimensions of T: ", size(T))
println("Dimensions of _sol[2,:]: ", size(_sol[2,:]))
error = mean(abs2,T.-_sol[2,:])
error_norm = error/(T[end] - T[1])
total_error = total_error + error_norm
end
return total_error
end
optimiser_ = BFGS()
adtype = Optimization.AutoZygote()
optf = Optimization.OptimizationFunction((x,p) -> total_loss(x),adtype)
optprob2 = Optimization.OptimizationProblem(optf,p_adam_extra)
res2 = Optimization.solve(optprob2,optimiser_,callback=callback,maxiters=100)
p_bfgs = res2.u
The dimensions are printed and MSE loss comes first. Then comes a long warning and error message
Dimensions of T: (3379,)
Dimensions of _sol[2,:]: (3379,)
Dimensions of T: (277,)
Dimensions of _sol[2,:]: (277,)
Dimensions of T: (1750,)
Dimensions of _sol[2,:]: (1750,)
Dimensions of T: (3575,)
Dimensions of _sol[2,:]: (3575,)
Dimensions of T: (253,)
Dimensions of _sol[2,:]: (253,)
Dimensions of T: (1812,)
Dimensions of _sol[2,:]: (1812,)
Dimensions of T: (3379,)
Dimensions of _sol[2,:]: (3379,)
Dimensions of T: (277,)
Dimensions of _sol[2,:]: (277,)
Dimensions of T: (1750,)
Dimensions of _sol[2,:]: (1750,)
Dimensions of T: (3575,)
Dimensions of _sol[2,:]: (3575,)
Dimensions of T: (253,)
Dimensions of _sol[2,:]: (253,)
Dimensions of T: (1812,)
Dimensions of _sol[2,:]: (1812,)
MSE Loss at iteration 704 is 0.6876666
┌ Warning: At t=0.052239357203810996, dt was forced below floating point epsilon 6.938893903907228e-18, and step error estimate = 52.62785553574862. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (3379,)
Dimensions of _sol[2,:]: (3379,)
┌ Warning: At t=2.2068473457389683, dt was forced below floating point epsilon 4.440892098500626e-16, and step error estimate = 52.62783313598562. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (277,)
Dimensions of _sol[2,:]: (277,)
┌ Warning: At t=0.27594268840767294, dt was forced below floating point epsilon 5.551115123125783e-17, and step error estimate = 52.62785514581653. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (1750,)
Dimensions of _sol[2,:]: (1750,)
┌ Warning: At t=0.049734966557859785, dt was forced below floating point epsilon 6.938893903907228e-18, and step error estimate = 52.62785087822632. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (3575,)
Dimensions of _sol[2,:]: (3575,)
┌ Warning: At t=0.05432313460574589, dt was forced below floating point epsilon 6.938893903907228e-18, and step error estimate = 52.62789286314034. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (253,)
Dimensions of _sol[2,:]: (253,)
┌ Warning: At t=0.054100158988645994, dt was forced below floating point epsilon 6.938893903907228e-18, and step error estimate = 52.62786883864311. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (1812,)
Dimensions of _sol[2,:]: (1812,)
┌ Warning: At t=1699.7619999999988, dt was forced below floating point epsilon -1.0240000000000001e-13, and step error estimate = 1.6785514216942121e6. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
┌ Warning: At t=6177.671000000002, dt was forced below floating point epsilon -5.12e-13, and step error estimate = 7.5892721393927215e6. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
┌ Warning: At t=3443.513000000001, dt was forced below floating point epsilon -1.0240000000000001e-13, and step error estimate = 1.7002288769444497e6. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
┌ Warning: At t=1641.9389999999985, dt was forced below floating point epsilon -1.0240000000000001e-13, and step error estimate = 208373.3217245348. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
┌ Warning: At t=6973.073999999999, dt was forced below floating point epsilon -5.12e-13, and step error estimate = 155487.34747782705. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
┌ Warning: At t=3251.811999999998, dt was forced below floating point epsilon -1.0240000000000001e-13, and step error estimate = 1.6710083063827937e6. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
┌ Warning: At t=0.052239357203810996, dt was forced below floating point epsilon 6.938893903907228e-18, and step error estimate = 52.62785553574862. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (3379,)
Dimensions of _sol[2,:]: (1,)
┌ Warning: At t=2.2068473457389683, dt was forced below floating point epsilon 4.440892098500626e-16, and step error estimate = 52.62783313598562. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (277,)
Dimensions of _sol[2,:]: (1,)
┌ Warning: At t=0.27594268840767294, dt was forced below floating point epsilon 5.551115123125783e-17, and step error estimate = 52.62785514581653. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (1750,)
Dimensions of _sol[2,:]: (1,)
┌ Warning: At t=0.049734966557859785, dt was forced below floating point epsilon 6.938893903907228e-18, and step error estimate = 52.62785087822632. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (3575,)
Dimensions of _sol[2,:]: (1,)
┌ Warning: At t=0.05432313460574589, dt was forced below floating point epsilon 6.938893903907228e-18, and step error estimate = 52.62789286314034. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (253,)
Dimensions of _sol[2,:]: (1,)
┌ Warning: At t=0.054100158988645994, dt was forced below floating point epsilon 6.938893903907228e-18, and step error estimate = 52.62786883864311. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (1812,)
Dimensions of _sol[2,:]: (1,)
┌ Warning: At t=0.6050178618171094, dt was forced below floating point epsilon 1.1102230246251565e-16, and step error estimate = 52.62783275643544. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (3379,)
Dimensions of _sol[2,:]: (3379,)
┌ Warning: At t=2.737000497297441, dt was forced below floating point epsilon 4.440892098500626e-16, and step error estimate = 52.62786067101712. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (277,)
Dimensions of _sol[2,:]: (277,)
┌ Warning: At t=0.8401324955473453, dt was forced below floating point epsilon 1.1102230246251565e-16, and step error estimate = 52.62788972449586. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (1750,)
Dimensions of _sol[2,:]: (1750,)
┌ Warning: At t=0.5757749197282299, dt was forced below floating point epsilon 1.1102230246251565e-16, and step error estimate = 52.62789933586094. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (3575,)
Dimensions of _sol[2,:]: (3575,)
┌ Warning: At t=0.6327587256368815, dt was forced below floating point epsilon 1.1102230246251565e-16, and step error estimate = 52.62784787958808. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (253,)
Dimensions of _sol[2,:]: (253,)
┌ Warning: At t=0.5991205280351264, dt was forced below floating point epsilon 1.1102230246251565e-16, and step error estimate = 52.62785054144224. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (1812,)
Dimensions of _sol[2,:]: (1812,)
┌ Warning: At t=1699.7619999999988, dt was forced below floating point epsilon -1.0240000000000001e-13, and step error estimate = 70379.64013275309. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
┌ Warning: At t=6177.671000000002, dt was forced below floating point epsilon -5.12e-13, and step error estimate = 640741.6811875995. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
┌ Warning: At t=3443.513000000001, dt was forced below floating point epsilon -1.0240000000000001e-13, and step error estimate = 74178.86896718017. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
┌ Warning: At t=1641.9389999999985, dt was forced below floating point epsilon -1.0240000000000001e-13, and step error estimate = 131071.10966930584. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
┌ Warning: At t=6973.073999999999, dt was forced below floating point epsilon -5.12e-13, and step error estimate = 97199.2499065072. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be
represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
┌ Warning: At t=3251.811999999998, dt was forced below floating point epsilon -1.0240000000000001e-13, and step error estimate = 69478.22193216026. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
┌ Warning: At t=0.6050178618171094, dt was forced below floating point epsilon 1.1102230246251565e-16, and step error estimate = 52.62783275643544. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (3379,)
Dimensions of _sol[2,:]: (1,)
┌ Warning: At t=2.737000497297441, dt was forced below floating point epsilon 4.440892098500626e-16, and step error estimate = 52.62786067101712. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (277,)
Dimensions of _sol[2,:]: (1,)
┌ Warning: At t=0.8401324955473453, dt was forced below floating point epsilon 1.1102230246251565e-16, and step error estimate = 52.62788972449586. Aborting. There is either an error in your model specification or the true solution is unstable (or the true solution can not be represented in the precision of Float64).
└ @ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\integrator_interface.jl:623
Dimensions of T: (1750,)
Dimensions of _sol[2,:]: (2,)
ERROR: DimensionMismatch: arrays could not be broadcast to a common size: a has axes Base.OneTo(1750) and b has axes Base.OneTo(2)
Stacktrace:
[1] _bcs1
@ .\broadcast.jl:528 [inlined]
[2] _bcs
@ .\broadcast.jl:522 [inlined]
[3] broadcast_shape
@ .\broadcast.jl:516 [inlined]
[4] combine_axes
@ .\broadcast.jl:497 [inlined]
[5] instantiate
@ .\broadcast.jl:307 [inlined]
[6] materialize(bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{…}, Nothing, typeof(-), Tuple{…}})
@ Base.Broadcast .\broadcast.jl:872
[7] total_loss(θ::ComponentVector{Float32, Vector{Float32}, Tuple{Axis{…}}})
@ Main e:\PhD Ashima\Neural ODE\Julia\IQgen_Tmixed\Updated_code_full_dataset.jl:238
[8] (::var"#21#22")(x::ComponentVector{Float32, Vector{Float32}, Tuple{Axis{…}}}, p::SciMLBase.NullParameters)
@ Main e:\PhD Ashima\Neural ODE\Julia\IQgen_Tmixed\Updated_code_full_dataset.jl:249
[9] (::OptimizationOptimJL.var"#7#13"{OptimizationCache{…}})(θ::ComponentVector{Float32, Vector{…}, Tuple{…}})
@ OptimizationOptimJL C:\Users\Kalath_A\.julia\packages\OptimizationOptimJL\e3bUa\src\OptimizationOptimJL.jl:158
[10] (::OptimizationOptimJL.var"#8#14"{…})(G::ComponentVector{…}, θ::ComponentVector{…})
@ OptimizationOptimJL C:\Users\Kalath_A\.julia\packages\OptimizationOptimJL\e3bUa\src\OptimizationOptimJL.jl:171
[11] value_gradient!!(obj::TwiceDifferentiable{…}, x::ComponentVector{…})
@ NLSolversBase C:\Users\Kalath_A\.julia\packages\NLSolversBase\kavn7\src\interface.jl:82
[12] value_gradient!(obj::TwiceDifferentiable{…}, x::ComponentVector{…})
@ NLSolversBase C:\Users\Kalath_A\.julia\packages\NLSolversBase\kavn7\src\interface.jl:69
[13] value_gradient!(obj::Optim.ManifoldObjective{TwiceDifferentiable{…}}, x::ComponentVector{Float32, Vector{…}, Tuple{…}})
@ NLSolversBase C:\Users\Kalath_A\.julia\packages\NLSolversBase\kavn7\src\interface.jl:69
[13] value_gradient!(obj::Optim.ManifoldObjective{TwiceDifferentiable{…}}, x::ComponentVector{Float32, Vector{…}, Tuple{…}})
[13] value_gradient!(obj::Optim.ManifoldObjective{TwiceDifferentiable{…}}, x::ComponentVector{Float32, Vector{…}, Tuple{…}})
@ Optim C:\Users\Kalath_A\.julia\packages\Optim\HvjCd\src\Manifolds.jl:50
[14] (::LineSearches.var"#ϕdϕ#6"{…})(α::Float32)
[14] (::LineSearches.var"#ϕdϕ#6"{…})(α::Float32)
@ LineSearches C:\Users\Kalath_A\.julia\packages\LineSearches\jgnxK\src\LineSearches.jl:83
[15] (::HagerZhang{…})(ϕ::Function, ϕdϕ::LineSearches.var"#ϕdϕ#6"{…}, c::Float32, phi_0::Float32, dphi_0::Float32)
@ LineSearches C:\Users\Kalath_A\.julia\packages\LineSearches\jgnxK\src\LineSearches.jl:83
[15] (::HagerZhang{…})(ϕ::Function, ϕdϕ::LineSearches.var"#ϕdϕ#6"{…}, c::Float32, phi_0::Float32, dphi_0::Float32)
@ LineSearches C:\Users\Kalath_A\.julia\packages\LineSearches\jgnxK\src\hagerzhang.jl:305
[16] HagerZhang
[16] HagerZhang
@ C:\Users\Kalath_A\.julia\packages\LineSearches\jgnxK\src\hagerzhang.jl:102 [inlined]
@ C:\Users\Kalath_A\.julia\packages\LineSearches\jgnxK\src\hagerzhang.jl:102 [inlined]
[17] perform_linesearch!(state::Optim.BFGSState{…}, method::BFGS{…}, d::Optim.ManifoldObjective{…})
@ Optim C:\Users\Kalath_A\.julia\packages\Optim\HvjCd\src\utilities\perform_linesearch.jl:58
[17] perform_linesearch!(state::Optim.BFGSState{…}, method::BFGS{…}, d::Optim.ManifoldObjective{…})
@ Optim C:\Users\Kalath_A\.julia\packages\Optim\HvjCd\src\utilities\perform_linesearch.jl:58
@ Optim C:\Users\Kalath_A\.julia\packages\Optim\HvjCd\src\utilities\perform_linesearch.jl:58
[18] update_state!(d::TwiceDifferentiable{…}, state::Optim.BFGSState{…}, method::BFGS{…})
[18] update_state!(d::TwiceDifferentiable{…}, state::Optim.BFGSState{…}, method::BFGS{…})
@ Optim C:\Users\Kalath_A\.julia\packages\Optim\HvjCd\src\multivariate\solvers\first_order\bfgs.jl:139
[19] optimize(d::TwiceDifferentiable{…}, initial_x::ComponentVector{…}, method::BFGS{…}, options::Optim.Options{…}, state::Optim.BFGSState{…})
@ Optim C:\Users\Kalath_A\.julia\packages\Optim\HvjCd\src\multivariate\optimize\optimize.jl:54
[20] optimize(d::TwiceDifferentiable{…}, initial_x::ComponentVector{…}, method::BFGS{…}, options::Optim.Options{…})
@ Optim C:\Users\Kalath_A\.julia\packages\Optim\HvjCd\src\multivariate\optimize\optimize.jl:36
[21] __solve(cache::OptimizationCache{…})
@ Optim C:\Users\Kalath_A\.julia\packages\Optim\HvjCd\src\multivariate\solvers\first_order\bfgs.jl:139
[19] optimize(d::TwiceDifferentiable{…}, initial_x::ComponentVector{…}, method::BFGS{…}, options::Optim.Options{…}, state::Optim.BFGSState{…})
@ Optim C:\Users\Kalath_A\.julia\packages\Optim\HvjCd\src\multivariate\optimize\optimize.jl:54
[20] optimize(d::TwiceDifferentiable{…}, initial_x::ComponentVector{…}, method::BFGS{…}, options::Optim.Options{…})
@ Optim C:\Users\Kalath_A\.julia\packages\Optim\HvjCd\src\multivariate\optimize\optimize.jl:36
[21] __solve(cache::OptimizationCache{…})
[19] optimize(d::TwiceDifferentiable{…}, initial_x::ComponentVector{…}, method::BFGS{…}, options::Optim.Options{…}, state::Optim.BFGSState{…})
@ Optim C:\Users\Kalath_A\.julia\packages\Optim\HvjCd\src\multivariate\optimize\optimize.jl:54
[20] optimize(d::TwiceDifferentiable{…}, initial_x::ComponentVector{…}, method::BFGS{…}, options::Optim.Options{…})
@ Optim C:\Users\Kalath_A\.julia\packages\Optim\HvjCd\src\multivariate\optimize\optimize.jl:36
[21] __solve(cache::OptimizationCache{…})
FGSState{…})
@ Optim C:\Users\Kalath_A\.julia\packages\Optim\HvjCd\src\multivariate\optimize\optimize.jl:54
[20] optimize(d::TwiceDifferentiable{…}, initial_x::ComponentVector{…}, method::BFGS{…}, options::Optim.Options{…})
@ Optim C:\Users\Kalath_A\.julia\packages\Optim\HvjCd\src\multivariate\optimize\optimize.jl:36
[21] __solve(cache::OptimizationCache{…})
@ Optim C:\Users\Kalath_A\.julia\packages\Optim\HvjCd\src\multivariate\optimize\optimize.jl:54
[20] optimize(d::TwiceDifferentiable{…}, initial_x::ComponentVector{…}, method::BFGS{…}, options::Optim.Options{…})
@ Optim C:\Users\Kalath_A\.julia\packages\Optim\HvjCd\src\multivariate\optimize\optimize.jl:36
[21] __solve(cache::OptimizationCache{…})
[20] optimize(d::TwiceDifferentiable{…}, initial_x::ComponentVector{…}, method::BFGS{…}, options::Optim.Options{…})
@ Optim C:\Users\Kalath_A\.julia\packages\Optim\HvjCd\src\multivariate\optimize\optimize.jl:36
[21] __solve(cache::OptimizationCache{…})
@ Optim C:\Users\Kalath_A\.julia\packages\Optim\HvjCd\src\multivariate\optimize\optimize.jl:36
[21] __solve(cache::OptimizationCache{…})
@ OptimizationOptimJL C:\Users\Kalath_A\.julia\packages\OptimizationOptimJL\e3bUa\src\OptimizationOptimJL.jl:218
@ OptimizationOptimJL C:\Users\Kalath_A\.julia\packages\OptimizationOptimJL\e3bUa\src\OptimizationOptimJL.jl:218
[22] solve!(cache::OptimizationCache{…})
@ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\solve.jl:187
[23] solve(::OptimizationProblem{…}, ::BFGS{…}; kwargs::@Kwargs{…})
@ SciMLBase C:\Users\Kalath_A\.julia\packages\SciMLBase\tWwhl\src\solve.jl:95
[24] top-level scope
@ e:\PhD Ashima\Neural ODE\Julia\IQgen_Tmixed\Updated_code_full_dataset.jl:251
Some type information was truncated. Use `show(err)` to see complete types.
I encountered a similar issue when using Relu and Gelu activation function without the multipilcation factor. When I added the 1e-3
factor to the initial parameter it went away. Then when I try BFGS after 700 iterations in ADAM, the issue popped up again.
Any idea why this happens? I think the problem is with ODE solving. But I don’t understand where it is going wrong