I am solving a series of problems which are in some sense very small modifications from one another.
At eash step, I modify my loss using the information from the inverse hessian obtained by BFGS, and then i restart BFGS on the new loss. I was wandering if there would be a way to keep the BFGS instance I have (an dnot only give its hessian to a new one) at each step ?
I have something that looks like the following (not an MWE sadly).
opts = Optim.Options( show_trace=true, show_every=100, allow_f_increases=true, allow_outer_f_increases = true, iterations=10000 ) my_loss(p,λ) = ... par = randn(10) λ = 0 for i in 1:100 println("------------------------------") invH = one(zeros(10,10)) m = BFGS(initial_invH = x-> invH) obj = OnceDifferentiable(p -> my_loss(p,λ),par; autodiff = :forward) bfgsstate = Optim.initial_state(m,opts,obj,par) # allows to get back the hessian. res = optimize(obj, par, m, opts, bfgsstate) par = Optim.minimizer(res) invH = bfgsstate.invH λ += some_correction_computed_from(invH) end
This looks wastefull. Is there a way to keep the bfgsstate alltogether and only change the loss in it ? I cannot come up with a good MWE sorry