What is the best way to modify existing or add new optimizer in Optim?

Specific Domains Optimization (Mathematical)
, ,
1

Hi everyone,

I am wondering what would be the best way to modify an existing optimizer or add a totally new on in Optim.

My interest is modify the update formula of the inverse Hessian approximation in BFGS algorithm. The modifications are not very hard to implement and they solely depend on the current and previous state and objective function gradient.

In this paper (I am one of the authors), we have shown that such modifications can considerably enhance the accuracy of physics-informed neural networks. We have implemented these changes in python by modifying the source code of scipy and doing a back and forth between tensorflow and scipy to use them, which obviously isn’t the most elegant or fast solution.

I am trying to translate our codes in Julia, and I would like to know if the way to go is to modify the Optim source code (and if so, what would be the steps exactly?) or if there is any other preferred solution for creating or modifying an optimizer.

Thanks for the help.

1 Like
2

I’d create a new environment, dev Optim and then @edit BFGS(); that is,

(@oldenvironment) pkg> activate ./newenvironment
(@newenvironment) pkg> dev Optim

then in the REPL

using Optim
@edit BFGS()

If you find the modification straightforward to implement, then just do so and see how performance varies (compared to the unmofidied version of Optim in the old environment).

If it’s tricky to modify, then it’s more reasonable create a new file, follow the same pattern and then include the file appropriately in src/Optim.jl.

3

Hi @Yuan-Ru-Lin and thanks a lot for your reply and the clear instructions.

I have two follow-up questions.

  1. What does dev do exactly in this case?
  2. Are there any other parts inside Optim which I need to modify? Or just creating a new file and using include is enough to have the same functionality for my optimizer as for the already implemented BFGS() (e.g. will solve recognise it, will it be fast etc).
4

What does dev do exactly in this case?

After you dev a package in an enrivonment, you’d be using the copy of the package in ~/.julia/dev/ instead of the original copy in ~/.julia/packages/. That way, any modification you make to the copy in ~/.julia/dev will not change the behavior of the package used in other environments. In short, you won’t be using prototypes in your production environment. (See more information by entering pkg> ?dev.)

Are there any other parts inside Optim which I need to modify? Or just creating a new file and using include is enough to have the same functionality for my optimizer as for the already implemented BFGS() (e.g. will solve recognise it, will it be fast etc).

You will need to export yournewmethod if you don’t want to be explicitly using it.

You probably will have to implement something in fminbox.jl since that’s the other place where the name occurs in src/ (as a non-comment).

➜  Optim git:(master) grep -ir " bfgs" src/
src//multivariate/optimize/optimize.jl:            break # it returns true if it's forced by something in update! to stop (eg dx_dg == 0.0 in BFGS, or linesearch errors)
src//multivariate/solvers/first_order/bfgs.jl:struct BFGS{IL, L, H, T, TM} <: FirstOrderOptimizer
src//multivariate/solvers/first_order/bfgs.jl:# BFGS
src//multivariate/solvers/first_order/bfgs.jl:function BFGS(; alphaguess = LineSearches.InitialStatic(), # TODO: benchmark defaults
src//multivariate/solvers/first_order/bfgs.jl:    BFGS(_alphaguess(alphaguess), linesearch, initial_invH, initial_stepnorm, manifold)
src//multivariate/solvers/first_order/bfgs.jl:mutable struct BFGSState{Tx, Tm, T,G} <: AbstractOptimizerState
src//multivariate/solvers/first_order/bfgs.jl:    BFGSState(initial_x, # Maintain current state in state.x
src//multivariate/solvers/first_order/l_bfgs.jl:The `LBFGS` method implements the limited-memory BFGS algorithm as described in
src//multivariate/solvers/constrained/ipnewton/interior.jl:        update_state!(d, constraints, state, method, options) && break # it returns true if it's forced by something in update! to stop (eg dx_dg == 0.0 in BFGS or linesearch errors)
src//multivariate/solvers/constrained/fminbox.jl:barrier_method(m::Union{NelderMead, SimulatedAnnealing, ParticleSwarm, BFGS, AbstractNGMRES},
src//Optim.jl:       BFGS,
src//utilities/assess_convergence.jl:# AcceleratedGradientDescentState, BFGSState, ConjugateGradientState,

As for the performance, it will depend on how you implement the method and how well you optimize it (in terms of performance). I can’t guarantee anything.

5

Great! Thank you for your detailed replies, your help is appreciated.

1 Like