# What is the best way to modify existing or add new optimizer in Optim?
**URL:** https://discourse.julialang.org/t/what-is-the-best-way-to-modify-existing-or-add-new-optimizer-in-optim/114553
**Category:** Optimization (Mathematical)
**Tags:** question, optim, neural-network
**Created:** [May 22, 2024, 9:43am UTC](https://discourse.julialang.org/t/what-is-the-best-way-to-modify-existing-or-add-new-optimizer-in-optim/114553 "2024-05-22T09:43:03Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@PetrosStefanou](https://discourse.julialang.org/u/PetrosStefanou)
#### Post date: [May 22, 2024, 9:43am UTC](https://discourse.julialang.org/t/what-is-the-best-way-to-modify-existing-or-add-new-optimizer-in-optim/114553/1 "2024-05-22T09:43:04Z")
Hi everyone,
I am wondering what would be the best way to modify an existing optimizer or add a totally new on in Optim.
My interest is modify the update formula of the inverse Hessian approximation in BFGS algorithm. The modifications are not very hard to implement and they solely depend on the current and previous state and objective function gradient.
In [this paper](https://doi.org/10.48550/arXiv.2405.04230) (I am one of the authors), we have shown that such modifications can considerably enhance the accuracy of physics-informed neural networks. We have implemented these changes in python by modifying the source code of scipy and doing a back and forth between tensorflow and scipy to use them, which obviously isn’t the most elegant or fast solution.
I am trying to translate our codes in Julia, and I would like to know if the way to go is to modify the Optim source code (and if so, what would be the steps exactly?) or if there is any other preferred solution for creating or modifying an optimizer.
Thanks for the help.
---
### Author:  [@Yuan-Ru-Lin](https://discourse.julialang.org/u/Yuan-Ru-Lin)
#### Post date: [May 22, 2024, 6:28pm UTC](https://discourse.julialang.org/t/what-is-the-best-way-to-modify-existing-or-add-new-optimizer-in-optim/114553/2 "2024-05-22T18:28:37Z")
I’d create a new environment, `dev Optim` and then `@edit BFGS()`; that is,
```jl
(@oldenvironment) pkg> activate ./newenvironment
(@newenvironment) pkg> dev Optim
```
then in the REPL
```jl
using Optim
@edit BFGS()
```
If you find the modification straightforward to implement, then just do so and see how performance varies (compared to the unmofidied version of `Optim` in the old environment).
If it’s tricky to modify, then it’s more reasonable create a new file, follow the same pattern and then `include` the file appropriately in `src/Optim.jl`.
---
### Author:  [@PetrosStefanou](https://discourse.julialang.org/u/PetrosStefanou)
#### Post date: [May 23, 2024, 9:54am UTC](https://discourse.julialang.org/t/what-is-the-best-way-to-modify-existing-or-add-new-optimizer-in-optim/114553/3 "2024-05-23T09:54:55Z")
Hi @Yuan-Ru-Lin and thanks a lot for your reply and the clear instructions.
I have two follow-up questions.
1. What does `dev` do exactly in this case?
2. Are there any other parts inside `Optim` which I need to modify? Or just creating a new file and using `include` is enough to have the same functionality for my optimizer as for the already implemented `BFGS()` (e.g. will solve recognise it, will it be fast etc).
---
### Author:  [@Yuan-Ru-Lin](https://discourse.julialang.org/u/Yuan-Ru-Lin)
#### Post date: [May 23, 2024, 9:36pm UTC](https://discourse.julialang.org/t/what-is-the-best-way-to-modify-existing-or-add-new-optimizer-in-optim/114553/4 "2024-05-23T21:36:32Z")
> What does dev do exactly in this case?
After you `dev` a package in an enrivonment, you’d be `using` the copy of the package in `~/.julia/dev/` instead of the original copy in `~/.julia/packages/`. That way, any modification you make to the copy in `~/.julia/dev` will not change the behavior of the package used in other environments. In short, you won’t be using prototypes in your production environment. (See more information by entering `pkg> ?dev`.)
> Are there any other parts inside `Optim` which I need to modify? Or just creating a new file and using `include` is enough to have the same functionality for my optimizer as for the already implemented `BFGS()` (e.g. will solve recognise it, will it be fast etc).
You will need to `export yournewmethod` if you don’t want to be explicitly `using` it.
You probably will have to implement something in `fminbox.jl` since that’s the other place where the name occurs in `src/` (as a non-comment).
```bash
➜ Optim git:(master) grep -ir " bfgs" src/
src//multivariate/optimize/optimize.jl: break # it returns true if it's forced by something in update! to stop (eg dx_dg == 0.0 in BFGS, or linesearch errors)
src//multivariate/solvers/first_order/bfgs.jl:struct BFGS{IL, L, H, T, TM} <: FirstOrderOptimizer
src//multivariate/solvers/first_order/bfgs.jl:# BFGS
src//multivariate/solvers/first_order/bfgs.jl:function BFGS(; alphaguess = LineSearches.InitialStatic(), # TODO: benchmark defaults
src//multivariate/solvers/first_order/bfgs.jl: BFGS(_alphaguess(alphaguess), linesearch, initial_invH, initial_stepnorm, manifold)
src//multivariate/solvers/first_order/bfgs.jl:mutable struct BFGSState{Tx, Tm, T,G} <: AbstractOptimizerState
src//multivariate/solvers/first_order/bfgs.jl: BFGSState(initial_x, # Maintain current state in state.x
src//multivariate/solvers/first_order/l_bfgs.jl:The `LBFGS` method implements the limited-memory BFGS algorithm as described in
src//multivariate/solvers/constrained/ipnewton/interior.jl: update_state!(d, constraints, state, method, options) && break # it returns true if it's forced by something in update! to stop (eg dx_dg == 0.0 in BFGS or linesearch errors)
src//multivariate/solvers/constrained/fminbox.jl:barrier_method(m::Union{NelderMead, SimulatedAnnealing, ParticleSwarm, BFGS, AbstractNGMRES},
src//Optim.jl: BFGS,
src//utilities/assess_convergence.jl:# AcceleratedGradientDescentState, BFGSState, ConjugateGradientState,
```
As for the performance, it will depend on how you implement the method and how well you optimize it (in terms of performance). I can’t guarantee anything.
---
### Author:  [@PetrosStefanou](https://discourse.julialang.org/u/PetrosStefanou)
#### Post date: [May 24, 2024, 7:59am UTC](https://discourse.julialang.org/t/what-is-the-best-way-to-modify-existing-or-add-new-optimizer-in-optim/114553/5 "2024-05-24T07:59:24Z")
Great! Thank you for your detailed replies, your help is appreciated.