# Using Optim.jl but failing to find minimum

**URL:** <https://discourse.julialang.org/t/using-optim-jl-but-failing-to-find-minimum/82751>\
**Category:** Optimization (Mathematical)\
**Created:** [June 14, 2022, 2:51pm UTC](https://discourse.julialang.org/t/using-optim-jl-but-failing-to-find-minimum/82751 "2022-06-14T14:51:54Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![BioTurboNick](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bioturbonick/32/6380_2.png) [@BioTurboNick](https://discourse.julialang.org/u/BioTurboNick)\
**Post date:** [June 14, 2022, 2:51pm UTC](https://discourse.julialang.org/t/using-optim-jl-but-failing-to-find-minimum/82751/1 "2022-06-14T14:51:54Z")

</div>

I’m trying to minimize parameters to fit an HMM. It works part of the way, but it seems to get stuck for some points, often not too distant from the initial guess.

If I examine the one-dimensional path of the objective function through each of the parameters (choosing the “solution” parameter and sweeping through the other), we clearly haven’t reached a minimum for either

![image](https://global.discourse-cdn.com/julialang/original/3X/b/e/bef0795f52524e08b0905238637975533b0ccffc.png)  
minimum = 181.22345833012542  
 ![image](https://global.discourse-cdn.com/julialang/original/3X/6/5/65ce03ddcc157ad47e00265268864a73b28fc7bd.png)  
minimum = 183.6876674990469

If I take an approximate minimizing parameter for each of those graphs, the value of the objective function is 177.41975067999894, so this is a real improvement. And there’s no clear local minimum that seems to be present.

The problem does have some constraints on the parameters, where the objective function produces an Inf if that is violated. I tried Fminbox but it was quite a bit slower and seemed to treat the boundaries as soft rather than hard?

I’m not sure how to read the trace:

```julia
Iter Function value Gradient norm
     0 1.873669e+02 4.310469e+02
 * Current step size: 1.0
 * time: 0.0
 * g(x): [-21.477490384417457, -431.04693893543464, 74.67741374598157]    
 * x: [0.4782608695652174, 0.029177718832891244, 5.0e-5]
     1 1.872548e+02 4.245517e+02
 * Current step size: 5.884583215091771e-7
 * time: 0.023000001907348633
 * g(x): [-21.500864521498478, -424.5516786074998, 74.8087284569869]      
 * x: [0.4782735081731592, 0.029431371991068857, 6.055454452393342e-6]    
     2 1.872548e+02 4.245517e+02
 * Current step size: 4.667441528647649e-19
 * time: 0.04400014877319336
 * g(x): [-21.500864521498478, -424.5516786074998, 74.8087284569869]      
 * x: [0.4782735081731592, 0.029431371991068857, 6.0554544523933395e-6] 

```

I’m calling it with:

```julia
options = Optim.Options(x_tol = 1e-6, f_tol = 1e-6, iterations = 20000, f_calls_limit = 20000, show_trace = true, show_every = 1, extended_trace = true)
Optim.minimizer(optimize(f, starting_values, LBFGS(), options))

```

Would someone be able to help me understand what’s happening or how I can further debug?

(NLopt.jl produces the same solution FWIW)

---

<div class="post-metadata">

**Author:** ![BioTurboNick](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bioturbonick/32/6380_2.png) [@BioTurboNick](https://discourse.julialang.org/u/BioTurboNick)\
**Post date:** [June 14, 2022, 3:59pm UTC](https://discourse.julialang.org/t/using-optim-jl-but-failing-to-find-minimum/82751/2 "2022-06-14T15:59:30Z")

</div>

Hmm… maybe it’s an algorithm choice issue.

Using Nelder-Mead in NLopt works well. Guess I’ll need to poke a bit more.

Still, would be nice to know _why_ LBFGS fails here.

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [June 14, 2022, 10:23pm UTC](https://discourse.julialang.org/t/using-optim-jl-but-failing-to-find-minimum/82751/3 "2022-06-14T22:23:59Z")

</div>

Do you have a reproducible example? Is `y1` your `f` function?

What did it terminate? It seems like the step size got very small.

---

<div class="post-metadata">

**Author:** ![BioTurboNick](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bioturbonick/32/6380_2.png) [@BioTurboNick](https://discourse.julialang.org/u/BioTurboNick)\
**Post date:** [June 18, 2022, 5:14pm UTC](https://discourse.julialang.org/t/using-optim-jl-but-failing-to-find-minimum/82751/4 "2022-06-18T17:14:30Z")

</div>

Unfortunately it’s a bit complicated to share. Under what conditions does it reduce the step size so much?

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [June 19, 2022, 3:38am UTC](https://discourse.julialang.org/t/using-optim-jl-but-failing-to-find-minimum/82751/5 "2022-06-19T03:38:24Z")

</div>

I don’t know. It’s hard to say anything without a reproducible example.

> where the objective function produces an Inf if that is violated.

I overlooked this initially. That would suggest that your problem is not smooth or differentiable? In which case, you’re violating the assumptions of BFGS.

---

<div class="post-metadata">

**Author:** ![cvanaret](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cvanaret/32/11594_2.png) [@cvanaret](https://discourse.julialang.org/u/cvanaret)\
**Post date:** [June 19, 2022, 10:23am UTC](https://discourse.julialang.org/t/using-optim-jl-but-failing-to-find-minimum/82751/6 "2022-06-19T10:23:56Z")

</div>

Two questions:

- you did a sweep to look for the minimum. Is the minimizer feasible?
- have you tried constrained optimization methods instead of penalizing infeasible points?

---

<div class="post-metadata">

**Author:** ![BioTurboNick](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bioturbonick/32/6380_2.png) [@BioTurboNick](https://discourse.julialang.org/u/BioTurboNick)\
**Post date:** [June 19, 2022, 1:44pm UTC](https://discourse.julialang.org/t/using-optim-jl-but-failing-to-find-minimum/82751/7 "2022-06-19T13:44:05Z")

</div>

- Yes, Nelder-Mead finds the expected minimum and it’s generally a smooth function (though lots of bumpiness due to numerical issues)
- Fminbox-wrapped LBFGS was too slow

I settled on Nelder-Mead which seems to work well, so I’ll just stick with that. I’ll do some more reading on the rest. Thanks for the assistance!