# Newton optimization with Optim doing strange things

**URL:** <https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008>\
**Category:** Optimization (Mathematical)\
**Created:** [March 27, 2018, 3:41pm UTC](https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008 "2018-03-27T15:41:59Z")\
**Posts on this page:** 17\
**Page:** 2

<div class="post-metadata">

**Author:** ![antoine-levitt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antoine-levitt/32/4008_2.png) [@antoine-levitt](https://discourse.julialang.org/u/antoine-levitt)\
**Post date:** [April 11, 2018, 7:44pm UTC](https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008/21 "2018-04-11T19:44:51Z")

</div>

> [@bdeonovic](#):
>
> but I think its because overflow occured in my objective function

Floats don’t work like that:

```julia
julia> 10.0^100
1.0e100

julia> 10.0^1000
Inf

```

As far as I can tell Optim takes a step, the linesearch tells it that the step is fine, it continues. But the second step is in a region where f(x1) is smaller that f(x0) but your gradient/hessian are NaNs, which I imagine is because x1 goes outside of the “physical” parameter space for x.

Also, the Hessian at convergence of the second algorithm is not positive definite, which means that there’s something fishy in your problem. If I had to guess, from the structure of the Hessian I’d say that you’re minimizing a Lagrangian?

---

<div class="post-metadata">

**Author:** ![pkofod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pkofod/32/2179_2.png) [@pkofod](https://discourse.julialang.org/u/pkofod)\
**Post date:** [April 11, 2018, 7:49pm UTC](https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008/22 "2018-04-11T19:49:20Z")

</div>

Your problem seems quite weird. That gradient with sup-norm 6.0e301 after the first step with step size 1.0? Something seems very wrong. When taking the second step your first element of the gradient is 5.44e180 - no wonder you’re experiencing weird steps here!

PF might not be perfect, but are you sure everything here is actually correct?

---

<div class="post-metadata">

**Author:** ![bdeonovic](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bdeonovic/32/3928_2.png) [@bdeonovic](https://discourse.julialang.org/u/bdeonovic)\
**Post date:** [April 11, 2018, 7:59pm UTC](https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008/23 "2018-04-11T19:59:58Z")

</div>

What could be the problem?

---

<div class="post-metadata">

**Author:** ![pkofod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pkofod/32/2179_2.png) [@pkofod](https://discourse.julialang.org/u/pkofod)\
**Post date:** [April 11, 2018, 8:00pm UTC](https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008/24 "2018-04-11T20:00:58Z")

</div>

So what is the trace for Newton with \ ?

Also, what exactly is the thing you’re minimizing here?

---

<div class="post-metadata">

**Author:** ![bdeonovic](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bdeonovic/32/3928_2.png) [@bdeonovic](https://discourse.julialang.org/u/bdeonovic)\
**Post date:** [April 11, 2018, 8:01pm UTC](https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008/25 "2018-04-11T20:01:07Z")

</div>

Yes I am minimizing a lagrangian.

---

<div class="post-metadata">

**Author:** ![bdeonovic](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bdeonovic/32/3928_2.png) [@bdeonovic](https://discourse.julialang.org/u/bdeonovic)\
**Post date:** [April 11, 2018, 8:04pm UTC](https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008/26 "2018-04-11T20:04:32Z")

</div>

The non-positive factorization trace is also in my post.

---

<div class="post-metadata">

**Author:** ![antoine-levitt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antoine-levitt/32/4008_2.png) [@antoine-levitt](https://discourse.julialang.org/u/antoine-levitt)\
**Post date:** [April 11, 2018, 8:04pm UTC](https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008/27 "2018-04-11T20:04:53Z")

</div>

Knew it! 🙂

Well, don’t. That’s not how constrained optimization works. The minimum of the Lagrangian is -Inf, pretty much always. The point you’re looking for is a saddle point (not a minimum) of the Lagrangian, which you can get by solving the nonlinear system of equations \nabla L = 0 using for instance NLSolve, or, better yet, specialized constrained optimization packages.

---

<div class="post-metadata">

**Author:** ![pkofod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pkofod/32/2179_2.png) [@pkofod](https://discourse.julialang.org/u/pkofod)\
**Post date:** [April 11, 2018, 8:07pm UTC](https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008/28 "2018-04-11T20:07:02Z")

</div>

So what Antoine is saying here is correct, and the fact that PF doesn’t work and \ does work is merely luck I think. Neither is supposed to work, as the line search and the whole setup in general will try to get you down hill.

---

<div class="post-metadata">

**Author:** ![bdeonovic](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bdeonovic/32/3928_2.png) [@bdeonovic](https://discourse.julialang.org/u/bdeonovic)\
**Post date:** [April 11, 2018, 8:13pm UTC](https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008/29 "2018-04-11T20:13:28Z")

</div>

What packages do you guys recommend?

---

<div class="post-metadata">

**Author:** ![bdeonovic](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bdeonovic/32/3928_2.png) [@bdeonovic](https://discourse.julialang.org/u/bdeonovic)\
**Post date:** [April 11, 2018, 8:22pm UTC](https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008/30 "2018-04-11T20:22:33Z")

</div>

JuMP interface is complete mess compared to optim

---

<div class="post-metadata">

**Author:** ![pkofod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pkofod/32/2179_2.png) [@pkofod](https://discourse.julialang.org/u/pkofod)\
**Post date:** [April 11, 2018, 8:26pm UTC](https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008/31 "2018-04-11T20:26:44Z")

</div>

Have you tried NLopt.jl?

---

<div class="post-metadata">

**Author:** ![anriseth](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/anriseth/32/3106_2.png) [@anriseth](https://discourse.julialang.org/u/anriseth)\
**Post date:** [April 11, 2018, 9:44pm UTC](https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008/32 "2018-04-11T21:44:41Z")

</div>

May I propose [ConstrainedOptim](https://github.com/JuliaNLSolvers/ConstrainedOptim.jl)? It has not been tested on many users yet, but I’d be keen on seeing how you find it. (We are hoping to move it into Optim “in the future”)

If the constraints of your original problem are linear, it will be quite easy to set up the problem and the constraints.

---

<div class="post-metadata">

**Author:** ![bdeonovic](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bdeonovic/32/3928_2.png) [@bdeonovic](https://discourse.julialang.org/u/bdeonovic)\
**Post date:** [April 11, 2018, 9:46pm UTC](https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008/33 "2018-04-11T21:46:42Z")

</div>

I will check both (NLopt and ConstrainedOptim) over the weekend. I’m leaning towards NLopt because i want to be able to run the optimization with or without inputing the hessian

---

<div class="post-metadata">

**Author:** ![anriseth](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/anriseth/32/3106_2.png) [@anriseth](https://discourse.julialang.org/u/anriseth)\
**Post date:** [April 11, 2018, 9:48pm UTC](https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008/34 "2018-04-11T21:48:26Z")

</div>

Fair enough re Hessian. We definitely need to get the ball rolling on implementing a quasi-Newton interior point algorithm.

---

<div class="post-metadata">

**Author:** ![anriseth](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/anriseth/32/3106_2.png) [@anriseth](https://discourse.julialang.org/u/anriseth)\
**Post date:** [April 11, 2018, 10:06pm UTC](https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008/35 "2018-04-11T22:06:59Z")

</div>

What do your (current and future) optimization problems and constraints look like? What dimensions, number of constraints, and type of constraints are we talking about? Do you have a reference for us?

---

<div class="post-metadata">

**Author:** ![pkofod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pkofod/32/2179_2.png) [@pkofod](https://discourse.julialang.org/u/pkofod)\
**Post date:** [April 12, 2018, 7:41am UTC](https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008/36 "2018-04-12T07:41:13Z")

</div>

> [@anriseth](#):
>
> What do your (current and future) optimization problems and constraints look like? What dimensions, number of constraints, and type of constraints are we talking about? Do you have a reference for us?

This is key if you want more help no matter the package to be used.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 12, 2018, 7:46am UTC](https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008/37 "2018-04-12T07:46:18Z")

</div>

> [@pkofod](#):
>
> This is key if you want more help no matter the package to be used.

I think this is a majorly undermentioned point with mathematical OSS. Seriously, the best things ever are tests and benchmarks. If you want packages to optimize for your problem and keep it working, donate some “integration tests” related to your domain along with some benchmarks in whatever format the package authors are looking for (and give a description of what it is for future citations). I think most authors will love to have a nice new test problem donated to them, even if it starts as `@test_broken`. Just make sure you abstract out any extraneous dependencies and boil it down to the key features of the problem and it’s a good addition to any test suite.

[Previous page](https://discourse.julialang.org/t/newton-optimization-with-optim-doing-strange-things/10008.md?page=1)