# How to undershand the Objective goes greater while itering?

**URL:** <https://discourse.julialang.org/t/how-to-undershand-the-objective-goes-greater-while-itering/29721>\
**Category:** Optimization (Mathematical)\
**Tags:** question\
**Created:** [October 10, 2019, 9:15am UTC](https://discourse.julialang.org/t/how-to-undershand-the-objective-goes-greater-while-itering/29721 "2019-10-10T09:15:18Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![kuro26](https://avatars.discourse-cdn.com/v4/letter/k/b487fb/32.png) [@kuro26](https://discourse.julialang.org/u/kuro26)\
**Post date:** [October 10, 2019, 9:15am UTC](https://discourse.julialang.org/t/how-to-undershand-the-objective-goes-greater-while-itering/29721/1 "2019-10-10T09:15:18Z")

</div>

![image](https://global.discourse-cdn.com/julialang/original/3X/1/f/1fcc31908ff4116e591a3493ffa8a51227f4b2b7.png)  
This is the iteration process for a problem, the result in iter 19 still acceptable.  
In my understanding of optimization, this should not happen!  
Is anybody know why waa that? I am really puzzled

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [October 10, 2019, 9:35am UTC](https://discourse.julialang.org/t/how-to-undershand-the-objective-goes-greater-while-itering/29721/2 "2019-10-10T09:35:33Z")

</div>

Welcome, please read [Please read: make it easier to help you](https://discourse.julialang.org/t/psa-make-it-easier-to-help-you/14757)  
It is much easier to help you if we know what package you are are usin and what options.  
It is also much preferred to post text copies of code rather than screenshots.

* * *

As a general rule though,  
the objective can increase because most optimization algorithm only has some samples of what the loss function looks like.  
Often just a single point and the gradient at that point (some times an approximation to the second order derivative also).  
So it can use the logic:  
_“seems like the loss function decrease in this direction. Lets try checking its value at a new point (say) 1.5 units in that direction.”_  
then it will check a new point,  
and either the pattern did contine for the next 1.5 units and the function value is lower.  
or it didn’t and it is higher – maybe it would have continued only if it had taken a 0.5 unit step.

---

<div class="post-metadata">

**Author:** ![frapac](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/frapac/32/6879_2.png) [@frapac](https://discourse.julialang.org/u/frapac)\
**Post date:** [October 10, 2019, 9:53am UTC](https://discourse.julialang.org/t/how-to-undershand-the-objective-goes-greater-while-itering/29721/3 "2019-10-10T09:53:55Z")

</div>

Also, Ipopt uses interior-point methods (hence its name) + line-search filter methods.

In non-convex optimization, the line-search does not apply directly to the objective function but rather to a _merit function_. Then, a step is accepted only if it leads to a sufficient decrease in the specified direction. So, in Ipopt, the evolution of the objective value along the iteration is not monotonous.

An example of merit function is for instance given in Equation (2.8), in this paper:

> **[nitro.pdf](https://users.iems.northwestern.edu/~nocedal/PDFfiles/nitro.pdf)**
>
> 308.20 KB

However, your problem is a bit awkward, as the objective goes from ~68.6 to 4,000 (and then to 3,000). Are you sure that your problem is smooth ? As stated by @oxinabox, a MWE would be of great help to debunk your problem.

---

<div class="post-metadata">

**Author:** ![kuro26](https://avatars.discourse-cdn.com/v4/letter/k/b487fb/32.png) [@kuro26](https://discourse.julialang.org/u/kuro26)\
**Post date:** [October 10, 2019, 2:52pm UTC](https://discourse.julialang.org/t/how-to-undershand-the-objective-goes-greater-while-itering/29721/4 "2019-10-10T14:52:54Z")

</div>

Thank you for your reply!  
Actually, objective function include a differential process, and there are some failure situation which is not proper to calculate using the same function and I also can’t set constraint here. So I used some punish function here, and it is not continuous in some place.  
Now the objective function keept in a proper palce for severial iters and then it came to a big value suddenly for a while, then back to the first proper place.  
I am using Ipopt with solver mumps, are there any good ways to avoid this?  
And this is my result with iter 100:

* * *

This is Ipopt version 3.12.10, running with linear solver mumps.  
NOTE: Other linear solvers might be more efficient (see Ipopt documentation).

Number of nonzeros in equality constraint Jacobian…: 0  
Number of nonzeros in inequality constraint Jacobian.: 0  
Number of nonzeros in Lagrangian Hessian…: 0

Total number of variables…: 4  
variables with only lower bounds: 0  
variables with lower and upper bounds: 4  
variables with only upper bounds: 0  
Total number of equality constraints…: 0  
Total number of inequality constraints…: 0  
inequality constraints with only lower bounds: 0  
inequality constraints with lower and upper bounds: 0  
inequality constraints with only upper bounds: 0

iter objective inf\_pr inf\_du lg(mu) ||d|| lg(rg) alpha\_du alpha\_pr ls  
0 2.9010486e+03 0.00e+00 6.25e+02 0.0 0.00e+00 - 0.00e+00 0.00e+00 0  
1 3.1440370e+03 0.00e+00 3.75e+02 2.4 2.75e+02 - 6.65e-01 5.66e-03f 1  
2 2.9880667e+03 0.00e+00 1.82e+02 2.1 5.59e-01 - 1.00e+00 1.00e+00f 1  
3 2.9388803e+03 0.00e+00 1.31e+04 1.4 2.45e-01 - 9.11e-01 1.00e+00f 1  
4 2.7704950e+03 0.00e+00 3.80e+04 1.4 2.55e+02 - 7.74e-03 1.36e-02f 2  
5 1.1971720e+03 0.00e+00 1.62e+05 3.4 1.12e+03 - 1.66e-02 7.80e-04f 3  
6 1.1971720e+03 0.00e+00 1.63e+05 3.0 1.12e+02 - 1.00e+00 2.09e-17f 51  
7 1.1971720e+03 0.00e+00 1.63e+05 3.2 1.42e+02 - 5.67e-02 1.29e-19f 58  
8 1.1971720e+03 0.00e+00 1.64e+05 2.5 4.00e+01 - 1.48e-01 5.71e-20f 61  
9 8.8787005e+02 0.00e+00 3.43e+04 1.8 1.31e+03 - 2.80e-03 1.99e-03f 2  
iter objective inf\_pr inf\_du lg(mu) ||d|| lg(rg) alpha\_du alpha\_pr ls  
10 4.7725300e+02 0.00e+00 6.09e+03 3.0 1.14e-01 - 9.88e-01 1.00e+00f 1  
11 9.7683633e+01 0.00e+00 3.43e+03 1.4 9.51e-02 - 1.00e+00 1.00e+00f 1  
12 2.5424607e+01 0.00e+00 1.01e+03 -0.0 4.10e-02 - 1.00e+00 1.00e+00f 1  
13 2.5420464e+01 0.00e+00 1.09e+03 -1.7 3.97e-02 - 1.00e+00 3.91e-03f 9  
14 2.5420464e+01 0.00e+00 3.30e+02 -2.0 3.02e+00 - 1.00e+00 1.88e-13f 43  
15 2.4792004e+01 0.00e+00 1.01e+03 -1.8 3.59e-03 - 1.00e+00 1.00e+00f 1  
16 2.4791476e+01 0.00e+00 9.93e+02 -3.5 5.57e-03 - 1.00e+00 1.25e-01f 4  
17 2.4791476e+01 0.00e+00 1.04e+03 -5.0 2.74e-02 - 1.00e+00 1.46e-11f 37  
18 2.4791476e+01 0.00e+00 1.00e+03 -5.1 2.37e-03 - 1.00e+00 2.33e-10f 33  
19 2.4791476e+01 0.00e+00 1.04e+03 -5.1 1.90e-05 - 1.00e+00 2.33e-10f 33  
iter objective inf\_pr inf\_du lg(mu) ||d|| lg(rg) alpha\_du alpha\_pr ls  
20 2.4791476e+01 0.00e+00 4.85e+01 -5.1 1.90e-05 - 1.00e+00 4.55e-13f 42  
21 2.4791449e+01 0.00e+00 1.05e+03 -5.1 1.04e-06 - 1.00e+00 1.00e+00F 1  
22 2.4790561e+01 0.00e+00 1.04e+03 -3.1 1.05e+03 - 1.94e-03 2.06e-08f 17  
23 2.4789909e+01 0.00e+00 9.95e+02 -6.6 2.29e-05 - 1.00e+00 5.00e-01f 2  
24 2.4789704e+01 0.00e+00 1.04e+03 -8.5 6.08e-05 - 1.00e+00 6.25e-02f 5  
25 2.4789675e+01 0.00e+00 1.04e+03 -10.4 5.98e-06 - 1.00e+00 1.25e-01f 4  
26 2.4788858e+01 0.00e+00 9.59e+02 -11.0 2.77e-05 - 1.00e+00 5.00e-01f 2  
27 2.4788858e+01 0.00e+00 1.04e+03 -11.0 4.02e-06 - 1.00e+00 5.96e-08f 25  
28 2.4788856e+01 0.00e+00 9.96e+02 -11.0 3.61e-06 - 1.00e+00 2.50e-01f 3  
29 2.4788856e+01 0.00e+00 9.59e+02 -11.0 1.30e-05 - 1.00e+00 9.31e-10f 31  
iter objective inf\_pr inf\_du lg(mu) ||d|| lg(rg) alpha\_du alpha\_pr ls  
30 2.4788855e+01 0.00e+00 1.00e+03 -11.0 3.61e-06 - 1.00e+00 2.50e-01f 3  
31 2.4788855e+01 0.00e+00 4.45e+01 -11.0 1.00e+03 - 1.61e-03 5.93e-18f 49  
32 2.4782395e+01 0.00e+00 9.62e+02 -11.0 4.45e+01 - 3.86e-02 1.95e-06f 15  
33 2.4782395e+01 0.00e+00 1.04e+03 -11.0 8.01e-05 - 1.00e+00 7.45e-09f 28  
34 2.4782395e+01 0.00e+00 1.04e+03 -11.0 2.40e-05 - 1.00e+00 5.68e-14f 45  
35 2.4782395e+01 0.00e+00 1.04e+03 -11.0 2.40e-05 - 1.00e+00 5.68e-14f 45  
36 2.4783762e+01 0.00e+00 1.00e+03 -11.0 2.40e-05 - 1.00e+00 1.00e+00w 1  
37 1.1048089e+03 0.00e+00 1.31e+04 -11.0 1.00e+03 - 6.33e-03 2.63e-03w 1  
In iteration 37, 1 Slack too small, adjusting variable bound  
38 1.1048089e+03 0.00e+00 2.27e+04 -11.0 6.90e+02 - 5.16e-02 3.82e-14w 1  
39 2.4782395e+01 0.00e+00 1.04e+03 -11.0 2.41e+02 - 1.00e+00 5.68e-14f 44  
iter objective inf\_pr inf\_du lg(mu) ||d|| lg(rg) alpha\_du alpha\_pr ls  
40 2.4725235e+01 0.00e+00 8.49e+02 -11.0 4.59e+02 - 1.38e-02 4.49e-05f 8  
41 2.4725235e+01 0.00e+00 8.49e+02 -8.9 8.49e+02 - 2.21e-03 3.24e-21f 60  
42 2.4725235e+01 0.00e+00 8.49e+02 -10.9 8.49e+02 - 2.24e-03 3.27e-21f 60  
43 2.4725235e+01 0.00e+00 8.49e+02 -10.9 8.49e+02 - 2.82e-03 3.27e-21f 60  
44 2.4725235e+01 0.00e+00 8.49e+02 -10.9 8.49e+02 - 5.05e-03 3.27e-21f 60  
45 2.4725235e+01 0.00e+00 8.49e+02 -10.9 8.49e+02 - 4.41e-03 3.27e-21f 60  
46 2.4725235e+01 0.00e+00 8.49e+02 -10.9 8.49e+02 - 3.07e-03 3.27e-21f 60  
47 2.4725235e+01 0.00e+00 8.49e+02 -10.9 8.49e+02 - 1.01e-02 3.27e-21f 60  
48 2.4725235e+01 0.00e+00 8.49e+02 -10.9 8.49e+02 - 2.21e-03 3.27e-21f 60  
49 2.4725235e+01 0.00e+00 8.49e+02 -10.9 8.49e+02 - 6.44e-01 3.27e-21f 60  
iter objective inf\_pr inf\_du lg(mu) ||d|| lg(rg) alpha\_du alpha\_pr ls  
50 2.4725235e+01 0.00e+00 8.49e+02 -10.9 8.49e+02 - 1.81e-03 3.27e-21f 60  
51 2.4725235e+01 0.00e+00 8.49e+02 -10.9 8.49e+02 - 2.95e-03 3.27e-21f 60  
52 4.0000000e+03 0.00e+00 5.00e+07 -10.9 8.49e+02 - 4.67e-03 1.89e-03w 1  
53 4.0000000e+03 0.00e+00 5.00e+07 -10.9 4.92e+00 - 3.89e-12 6.38e-01w 1  
54 3.1203239e+03 0.00e+00 3.38e+02 -10.9 1.57e+00 - 3.14e-11 1.00e+00w 1  
55 4.0000000e+03 0.00e+00 5.00e+07 -10.9 1.06e-05 - 1.89e-03 1.89e-03S 12  
56 3.1183745e+03 0.00e+00 3.15e+02 -10.9 1.33e+00 - 1.44e-11 1.00e+00f 1  
57 3.1183719e+03 0.00e+00 3.15e+02 -10.9 8.35e-06 - 1.00e+00 1.00e+00f 1  
58 3.1183693e+03 0.00e+00 3.15e+02 -10.9 8.35e-06 - 1.00e+00 1.00e+00f 1  
59 3.1183666e+03 0.00e+00 3.15e+02 -10.9 8.35e-06 - 1.00e+00 1.00e+00f 1  
iter objective inf\_pr inf\_du lg(mu) ||d|| lg(rg) alpha\_du alpha\_pr ls  
60 2.7343984e+03 0.00e+00 2.39e+03 -10.9 3.15e+02 - 1.57e-04 2.46e-03f 3  
61 2.6884001e+03 0.00e+00 4.38e+03 -10.9 2.39e+03 - 1.31e-03 4.85e-04f 2  
62 2.6884001e+03 0.00e+00 1.74e+03 -10.9 4.38e+03 - 5.28e-04 2.94e-20f 54  
63 2.1649391e+02 0.00e+00 4.46e+03 -10.9 1.74e+03 - 6.67e-04 2.85e-04f 3  
64 7.8374439e+01 0.00e+00 2.96e+03 -10.9 4.46e+03 - 6.03e-04 2.08e-05f 5  
65 2.6149164e+01 0.00e+00 4.01e+02 -10.9 3.57e-02 - 1.00e+00 1.00e+00f 1  
66 2.5976194e+01 0.00e+00 5.51e+02 -10.9 4.38e-03 - 1.00e+00 5.00e-01f 2  
67 2.5940039e+01 0.00e+00 5.33e+02 -10.9 9.06e-03 - 1.00e+00 2.50e-01f 3  
68 2.5622437e+01 0.00e+00 5.34e+02 -10.9 1.62e-03 - 1.00e+00 1.00e+00f 1  
69 2.5600510e+01 0.00e+00 5.44e+02 -10.9 5.23e-02 - 1.00e+00 3.12e-02f 6  
iter objective inf\_pr inf\_du lg(mu) ||d|| lg(rg) alpha\_du alpha\_pr ls  
70 2.5564467e+01 0.00e+00 5.37e+02 -10.9 5.45e-04 - 1.00e+00 1.00e+00f 1  
71 2.5553100e+01 0.00e+00 5.40e+02 -10.9 2.12e-03 - 1.00e+00 5.00e-01f 2  
72 2.5282679e+01 0.00e+00 3.58e+02 -10.9 1.56e-03 - 1.00e+00 1.00e+00f 1  
73 2.5046210e+01 0.00e+00 8.41e+02 -10.9 9.36e-04 - 1.00e+00 1.00e+00f 1  
74 2.4904612e+01 0.00e+00 5.59e+02 -10.9 1.08e-01 - 1.00e+00 1.56e-02f 7  
75 2.4903474e+01 0.00e+00 5.50e+02 -10.9 1.37e-01 - 1.00e+00 9.77e-04f 11  
76 2.4903474e+01 0.00e+00 5.50e+02 -10.9 2.03e-04 - 1.00e+00 1.42e-14f 47  
77 2.4903474e+01 0.00e+00 5.50e+02 -10.9 2.03e-04 - 1.00e+00 1.42e-14f 47  
78 2.4903474e+01 0.00e+00 5.50e+02 -10.9 2.03e-04 - 1.00e+00 1.42e-14f 47  
79 2.4563135e+01 0.00e+00 6.85e+02 -10.9 5.50e+02 - 4.68e-03 1.50e-05f 10  
iter objective inf\_pr inf\_du lg(mu) ||d|| lg(rg) alpha\_du alpha\_pr ls  
80 2.4552262e+01 0.00e+00 6.46e+02 -11.0 3.56e-03 - 1.00e+00 5.00e-01f 2  
81 2.4552256e+01 0.00e+00 6.15e+02 -11.0 1.08e-03 - 1.00e+00 4.88e-04f 12  
82 2.4550513e+01 0.00e+00 6.63e+02 -11.0 1.05e-04 - 1.00e+00 2.50e-01f 3  
83 2.4530383e+01 0.00e+00 6.09e+02 -11.0 4.06e-04 - 1.00e+00 1.00e+00f 1  
84 2.4524251e+01 0.00e+00 6.15e+02 -11.0 6.75e-02 - 1.00e+00 1.56e-02f 7  
85 2.4524003e+01 0.00e+00 5.99e+02 -11.0 4.23e-04 - 1.00e+00 6.25e-02f 5  
86 2.4524003e+01 0.00e+00 6.15e+02 -11.0 4.45e-03 - 1.00e+00 1.53e-05f 17  
87 2.4523860e+01 0.00e+00 5.99e+02 -11.0 1.96e-05 - 1.00e+00 5.00e-01f 2  
88 2.4523860e+01 0.00e+00 5.91e+02 -11.0 7.14e-06 - 1.00e+00 1.49e-08f 27  
89 2.4523860e+01 0.00e+00 5.74e+02 -11.0 8.49e-06 - 1.00e+00 2.98e-08f 26  
iter objective inf\_pr inf\_du lg(mu) ||d|| lg(rg) alpha\_du alpha\_pr ls  
90 2.4523860e+01 0.00e+00 5.74e+02 -11.0 2.17e-06 - 1.00e+00 7.28e-12f 38  
91 2.4523860e+01 0.00e+00 5.74e+02 -11.0 2.17e-06 - 1.00e+00 7.28e-12f 38  
92 2.4523860e+01 0.00e+00 5.74e+02 -11.0 2.17e-06 - 1.00e+00 7.28e-12f 38  
93 2.4523860e+01 0.00e+00 5.74e+02 -11.0 5.74e+02 - 2.48e-03 8.30e-20f 56  
94 2.4523860e+01 0.00e+00 5.74e+02 -11.0 5.74e+02 - 8.52e-02 2.08e-20f 58  
95 2.4523860e+01 0.00e+00 5.74e+02 -11.0 5.74e+02 - 2.55e-03 2.08e-20f 58  
96 2.4523860e+01 0.00e+00 5.74e+02 -11.0 5.74e+02 - 8.01e-01 2.08e-20f 58  
97 2.4523860e+01 0.00e+00 5.74e+02 -11.0 5.74e+02 - 2.48e-03 2.08e-20f 58  
98 2.4523860e+01 0.00e+00 5.74e+02 -11.0 5.74e+02 - 8.74e-02 2.08e-20f 58  
99 4.5505774e+03 0.00e+00 1.51e+03 -11.0 5.74e+02 - 2.55e-03 2.99e-03w 1  
iter objective inf\_pr inf\_du lg(mu) ||d|| lg(rg) alpha\_du alpha\_pr ls  
100 4.5480459e+03 0.00e+00 1.52e+03 -11.0 6.73e-03 - 1.05e-08 1.00e+00w 1

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [October 10, 2019, 7:26pm UTC](https://discourse.julialang.org/t/how-to-undershand-the-objective-goes-greater-while-itering/29721/5 "2019-10-10T19:26:39Z")

</div>

As mentioned previously, it’s hard to offer advice without code.

Ipopt assumes that functions are differentiable, which doesn’t appear to be the case for your problem. So, the best way to avoid it is to reformulate your problem into a smooth, once (or twice) differentiable form (e.g., use a different penalty function).

You may want to read: [https://projects.coin-or.org/CoinBinary/export/837/CoinAll/trunk/Installer/files/doc/Short%20tutorial%20Ipopt.pdf](https://projects.coin-or.org/CoinBinary/export/837/CoinAll/trunk/Installer/files/doc/Short%20tutorial%20Ipopt.pdf)

---

<div class="post-metadata">

**Author:** ![kuro26](https://avatars.discourse-cdn.com/v4/letter/k/b487fb/32.png) [@kuro26](https://discourse.julialang.org/u/kuro26)\
**Post date:** [October 19, 2019, 2:46pm UTC](https://discourse.julialang.org/t/how-to-undershand-the-objective-goes-greater-while-itering/29721/6 "2019-10-19T14:46:05Z")

</div>

I think I should choose another solver even the time should be longer.Great thanks to your advise again!