# Plot Optim trace/error convergence

**URL:** <https://discourse.julialang.org/t/plot-optim-trace-error-convergence/32632>\
**Category:** Optimization (Mathematical)\
**Tags:** question, plotting\
**Created:** [December 23, 2019, 7:27pm UTC](https://discourse.julialang.org/t/plot-optim-trace-error-convergence/32632 "2019-12-23T19:27:18Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Andre\_Mello](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andre_mello/32/7333_2.png) [@Andre\_Mello](https://discourse.julialang.org/u/Andre_Mello)\
**Post date:** [December 23, 2019, 7:27pm UTC](https://discourse.julialang.org/t/plot-optim-trace-error-convergence/32632/1 "2019-12-23T19:27:18Z")

</div>

Hi, all.

I’m using `Optim.jl` to solve an optimization problem and I want to plot its trace/error convergence. However I have some issues and I couldn’t find much info on that so far.

Just to make things clearer, I want something like this:

![image](https://global.discourse-cdn.com/julialang/original/3X/e/d/ed682016fc58b17edbea422feb56f1b908543399.png)

So my first problem is: I’m dealing with a box-constrained optimization problem. I’m calling the `optimize` function as follows

`res = opt.optimize(od, lb, ub, p0, opt.Fminbox(inner_optimizer), Grad_options)`

and it seems that the `store_trace` parameter is not available for this kind of optimization. Is there any way around this?

* * *

The other thing is, if I get rid of the box constraints I can do something along the lines of

```julia
import Optim
const opt = Optim

res = opt.optimize((x-> sum(x.^2)), [100.0, 200.0], store_trace=true)
trace = opt.trace(res)

trace_err = []
trace_time = []
for i in 1:length(trace)
    append!(trace_err, parse(Float64, split(string(trace[i]))[2]))
    append!(trace_time, parse(Float64, split(string(trace[i]))[end]))
end
plot(log10.(trace_time), log10.(trace_err/trace_err[end]))

```

and then I get back something close to what I want.

![image](https://global.discourse-cdn.com/julialang/original/3X/4/f/4fb0f32138f82eca68b45a9c7d3cf649432c12a5.png)

But it really feels like I’m hacking my way through this, and not in a good way. Is there any standard way of plotting the trace?

---

<div class="post-metadata">

**Author:** ![antoine-levitt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antoine-levitt/32/4008_2.png) [@antoine-levitt](https://discourse.julialang.org/u/antoine-levitt)\
**Post date:** [December 23, 2019, 8:21pm UTC](https://discourse.julialang.org/t/plot-optim-trace-error-convergence/32632/2 "2019-12-23T20:21:55Z")

</div>

Have you tried with `Optim.Options`?

---

<div class="post-metadata">

**Author:** ![Andre\_Mello](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andre_mello/32/7333_2.png) [@Andre\_Mello](https://discourse.julialang.org/u/Andre_Mello)\
**Post date:** [December 23, 2019, 8:32pm UTC](https://discourse.julialang.org/t/plot-optim-trace-error-convergence/32632/3 "2019-12-23T20:32:29Z")

</div>

Oh, thanks @antoine-levitt, this totally went over my head!

I’m just left looking for a better way to plot these results.

---

<div class="post-metadata">

**Author:** ![Andre\_Mello](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andre_mello/32/7333_2.png) [@Andre\_Mello](https://discourse.julialang.org/u/Andre_Mello)\
**Post date:** [December 23, 2019, 9:02pm UTC](https://discourse.julialang.org/t/plot-optim-trace-error-convergence/32632/4 "2019-12-23T21:02:32Z")

</div>

Actually, now I’m not so sure output the trace’s output.

I’ve just got this back when printing the trace:

```bash
Iter Function value Gradient norm 
------ -------------- --------------
     0 2.260522e-01 3.247800e-01
 * time: 1.1920928955078125e-6
     1 8.766649e-02 2.633301e-01
 * time: 0.01271200180053711
     2 1.753852e-02 1.377736e-01
 * time: 0.025365114212036133
     3 3.277411e-03 1.703786e-02
 * time: 0.03435707092285156
     4 1.347663e-03 1.195144e-02
 * time: 0.04666304588317871
     5 1.280119e-03 1.866732e-04
 * time: 0.05624198913574219
     6 1.279841e-03 7.308482e-06
 * time: 0.0635230541229248
     7 1.279841e-03 4.129017e-06
 * time: 0.07249212265014648
     0 2.816947e-03 2.614618e-04
 * time: 9.5367431640625e-7
     1 2.816918e-03 6.837274e-06
 * time: 0.0038449764251708984
     2 2.816917e-03 6.098678e-06
 * time: 0.015560150146484375
     3 2.816917e-03 2.112449e-06
 * time: 0.02004694938659668
     4 2.816917e-03 2.664522e-06
 * time: 0.023893117904663086
     0 2.818454e-03 2.366669e-06
 * time: 1.9073486328125e-6
     1 2.818454e-03 6.654087e-07
 * time: 0.004698991775512695

```

Why does it go back to iteration 0 after iteration 7, and then back to zero again after the second 4th iteration?

---

<div class="post-metadata">

**Author:** ![bashonubuntu](https://avatars.discourse-cdn.com/v4/letter/b/f19dbf/32.png) [@bashonubuntu](https://discourse.julialang.org/u/bashonubuntu)\
**Post date:** [December 23, 2019, 9:11pm UTC](https://discourse.julialang.org/t/plot-optim-trace-error-convergence/32632/6 "2019-12-23T21:11:13Z")

</div>

Can you show how you are using `Optim.Options()`?

---

<div class="post-metadata">

**Author:** ![Andre\_Mello](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andre_mello/32/7333_2.png) [@Andre\_Mello](https://discourse.julialang.org/u/Andre_Mello)\
**Post date:** [December 23, 2019, 9:16pm UTC](https://discourse.julialang.org/t/plot-optim-trace-error-convergence/32632/7 "2019-12-23T21:16:20Z")

</div>

Sure, here it goes:

`Grad_options = opt.Options(x_tol=10^-6, f_tol=10^-12, iterations=10^4, store_trace=true)`

---

<div class="post-metadata">

**Author:** ![pkofod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pkofod/32/2179_2.png) [@pkofod](https://discourse.julialang.org/u/pkofod)\
**Post date:** [December 23, 2019, 10:28pm UTC](https://discourse.julialang.org/t/plot-optim-trace-error-convergence/32632/8 "2019-12-23T22:28:34Z")

</div>

It’s because you’re seeing the output of the inner trace. I have never gotten around to it, but you have to be aware that what you’re seeing is not really the objective itself for box constrained optimization, it includes a penalty term. Is your objective expensive to evaluate?

---

<div class="post-metadata">

**Author:** ![Andre\_Mello](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andre_mello/32/7333_2.png) [@Andre\_Mello](https://discourse.julialang.org/u/Andre_Mello)\
**Post date:** [December 24, 2019, 7:44pm UTC](https://discourse.julialang.org/t/plot-optim-trace-error-convergence/32632/9 "2019-12-24T19:44:20Z")

</div>

Yes, it can get a bit expensive.

But I just came across this tip using a callback function, maybe I could do something with that.

[https://github.com/JuliaNLSolvers/Optim.jl/blob/master/docs/src/user/tipsandtricks.md#early-stopping](https://github.com/JuliaNLSolvers/Optim.jl/blob/master/docs/src/user/tipsandtricks.md#early-stopping)

But then I guess I’d have to use some global variables to store the time and function evaluation results for each iteration, right?

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [December 24, 2019, 8:16pm UTC](https://discourse.julialang.org/t/plot-optim-trace-error-convergence/32632/10 "2019-12-24T20:16:33Z")

</div>

For example of using callback for plotting:  
TensorBoardLogger.jl has an example for Optim

[https://philipvinc.github.io/TensorBoardLogger.jl/dev/examples/optim/](https://philipvinc.github.io/TensorBoardLogger.jl/dev/examples/optim/)

Which if you wanted to use TensorBoard (which is not a bad choice for this kind of thing),  
you could adapt directly.

---

<div class="post-metadata">

**Author:** ![Andre\_Mello](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andre_mello/32/7333_2.png) [@Andre\_Mello](https://discourse.julialang.org/u/Andre_Mello)\
**Post date:** [January 7, 2020, 6:43pm UTC](https://discourse.julialang.org/t/plot-optim-trace-error-convergence/32632/11 "2020-01-07T18:43:46Z")

</div>

Oh I hadn’t heard of TensorBoardLogger before and this might be what I’m looking for.  
Thanks for pointing it out, I’ll give it a try! 🙂