# Optim.jl --- Do all Methods Allow Box Constraints? Should all Work Without Them?

**URL:** <https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209>\
**Category:** Optimization (Mathematical)\
**Created:** [April 6, 2018, 10:24pm UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209 "2018-04-06T22:24:53Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [April 6, 2018, 10:24pm UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/1 "2018-04-06T22:24:53Z")

</div>

- I have been experimenting with Optim.jl. Do all optimizers offer box constraints?

- NOTE: All optimizers I tried can work without box constraints, except the brand new SAMIN.

This example failed to use them:

```julia
julia> using Optim

julia> myfun(x)= (x[1]-2)^2 + (x[2]-3)^2 + ((x[1]-2)*(x[2]-3))^2;

julia> optimize(myfun, [0.0, 0.0], fill(6.0, 2), fill(100.0, 2), Optim.Options(iterations=20000))

```

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 6, 2018, 10:34pm UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/2 "2018-04-06T22:34:03Z")

</div>

See `Fminbox`.

[http://julianlsolvers.github.io/Optim.jl/latest/user/minimization/](http://julianlsolvers.github.io/Optim.jl/latest/user/minimization/)

---

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [April 6, 2018, 11:10pm UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/3 "2018-04-06T23:10:38Z")

</div>

thx, chris. I thought I had followed it, but did not look close enough. I wonder whether this is sort of usage buglet, too. See

```julia
julia> optimize(myfun, [0.0, 0.0], fill(6.0, 2), fill(100.0, 2), Optim.Options(iterations=20000))
Results of Optimization Algorithm
 * Algorithm: Nelder-Mead
 * Starting Point: [100.0,100.0]
 * Minimizer: [1.9999890189817244,3.0000111941805336]
 * Minimum: 2.458924e-10
 * Iterations: 58
 * Convergence: true
   * √(Σ(yᵢ-ȳ)²)/n < 1.0e-08: true
   * Reached Maximum Number of Iterations: false
 * Objective Calls: 113

```

above happily seems to work but ignores the box constraints. instead, it should do the same thing as the following.

now, omit the Optim.Options, and I get the correct error message:

```julia
julia> optimize(myfun, [0.0, 0.0], fill(6.0, 2), fill(100.0, 2))
ERROR: Initial position must be inside the box

```

Moreover, with the Options, it can fail altogether. Are `Optim.Options` incompatible with `Fminbox`?

```julia
julia> optimize(myfun, [50.0, 50.0], fill(6.0, 2), fill(100.0, 2), Fminbox{NelderMead}(), Optim.Options(iterations=200000))
ERROR: No default objective type for Optim.Fminbox{Optim.NelderMead}() and (myfun, [50.0, 50.0], [6.0, 6.0]).

```

---

<div class="post-metadata">

**Author:** ![Yifan\_Liu](https://avatars.discourse-cdn.com/v4/letter/y/4da419/32.png) [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)\
**Post date:** [April 7, 2018, 12:23am UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/4 "2018-04-07T00:23:40Z")

</div>

> [@Optim DomainError](https://discourse.julialang.org/t/optim-domainerror/10101):
>
> I am fitting U.S. Kerosene daily price percentage change with Heston Nandi GARCH model. I use the Optim.optimize function for my MLE but got a domain error. My code using RData, Optim function LoadData() return load("Sample.RData")["Sample"] end function HNGARCH(param, sample = Sample) lambda = param[1] omega = param[2] beta = param[3] alpha = param[4] gamma = param[5] delta = param[6] y = sample[:log\_return] r = sample[:rf\_r…

I asked a similar question before.

---

<div class="post-metadata">

**Author:** ![anriseth](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/anriseth/32/3106_2.png) [@anriseth](https://discourse.julialang.org/u/anriseth)\
**Post date:** [April 7, 2018, 7:38am UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/5 "2018-04-07T07:38:51Z")

</div>

Fminbox behaves a bit differently than the unconstrained Optim algorithms. In the future we will hopefully manage to merge these approaches in a more intuitive way. I would recommend that you use it with the first-order optimization algorithms, rather than `NelderMead` etc.

Please read all the instructions under “Box minimization” in the [docs](http://julianlsolvers.github.io/Optim.jl/latest/user/minimization/) carefully.

Regarding the passing of options. In Fminbox, the options to the outer optimization algorithm (which handles the box constraints) are specified with keyword arguments. See the [source code](https://github.com/JuliaNLSolvers/Optim.jl/blob/master/src/multivariate/solvers/constrained/fminbox.jl#L118). Options to the inner optimizer, such as `GradientDescent`, or `LBFGS`, is passed via the keyword argument `optimizer_o`.

To use box constraints with LBFGS, you can do the following. Note that this will calculate derivatives using finite differences. It tells `Fminbox` to run 10 outer iterations, and LBFGS to run 2 iterations for each time it is called.

```nohighlight
f(x)= (x[1]-2)^2 + (x[2]-3)^2 + ((x[1]-2)*(x[2]-3))^2
x0 = fill(50.0, 2)
res = optimize(f, x0, fill(6.0,2), fill(100.0,2), Fminbox{LBFGS}(); show_trace=true, optimizer_o = Optim.Options(show_trace=true))

```

---

<div class="post-metadata">

**Author:** ![pkofod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pkofod/32/2179_2.png) [@pkofod](https://discourse.julialang.org/u/pkofod)\
**Post date:** [April 7, 2018, 11:19am UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/6 "2018-04-07T11:19:24Z")

</div>

To clean up this interface a bit, it might make sense to make this a dedicated function `fminbox(obj, x, lower, upper, inner_method, inner_options, possibly_fminbox_options)` or similar…

---

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [April 7, 2018, 5:23pm UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/7 "2018-04-07T17:23:14Z")

</div>

piping in from the end user perspective, I think it could also be

Options.optim( min=…, max=… )

with intelligence coded into optimize to handle this appropriately.

/iaw

---

<div class="post-metadata">

**Author:** ![pkofod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pkofod/32/2179_2.png) [@pkofod](https://discourse.julialang.org/u/pkofod)\
**Post date:** [April 7, 2018, 5:35pm UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/8 "2018-04-07T17:35:07Z")

</div>

I have a plan, stay tuned 🙂

---

<div class="post-metadata">

**Author:** ![pkofod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pkofod/32/2179_2.png) [@pkofod](https://discourse.julialang.org/u/pkofod)\
**Post date:** [April 10, 2018, 8:28pm UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/9 "2018-04-10T20:28:26Z")

</div>

You can take part in the discussion at [https://github.com/JuliaNLSolvers/Optim.jl/pull/584](https://github.com/JuliaNLSolvers/Optim.jl/pull/584) where I’ve moved some things around. (still wip, but you can hopefully see what I aim for)

---

<div class="post-metadata">

**Author:** ![Johncowk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johncowk/32/8032_2.png) [@Johncowk](https://discourse.julialang.org/u/Johncowk)\
**Post date:** [April 23, 2019, 7:08pm UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/10 "2019-04-23T19:08:31Z")

</div>

What is the correct synthax now ? everything that I try keeps returning weird error messages and the doc isn’t helping.

---

<div class="post-metadata">

**Author:** ![pkofod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pkofod/32/2179_2.png) [@pkofod](https://discourse.julialang.org/u/pkofod)\
**Post date:** [April 23, 2019, 9:29pm UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/11 "2019-04-23T21:29:32Z")

</div>

What did you try? Doesn’t the doc example work?

---

<div class="post-metadata">

**Author:** ![Johncowk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johncowk/32/8032_2.png) [@Johncowk](https://discourse.julialang.org/u/Johncowk)\
**Post date:** [April 24, 2019, 8:32am UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/12 "2019-04-24T08:32:24Z")

</div>

> [@pkofod](#):
>
> What did you try? Doesn’t the doc example work?

The problem (maybe ?) comes from the fact that I have a function with constant parameters so I have to give an input like this :

```julia
result = optimize(params -> cost(params,x,y,folds),start, LBFGS())

```

this works fine (although suuuuper slow), and I thought that giving constraint might speed the process up since I now that the parameters I am trying to optimize are both positive.

I have tried doing like in the docs :

```julia
result = optimize(params -> cost(params,x,y,folds),fill(0,2),fill(Inf,2),start, Fminbox(LBFGS())

```

I also tried without Fminbox, I hve tried the order you discussed in this thread :

```julia
result = optimize(params -> cost(params,x,y,folds),start,fill(0,2),fill(Inf,2), Fminbox(LBFGS()))

```

and I also tried the {} around Fminbox, nothing worked so far.

---

<div class="post-metadata">

**Author:** ![pkofod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pkofod/32/2179_2.png) [@pkofod](https://discourse.julialang.org/u/pkofod)\
**Post date:** [April 24, 2019, 10:24am UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/13 "2019-04-24T10:24:58Z")

</div>

Okay… Probably it’s the `fill(0, 2)` that should be `fill(0.0, 2)`, but without seeing the error it’s impossible for me to help, because I don’t have `cost` either 🙂

---

<div class="post-metadata">

**Author:** ![Johncowk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johncowk/32/8032_2.png) [@Johncowk](https://discourse.julialang.org/u/Johncowk)\
**Post date:** [April 25, 2019, 3:08pm UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/14 "2019-04-25T15:08:48Z")

</div>

Alright, I modified it to `fill(0.0,2)`, and still get the same answer from julia.  
the following syntax

```julia
result = optimize(params -> cost(params,x,y,folds),start,fill(0.0,2),fill(1000.0,2), Fminbox{LBFGS}())

```

returns

```julia
LoadError: MethodError: no method matching Fminbox{LBFGS,T,P} where P where T()
in expression starting at C:\Users\cnelias\Downloads\ChangePointDetection3.jl:71
#lsdd3#554(::Int64, ::Function, ::Array{Float64,1}, ::Array{Float64,1}) at ChangePointDetection3.jl:57
lsdd3(::Array{Float64,1}, ::Array{Float64,1}) at ChangePointDetection3.jl:54
top-level scope at util.jl:156

```

Trying it like in the docs, i.e, without `Fminbox` returns :

```julia
 LoadError: MethodError: objects of type Array{Float64,1} are not callable
Use square brackets [] for indexing an Array.
in expression starting at C:\Users\cnelias\Downloads\ChangePointDetection3.jl:71
(::getfield(NLSolversBase, Symbol("#fg!#8")){getfield(Main, Symbol("##586#587")){Int64,Array{Float64,1},Array{Float64,1}},Array{Float64,1}})(::Array{Float64,1}, ::Array{Float64,1}) at abstract.jl:13
value_gradient!!(::OnceDifferentiable{Float64,Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}) at interface.jl:82
initial_state(::LBFGS{Nothing,LineSearches.InitialStatic{Float64},LineSearches.HagerZhang{Float64,Base.RefValue{Bool}},getfield(Optim, Symbol("##22#24"))}, ::Optim.Options{Float64,Nothing}, ::OnceDifferentiable{Float64,Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}) at l_bfgs.jl:158
optimize(::OnceDifferentiable{Float64,Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::LBFGS{Nothing,LineSearches.InitialStatic{Float64},LineSearches.HagerZhang{Float64,Base.RefValue{Bool}},getfield(Optim, Symbol("##22#24"))}, ::Optim.Options{Float64,Nothing}) at optimize.jl:33
#optimize#89 at interface.jl:130 [inlined]
optimize(::Function, ::Array{Float64,1}, ::Array{Float64,1}, ::Array{Float64,1}, ::LBFGS{Nothing,LineSearches.InitialStatic{Float64},LineSearches.HagerZhang{Float64,Base.RefValue{Bool}},getfield(Optim, Symbol("##22#24"))}, ::Optim.Options{Float64,Nothing}) at interface.jl:128
optimize(::Function, ::Array{Float64,1}, ::Array{Float64,1}, ::Array{Float64,1}, ::LBFGS{Nothing,LineSearches.InitialStatic{Float64},LineSearches.HagerZhang{Float64,Base.RefValue{Bool}},getfield(Optim, Symbol("##22#24"))}) at interface.jl:128
#lsdd3#585(::Int64, ::Function, ::Array{Float64,1}, ::Array{Float64,1}) at ChangePointDetection3.jl:57
lsdd3(::Array{Float64,1}, ::Array{Float64,1}) at ChangePointDetection3.jl:54
top-level scope at util.jl:156

```

The costs function is kinda complicated, should I paste it here ? Overall i think it would be best for me to directly provide the gradient but I don’t think I can find an analytical expression for them…

---

<div class="post-metadata">

**Author:** ![pkofod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pkofod/32/2179_2.png) [@pkofod](https://discourse.julialang.org/u/pkofod)\
**Post date:** [April 25, 2019, 7:10pm UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/15 "2019-04-25T19:10:47Z")

</div>

You are using very old syntax… where are you getting your examples from? The bounds are first and then the initial guess (`start` should be the fourth input), and it’s `Fminbox(LBFGS())` not `Fminbox{LBFGS}()`.

---

<div class="post-metadata">

**Author:** ![Johncowk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johncowk/32/8032_2.png) [@Johncowk](https://discourse.julialang.org/u/Johncowk)\
**Post date:** [April 26, 2019, 6:54am UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/16 "2019-04-26T06:54:24Z")

</div>

Thanks, it worked. It’s still super slow though, I think the only way around here is to provide the gradients but I don’t know if i can manage to compute them.

---

<div class="post-metadata">

**Author:** ![pkofod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pkofod/32/2179_2.png) [@pkofod](https://discourse.julialang.org/u/pkofod)\
**Post date:** [April 26, 2019, 9:20am UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/17 "2019-04-26T09:20:43Z")

</div>

Did you profile it? Where is the time spent?

---

<div class="post-metadata">

**Author:** ![Johncowk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johncowk/32/8032_2.png) [@Johncowk](https://discourse.julialang.org/u/Johncowk)\
**Post date:** [April 29, 2019, 10:48am UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/18 "2019-04-29T10:48:52Z")

</div>

2/3 of the time is spend in Linesearch-related functions

---

<div class="post-metadata">

**Author:** ![pkofod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pkofod/32/2179_2.png) [@pkofod](https://discourse.julialang.org/u/pkofod)\
**Post date:** [April 29, 2019, 11:03am UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/19 "2019-04-29T11:03:33Z")

</div>

try `Fminbox(LBFGS(linesearch=LineSearches.BackTracking()))` and `using LineSearches` first obviously.

---

<div class="post-metadata">

**Author:** ![Johncowk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johncowk/32/8032_2.png) [@Johncowk](https://discourse.julialang.org/u/Johncowk)\
**Post date:** [April 29, 2019, 11:52am UTC](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209/20 "2019-04-29T11:52:37Z")

</div>

It’s better this way (3 times faster) thanks ! But it still is too slow for the application I have in mind. A rough estimation of the parameters with nested for-loops is still more effecient. Most of the time seems to be spend in optimization related functions (Backtracking, manifold, interface).  
Also I get the following warning :

```julia
┌ Warning: Linesearch failed, using alpha = 0.0 and exiting optimization.
│ The linesearch exited with message:
│ Linesearch failed to converge, reached maximum iterations 1000.
└ @ Optim C:\Users\cnelias\.julia\packages\Optim\Agd3B\src\utilities\perform_linesearch.jl:47
# 

```

[Next page](https://discourse.julialang.org/t/optim-jl-do-all-methods-allow-box-constraints-should-all-work-without-them/10209.md?page=2)