# Nonlinear optimization help. Optim, JuMP, something else?

I have a nonlinear optimization problem that I’ve tried solving with scipy. For small scale it (the ‘TNC’ option in scipy minimize, specifically) works well, but it’s too slow when I get to thousands of variables. I’m learning Julia to see if I can find something better here.

Here is an example of a small scale version of the problem:

(Using `ComponentArrays` for convenience)

``````
k_unr_out = [ 0.,  3.,  1.,  5.,  2.,  4.,  5.,  0.,  0.,  2.,  0.,  4.,  5.,
6.,  1., 22.,  0.,  1., 19.,  0.,  0.,  7.,  0.,  4.,  0.,  6.]

k_unr_in = [ 7.,  4.,  9.,  3.,  2.,  3.,  4., 11.,  4.,  4.,  5.,  4.,  3.,
3.,  3.,  0.,  2.,  6.,  0.,  3.,  0.,  1.,  8.,  2.,  3.,  3.]

k_recip = [ 1.,  8.,  2.,  8.,  3.,  7.,  6.,  1.,  1.,  7.,  4.,  5.,  4.,
2.,  9.,  3.,  2.,  6.,  6.,  2., 25.,  6.,  4.,  3.,  3.,  6.]

N = length(k_unr_out)

v0 = ComponentArray(x=repeat([0.1], N), y=repeat([0.1], N), z=repeat([0.1], N))

function neg_llhood(v, k_unr_out, k_unr_in, k_recip)
llhood = sum(k_unr_out .* log.(v.x))
llhood += sum(k_unr_in .* log.(v.y))
llhood += sum(k_recip .* log.(v.z))

xy = v.x .* v.y'
zz = v.z .* v.z'

Q = log.(1 .+ xy + xy' + zz)

llhood -= sum(tril(Q, -1))

-llhood
end

f = v -> neg_llhood(v, k_unr_out, k_unr_in, k_recip)
``````

Since logarithms are taken for everything, my only constraints are `x_i, y_i, z_i > 0 for all i`. I’d like to use Optim to solve the above but I’m not sure exactly how to do that.

In scipy I was using JAX to do auto differentiation so I’m trying to do that here as well (I could compute the gradient analytically if I need to tho). For box constraints the docs for Optim give this example,

``````lower = [1.25, -2.1]
upper = [Inf, Inf]
initial_x = [2.0, 2.0]
results = optimize(f, g!, lower, upper, initial_x, Fminbox(inner_optimizer))
``````

I’m confused by the docs. What is `Fminbox` doing? What is an inner optimizer? How do I specify that I want to use AD?

I tried

``````lower = repeat([floatmin()], 3*N)
upper = repeat([Inf], 3*N)

results = optimize(f, lower, upper, v0, Fminbox(inner_optimizer); autodiff=:forward)
``````

And I get

``````MethodError: no method matching ldiv!(::ComponentVector{Float64}, ::Optim.InverseDiagonal, ::ComponentVector{Float64})
Closest candidates are:
...
``````

I’m not sure if JuMP + a solver is preferred for my given problem. I gave it a try

``````model = Model(Ipopt.Optimizer)

@variable(model, x[1:N] >= floatmin(), start=0.1)
@variable(model, y[1:N] >= floatmin(), start=0.1)
@variable(model, z[1:N] >= floatmin(), start=0.1)

@NLobjective(model, Max,
sum(k_unr_out[i] * log(x[i]) + k_unr_in[i] * log(y[i]) + k_recip[i] * log(z[i]) for i in 1:N)
- sum(log(1 + x[i] * y[j] + x[j] * y[i] + z[i] * z[j]) for i in 1:N for j in i+1:N))

optimize!(model)
``````

This causes my Jupyter kernel to crash.

I’m a bit lost in how to solve this problem most efficiently. Any help appreciated.

1 Like

JuMP + Ipopt works well for me. Note that you need small positive lower bounds on the variables to avoid `log(0)`.

``````using JuMP
using Ipopt
k_unr_out = [ 0.,  3.,  1.,  5.,  2.,  4.,  5.,  0.,  0.,  2.,  0.,  4.,  5.,
6.,  1., 22.,  0.,  1., 19.,  0.,  0.,  7.,  0.,  4.,  0.,  6.]
k_unr_in = [ 7.,  4.,  9.,  3.,  2.,  3.,  4., 11.,  4.,  4.,  5.,  4.,  3.,
3.,  3.,  0.,  2.,  6.,  0.,  3.,  0.,  1.,  8.,  2.,  3.,  3.]
k_recip = [ 1.,  8.,  2.,  8.,  3.,  7.,  6.,  1.,  1.,  7.,  4.,  5.,  4.,
2.,  9.,  3.,  2.,  6.,  6.,  2., 25.,  6.,  4.,  3.,  3.,  6.]
N = length(k_unr_out)

model = Model(Ipopt.Optimizer)
@variable(model, x[1:N] >= 0.001, start = 0.1)
@variable(model, y[1:N] >= 0.001, start = 0.1)
@variable(model, z[1:N] >= 0.001, start = 0.1)
@NLobjective(
model,
Max,
sum(
k_unr_out[i] * log(x[i]) +
k_unr_in[i] * log(y[i]) +
k_recip[i] * log(z[i])
for i in 1:N
)
- sum(
log(1 + x[i] * y[j] + x[j] * y[i] + z[i] * z[j])
for i = 1:N, j = (i + 1):N
)
)

julia> optimize!(model)
This is Ipopt version 3.13.2, running with linear solver mumps.
NOTE: Other linear solvers might be more efficient (see Ipopt documentation).

Number of nonzeros in equality constraint Jacobian...:        0
Number of nonzeros in inequality constraint Jacobian.:        0
Number of nonzeros in Lagrangian Hessian.............:     3081

Total number of variables............................:       78
variables with only lower bounds:       78
variables with lower and upper bounds:        0
variables with only upper bounds:        0
Total number of equality constraints.................:        0
Total number of inequality constraints...............:        0
inequality constraints with only lower bounds:        0
inequality constraints with lower and upper bounds:        0
inequality constraints with only upper bounds:        0

iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
0  7.6485452e+02 0.00e+00 1.01e+02  -1.0 0.00e+00    -  0.00e+00 0.00e+00   0
1  5.6576765e+02 0.00e+00 4.92e+01  -1.0 9.82e-02    -  1.00e+00 1.00e+00f  1
2  4.1243350e+02 0.00e+00 2.38e+01  -1.0 1.89e-01    -  1.00e+00 1.00e+00f  1
3  3.2077731e+02 0.00e+00 1.08e+01  -1.0 3.35e-01    -  7.39e-01 1.00e+00f  1
4  2.7086780e+02 0.00e+00 5.04e+00  -1.0 5.72e-01    -  1.00e+00 1.00e+00f  1
5  2.4293161e+02 0.00e+00 2.26e+00  -1.0 9.31e-01    -  8.64e-01 1.00e+00f  1
6  2.2531364e+02 0.00e+00 1.08e+00  -1.0 1.57e+00    -  1.00e+00 1.00e+00f  1
7  2.1391224e+02 0.00e+00 4.79e-01  -1.0 2.56e+00    -  9.21e-01 1.00e+00f  1
8  2.0369603e+02 0.00e+00 1.18e+00  -1.7 4.23e+00    -  7.79e-01 8.83e-01f  1
9  1.9855275e+02 0.00e+00 4.09e-01  -1.7 6.47e+00    -  1.00e+00 1.00e+00f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
10  1.9484320e+02 0.00e+00 8.00e-01  -1.7 1.05e+01    -  1.00e+00 1.00e+00f  1
11  1.9256323e+02 0.00e+00 4.34e-01  -1.7 1.67e+01    -  1.00e+00 1.00e+00f  1
12  1.9100012e+02 0.00e+00 4.82e-01  -1.7 2.70e+01    -  1.00e+00 1.00e+00f  1
13  1.8989596e+02 0.00e+00 5.62e-01  -1.7 4.30e+01    -  1.00e+00 1.00e+00f  1
14  1.8905100e+02 0.00e+00 5.65e-01  -1.7 6.99e+01    -  1.00e+00 1.00e+00f  1
15  1.8847647e+02 0.00e+00 6.14e-01  -1.7 1.12e+02    -  1.00e+00 1.00e+00f  1
16  1.8812660e+02 0.00e+00 8.63e-01  -1.7 1.90e+02    -  7.43e-01 1.00e+00f  1
17  1.8811339e+02 0.00e+00 6.44e-02  -1.7 4.18e-01  -2.0 1.00e+00 1.00e+00f  1
18  1.8777458e+02 0.00e+00 1.63e+01  -2.5 4.59e+02    -  3.37e-01 1.02e-01f  1
19  1.8726075e+02 0.00e+00 2.49e+00  -2.5 2.80e+02    -  7.58e-01 3.61e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
20  1.8694821e+02 0.00e+00 1.47e+00  -2.5 3.88e+02    -  1.00e+00 1.00e+00f  1
21  1.8671781e+02 0.00e+00 8.31e-01  -2.5 5.85e+02    -  1.00e+00 1.00e+00f  1
22  1.8661940e+02 0.00e+00 8.84e-01  -2.5 9.77e+02    -  1.00e+00 1.00e+00f  1
23  1.8639436e+02 0.00e+00 9.21e-01  -2.5 1.56e+03    -  1.00e+00 1.00e+00f  1
24  1.8639424e+02 0.00e+00 9.64e-02  -2.5 1.24e-01  -2.5 1.00e+00 1.00e+00f  1
25  1.8639322e+02 0.00e+00 1.30e-03  -2.5 1.70e-01  -3.0 1.00e+00 1.00e+00f  1
26  1.8631537e+02 0.00e+00 2.99e-02  -3.8 9.35e-01  -3.4 1.00e+00 9.26e-01f  1
27  1.8630837e+02 0.00e+00 4.02e-04  -3.8 1.85e+00  -3.9 1.00e+00 1.00e+00f  1
28  1.8630134e+02 0.00e+00 4.08e-03  -5.7 4.22e+00  -4.4 1.00e+00 9.75e-01f  1
29  1.8629600e+02 0.00e+00 4.84e-03  -5.7 9.08e+00  -4.9 1.00e+00 1.00e+00f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
30  1.8628450e+02 0.00e+00 2.19e-02  -5.7 2.84e+01  -5.3 1.00e+00 1.00e+00f  1
31  1.8625887e+02 0.00e+00 1.18e-01  -5.7 7.65e+01  -5.8 1.00e+00 1.00e+00f  1
32  1.8623095e+02 0.00e+00 5.00e+00  -5.7 2.68e+03    -  1.00e+00 1.91e-01f  2
33  1.8614046e+02 0.00e+00 1.53e+00  -5.7 2.99e+03    -  7.96e-01 1.00e+00f  1
34  1.8604107e+02 0.00e+00 1.06e+00  -5.7 4.48e+03    -  1.00e+00 1.00e+00f  1
35  1.8602667e+02 0.00e+00 4.98e+00  -5.7 7.32e+03    -  9.05e-01 2.41e-01f  2
36  1.8602667e+02 0.00e+00 8.77e-01  -5.7 6.79e-01  -0.9 1.00e+00 2.57e-05f  2
37  1.8595026e+02 0.00e+00 1.20e+00  -5.7 8.49e+03    -  1.00e+00 1.00e+00f  1
38  1.8591162e+02 0.00e+00 1.13e+00  -5.7 1.35e+04    -  1.00e+00 1.00e+00f  1
39  1.8590020e+02 0.00e+00 9.40e-01  -5.7 2.16e+04    -  1.00e+00 2.60e-01f  2
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
40  1.8585862e+02 0.00e+00 9.98e-01  -5.7 2.54e+04    -  1.00e+00 8.11e-01f  1
41  1.8583513e+02 0.00e+00 2.32e+00  -5.7 3.36e+04    -  1.00e+00 6.81e-01f  1
42  1.8581185e+02 0.00e+00 1.10e+00  -5.7 4.47e+04    -  1.00e+00 1.00e+00f  1
43  1.8580976e+02 0.00e+00 1.42e+00  -5.7 6.95e+04    -  1.00e+00 7.43e-02f  1
44  1.8579368e+02 0.00e+00 5.05e-01  -5.7 6.90e+04    -  1.00e+00 1.00e+00f  1
45  1.8577710e+02 0.00e+00 3.51e-01  -5.7 1.02e+05    -  1.00e+00 9.40e-01f  1
46  1.8577088e+02 0.00e+00 4.80e-01  -5.7 1.50e+05    -  1.00e+00 5.00e-01f  2
47  1.8575853e+02 0.00e+00 1.75e-01  -5.7 1.87e+05    -  1.00e+00 1.00e+00f  1
48  1.8575842e+02 0.00e+00 1.84e+00  -5.7 2.80e+05    -  7.97e-01 1.14e-01f  2
49  1.8575799e+02 0.00e+00 6.08e-02  -5.7 4.10e-02  -1.4 1.00e+00 1.19e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
50  1.8575700e+02 0.00e+00 6.57e-01  -5.7 2.96e+05    -  5.99e-01 5.00e-01f  2
51  1.8574447e+02 0.00e+00 3.73e-02  -5.7 2.48e-02  -1.8 1.00e+00 1.00e+00f  1
52  1.8574444e+02 0.00e+00 2.48e-04  -5.7 7.59e-03  -2.3 1.00e+00 1.00e+00f  1
53  1.8574444e+02 0.00e+00 6.66e-06  -5.7 4.05e-03  -2.8 1.00e+00 1.00e+00f  1
54  1.8574437e+02 0.00e+00 1.05e-05  -8.6 1.91e-02  -3.3 1.00e+00 1.00e+00f  1
55  1.8574437e+02 0.00e+00 6.65e-06  -8.6 3.64e-02  -3.7 1.00e+00 8.93e-01f  1
56  1.8574437e+02 0.00e+00 6.65e-06  -8.6 1.09e-01  -4.2 1.00e+00 1.00e+00f  1
57  1.8574436e+02 0.00e+00 6.65e-06  -8.6 3.28e-01  -4.7 1.00e+00 1.00e+00f  1
58  1.8574434e+02 0.00e+00 6.64e-06  -8.6 9.82e-01  -5.2 1.00e+00 1.00e+00f  1
59  1.8574429e+02 0.00e+00 9.44e-06  -8.6 2.94e+00  -5.6 1.00e+00 1.00e+00f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
60  1.8574414e+02 0.00e+00 8.62e-05  -8.6 8.77e+00  -6.1 1.00e+00 1.00e+00f  1
61  1.8574368e+02 0.00e+00 7.90e-04  -8.6 2.61e+01  -6.6 1.00e+00 1.00e+00f  1
62  1.8574237e+02 0.00e+00 7.18e-03  -8.6 7.67e+01  -7.1 1.00e+00 1.00e+00f  1
63  1.8574128e+02 0.00e+00 2.76e+00  -8.6 1.27e+06    -  7.86e-02 4.23e-03f  3
64  1.8574128e+02 0.00e+00 9.99e-01  -8.6 1.18e+00  -4.8 1.00e+00 9.54e-07f 21
65  1.8573645e+02 0.00e+00 2.18e-01  -8.6 3.72e+05    -  1.00e+00 5.00e-01f  2
66  1.8573065e+02 0.00e+00 1.43e-01  -8.6 4.65e+05    -  1.00e+00 1.00e+00f  1
67  1.8573054e+02 0.00e+00 1.80e-03  -8.6 8.09e-01  -5.3 1.00e+00 1.00e+00f  1
68  1.8573050e+02 0.00e+00 6.63e-06  -8.6 2.30e+00  -5.8 1.00e+00 1.00e+00f  1
69  1.8573039e+02 0.00e+00 7.46e-05  -8.6 7.32e+00  -6.3 1.00e+00 1.00e+00f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
70  1.8573009e+02 0.00e+00 6.04e-04  -8.6 2.34e+01  -6.8 1.00e+00 1.00e+00f  1
71  1.8572923e+02 0.00e+00 5.13e-03  -8.6 7.18e+01  -7.2 1.00e+00 1.00e+00f  1
72  1.8572730e+02 0.00e+00 3.99e-02  -8.6 1.95e+02  -7.7 1.00e+00 1.00e+00f  1
73  1.8572321e+02 0.00e+00 4.60e-01  -8.6 6.98e+05    -  1.00e+00 1.00e+00f  1
74  1.8572026e+02 0.00e+00 3.78e-02  -8.6 2.02e+02  -8.2 1.00e+00 1.00e+00f  1
75  1.8571858e+02 0.00e+00 4.89e-01  -8.6 1.05e+06    -  1.00e+00 5.00e-01f  2
76  1.8571814e+02 0.00e+00 1.77e-01  -8.6 1.94e+06    -  6.10e-02 3.99e-02f  5
77  1.8571814e+02 0.00e+00 1.77e-01  -8.6 3.88e-02  -3.2 1.00e+00 2.67e-06f  2
78  1.8571379e+02 0.00e+00 1.00e-01  -8.6 1.33e+06    -  1.00e+00 1.00e+00f  1
79  1.8571377e+02 0.00e+00 6.51e-04  -8.6 6.78e-03  -3.7 1.00e+00 1.00e+00f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
80  1.8571377e+02 0.00e+00 1.28e-06  -8.6 2.03e-02  -4.2 1.00e+00 1.00e+00f  1
81  1.8571377e+02 0.00e+00 1.28e-06  -8.6 6.08e-02  -4.7 1.00e+00 1.00e+00f  1
82  1.8571377e+02 0.00e+00 1.28e-06  -8.6 1.82e-01  -5.2 1.00e+00 1.00e+00f  1
83  1.8571377e+02 0.00e+00 1.27e-06  -8.6 5.40e-01  -5.6 1.00e+00 1.00e+00f  1
84  1.8571376e+02 0.00e+00 1.23e-06  -8.6 1.58e+00  -6.1 1.00e+00 1.00e+00f  1
85  1.8571373e+02 0.00e+00 6.69e-06  -8.6 4.40e+00  -6.6 1.00e+00 1.00e+00f  1
86  1.8571366e+02 0.00e+00 5.67e-05  -8.6 1.40e+01  -7.1 1.00e+00 1.00e+00f  1
87  1.8571347e+02 0.00e+00 4.69e-04  -8.6 4.51e+01  -7.5 1.00e+00 1.00e+00f  1
88  1.8571292e+02 0.00e+00 4.03e-03  -8.6 1.41e+02  -8.0 1.00e+00 1.00e+00f  1
89  1.8571152e+02 0.00e+00 3.45e-02  -8.6 4.04e+02  -8.5 1.00e+00 1.00e+00f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
90  1.8571061e+02 0.00e+00 2.62e-01  -8.6 2.00e+06    -  1.00e+00 2.44e-01f  3
91  1.8570871e+02 0.00e+00 3.01e-01  -8.6 2.24e+06    -  8.43e-01 5.00e-01f  2
92  1.8570690e+02 0.00e+00 1.63e-02  -8.6 3.66e+02  -9.0 1.00e+00 1.00e+00f  1
93  1.8570654e+02 0.00e+00 5.92e-01  -8.6 2.80e+06    -  1.00e+00 5.00e-01f  2
94  1.8570353e+02 0.00e+00 2.87e-02  -8.6 3.18e+02  -9.4 1.00e+00 1.00e+00f  1
95  1.8570255e+02 0.00e+00 4.12e-01  -8.6 3.50e+06    -  1.00e+00 2.50e-01f  3
96  1.8570088e+02 0.00e+00 2.40e-02  -8.6 4.11e+02  -9.9 1.00e+00 1.00e+00f  1
97  1.8570077e+02 0.00e+00 8.22e-01  -8.6 3.94e+06    -  1.00e+00 5.00e-01f  2
98  1.8569813e+02 0.00e+00 4.93e-02  -8.6 5.24e+02 -10.4 1.00e+00 1.00e+00f  1
99  1.8569757e+02 0.00e+00 6.06e-01  -8.6 4.92e+06    -  8.40e-01 2.50e-01f  3
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
100  1.8569598e+02 0.00e+00 3.60e-02  -8.6 5.85e+02 -10.9 1.00e+00 1.00e+00f  1
101  1.8569521e+02 0.00e+00 3.21e-01  -8.6 5.53e+06    -  1.00e+00 2.50e-01f  3
102  1.8569435e+02 0.00e+00 6.44e-02  -8.6 8.65e+02 -11.4 1.00e+00 1.00e+00f  1
103  1.8569397e+02 0.00e+00 1.14e+00  -8.6 6.22e+06    -  1.00e+00 1.00e+00f  1
104  1.8569213e+02 0.00e+00 1.13e-01  -8.6 1.42e+03 -11.8 1.00e+00 1.00e+00f  1
105  1.8569157e+02 0.00e+00 5.27e-01  -8.6 9.28e+06    -  5.92e-01 2.50e-01f  3
106  1.8569073e+02 0.00e+00 3.97e-02  -8.6 1.26e+03 -12.3 1.00e+00 1.00e+00f  1
107  1.8569041e+02 0.00e+00 7.81e-01  -8.6 1.04e+07    -  1.00e+00 5.00e-01f  2
108  1.8568944e+02 0.00e+00 5.65e-02  -8.6 1.87e+03 -12.8 1.00e+00 1.00e+00f  1
109  1.8568905e+02 0.00e+00 3.04e-01  -8.6 1.29e+07    -  8.79e-01 2.50e-01f  3
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
110  1.8568856e+02 0.00e+00 9.86e-02  -8.6 2.35e+03 -13.3 1.00e+00 1.00e+00f  1
111  1.8568823e+02 0.00e+00 1.10e+00  -8.6 1.45e+07    -  1.00e+00 1.00e+00f  1
112  1.8568744e+02 0.00e+00 1.43e-01  -8.6 1.65e+04 -13.7 1.00e+00 1.00e+00f  1
113  1.8568704e+02 0.00e+00 3.77e-01  -8.6 2.11e+07    -  4.74e-01 2.50e-01f  3
114  1.8568650e+02 0.00e+00 2.63e-02  -8.6 2.41e+03 -14.2 1.00e+00 1.00e+00f  1
115  1.8568627e+02 0.00e+00 3.93e-01  -8.6 2.34e+07    -  1.00e+00 2.50e-01f  3
116  1.8568595e+02 0.00e+00 1.46e-01  -8.6 1.88e+04 -14.7 1.00e+00 1.00e+00f  1
117  1.8568567e+02 0.00e+00 1.18e+00  -8.6 2.59e+07    -  1.00e+00 1.00e+00f  1
118  1.8568555e+02 0.00e+00 2.92e-01  -8.6 1.43e+05 -15.2 1.00e+00 1.00e+00f  1
119  1.8568463e+02 0.00e+00 9.65e-03  -8.6 2.06e+04 -15.6 1.00e+00 1.00e+00f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
120  1.8568443e+02 0.00e+00 2.52e-01  -8.6 3.51e+07    -  1.00e+00 1.25e-01f  4
121  1.8568421e+02 0.00e+00 1.56e-01  -8.6 5.86e+04 -16.1 1.00e+00 1.00e+00f  1
122  1.8568404e+02 0.00e+00 1.24e+00  -8.6 3.64e+07    -  1.00e+00 1.00e+00f  1
123  1.8568377e+02 0.00e+00 3.76e-01  -8.6 2.71e+06 -16.6 1.00e+00 5.00e-01f  2
124  1.8568332e+02 0.00e+00 3.59e-01  -8.6 4.28e+07    -  1.00e+00 1.00e+00f  1
125  1.8568316e+02 0.00e+00 3.17e-01  -8.6 4.66e+06 -17.1 1.00e+00 9.92e-02f  4
126  1.8568296e+02 0.00e+00 1.24e-01  -8.6 1.06e+06 -17.6 1.00e+00 1.00e+00f  1
127  1.8568273e+02 0.00e+00 2.11e-01  -8.6 2.51e+06 -18.0 1.00e+00 1.00e+00f  1
128  1.8568266e+02 0.00e+00 2.35e-01  -8.6 1.10e+07 -18.5 1.00e+00 5.00e-01f  2
129  1.8568259e+02 0.00e+00 6.94e-01  -8.6 4.74e+07 -18.1 6.62e-01 2.28e-01f  3
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
130  1.8568234e+02 0.00e+00 1.64e-01  -8.6 1.38e+07 -18.6 1.00e+00 5.00e-01f  2
131  1.8568225e+02 0.00e+00 7.55e-01  -8.6 3.20e+07 -19.0 1.00e+00 5.00e-01f  2
132  1.8568202e+02 0.00e+00 2.00e-01  -8.6 3.09e+06 -17.7 1.00e+00 1.00e+00f  1
133  1.8568192e+02 0.00e+00 6.66e-01  -8.6 6.77e+06 -18.2 1.00e+00 1.00e+00f  1
134  1.8568180e+02 0.00e+00 2.57e-01  -8.6 4.27e+07 -18.7 1.00e+00 2.26e-01f  1
135  1.8568164e+02 0.00e+00 1.85e-01  -8.6 3.75e+07    -  1.00e+00 1.00e+00f  1
136  1.8568159e+02 0.00e+00 6.35e-02  -8.6 3.20e+07    -  1.00e+00 1.00e+00f  1
137  1.8568156e+02 0.00e+00 1.50e-01  -8.6 3.57e+07    -  1.00e+00 8.68e-01f  1
138  1.8568155e+02 0.00e+00 2.96e-02  -8.6 5.01e+07    -  1.00e+00 1.00e+00f  1
139  1.8568154e+02 0.00e+00 9.85e-02  -8.6 4.35e+07    -  1.00e+00 1.00e+00f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
140  1.8568153e+02 0.00e+00 8.67e-03  -8.6 3.15e+07    -  1.00e+00 1.00e+00f  1
141  1.8568153e+02 0.00e+00 3.18e-03  -8.6 1.17e+07    -  1.00e+00 1.00e+00f  1
142  1.8568153e+02 0.00e+00 4.62e-05  -8.6 1.80e+06    -  1.00e+00 1.00e+00f  1
143  1.8568153e+02 0.00e+00 2.05e-08  -8.6 3.60e+04    -  1.00e+00 1.00e+00f  1
144  1.8568152e+02 0.00e+00 1.00e-03  -9.0 8.35e+07    -  1.00e+00 1.00e+00f  1
145  1.8568151e+02 0.00e+00 5.92e-05  -9.0 6.38e+07    -  1.00e+00 1.00e+00f  1
146  1.8568151e+02 0.00e+00 9.32e-05  -9.0 2.35e+07    -  1.00e+00 1.00e+00f  1
147  1.8568151e+02 0.00e+00 9.82e-07  -9.0 2.22e+06    -  1.00e+00 1.00e+00f  1
148  1.8568151e+02 0.00e+00 2.38e-10  -9.0 1.72e+04    -  1.00e+00 1.00e+00f  1

Number of Iterations....: 148

(scaled)                 (unscaled)
Objective...............:   7.5000768067322724e+01    1.8568151317638171e+02
Dual infeasibility......:   2.3818402275727155e-10    5.8967889129227490e-10
Constraint violation....:   0.0000000000000000e+00    0.0000000000000000e+00
Complementarity.........:   9.0909090937668046e-10    2.2506619601073186e-09
Overall NLP error.......:   9.0909090937668046e-10    2.2506619601073186e-09

Number of objective function evaluations             = 345
Number of objective gradient evaluations             = 149
Number of equality constraint evaluations            = 0
Number of inequality constraint evaluations          = 0
Number of equality constraint Jacobian evaluations   = 0
Number of inequality constraint Jacobian evaluations = 0
Number of Lagrangian Hessian evaluations             = 148
Total CPU secs in IPOPT (w/o function evaluations)   =      0.158
Total CPU secs in NLP function evaluations           =      0.148

EXIT: Optimal Solution Found.
``````

• changing the upper and lower vectors to `ComponentArrays`
• patching optim via defining a `ldiv!` method for `ComponentArrays`

The working code is:

``````
lower = ComponentArray(x=repeat([floatmin()], N), y=repeat([floatmin()], N), z=repeat([floatmin()], N))
upper = ComponentArray(x=repeat([Inf], N), y=repeat([Inf], N), z=repeat([Inf], N))
Optim.ldiv!(out::ComponentVector, P::Optim.InverseDiagonal, A::ComponentVector) = copyto!(out, A .* P.diag)
opts = Optim.Options(outer_iterations = 2)
results = optimize(f, lower, upper, v0, Fminbox(inner_optimizer),opts,autodiff=:forward)
``````

it returns:

`````` * Status: failure (reached maximum number of iterations)

* Candidate solution
Final objective value:     1.923543e+02

* Found with

* Convergence measures
|x - x'|               = 1.01e+00 ≰ 0.0e+00
|x - x'|/|x'|          = 1.91e-02 ≰ 0.0e+00
|f(x) - f(x')|         = 0.00e+00 ≤ 0.0e+00
|f(x) - f(x')|/|f(x')| = 0.00e+00 ≤ 0.0e+00
|g(x)|                 = 8.51e-02 ≰ 1.0e-08

* Work counters
Seconds run:   12  (vs limit Inf)
Iterations:    2
f(x) calls:    5294
∇f(x) calls:   5294
``````

@odow

Thanks. In my code above I did have small positive lower bounds: `floatmin()`

Tried out yours and it worked. Seems like anything lower than about `1e-9` is causing a crash. Weird.

When I use `1e-8` it works. Here is the solution for x, for example:

`````` 1.0e-8
0.007290631774874027
0.0015940689082409119
0.016124419812656176
0.002324386840220619
0.009408701041340888
0.014141186416612538
1.0e-8
1.0e-8
0.003622059665799834
1.0e-8
0.008444562981428526
0.010041379467192309
0.011468246493959913
0.00164553780292984
3.675460461861131e6
1.0e-8
0.0016481368874902372
1.5709758508294916e6
1.0e-8
3.5883307937084737e-7
0.0188577025351847
1.0e-8
0.0058906658490472125
1.0e-8
0.017933252589371566
``````

I notice some of these are taking the lowest allowed value. In the end I doubt not being able to go lower than this affects the solution, though.

Thanks for the help! Now, I wonder how this will fare with N=5000 for a total of 15k variables…

@longemen3000

I’m a bit confused about why the upper and lower vectors need to be ComponentArrays. And, what exactly is `ldiv` and why do we need it to implement it?

Anyway, it’s working now so I can test out different algorithms and see what scales. Thanks!

1 Like

Seems like anything lower than about `1e-9` is causing a crash. Weird.

Solvers use tolerances for floating point comparisons. From memory, Ipopt’s is `10e-8`. It can’t meaningfully distinguish between a lower bound of `0` and a lower bound of `floatmin()`

4 Likes

I put in a PR with Optim about the `ldiv!` thing. It’s because `Optim.InverseDiagonal` only has `ldiv!` defined for plain `Array`s in the other argument slots rather than `AbstractArray`s.

Also, I don’t think the upper and lower bound have to be `ComponentArrays`. I was able to run this example without them being.

1 Like