A look into the dungeon. I’ll tag this change as soon as I iron out some corner cases.

```
julia> using Optim, RecursiveArrayTools
julia> function polynomial(x)
return (10.0 - x[1])^2 + (7.0 - x[2])^4 + (108.0 - x[3])^4
end
polynomial (generic function with 1 method)
julia> function polynomial_gradient!(storage, x)
storage[1] = -2.0 * (10.0 - x[1])
storage[2] = -4.0 * (7.0 - x[2])^3
storage[3] = -4.0 * (108.0 - x[3])^3
end
polynomial_gradient! (generic function with 1 method)
julia> function polynomial_hessian!(storage, x)
storage[1, 1] = 2.0
storage[1, 2] = 0.0
storage[1, 3] = 0.0
storage[2, 1] = 0.0
storage[2, 2] = 12.0 * (7.0 - x[2])^2
storage[2, 3] = 0.0
storage[3, 1] = 0.0
storage[3, 2] = 0.0
storage[3, 3] = 12.0 * (108.0 - x[3])^2
end
polynomial_hessian! (generic function with 1 method)
julia> ap = ArrayPartition(rand(1), rand(2))
([0.926164], [0.821162, 0.374145])
julia> optimize(polynomial, polynomial_gradient!, polynomial_hessian!, ap, ConjugateGradient())
Results of Optimization Algorithm
* Algorithm: Conjugate Gradient
* Starting Point: [0.9261639118389935,0.8211620082823579, ...]
* Minimizer: [9.999999997479323,7.000857916457364, ...]
* Minimum: 1.932692e-12
* Iterations: 30
* Convergence: true
* |x - x'| ≤ 1.0e-32: false
|x - x'| = 7.67e-04
* |f(x) - f(x')| ≤ 1.0e-32 |f(x)|: false
|f(x) - f(x')| = 6.94e+00 |f(x)|
* |g(x)| ≤ 1.0e-08: true
|g(x)| = 5.12e-09
* Stopped by an increasing objective: false
* Reached Maximum Number of Iterations: false
* Objective Calls: 62
* Gradient Calls: 34
julia> optimize(polynomial, polynomial_gradient!, polynomial_hessian!, ap, GradientDescent())
Results of Optimization Algorithm
* Algorithm: Gradient Descent
* Starting Point: [0.9261639118389935,0.8211620082823579, ...]
* Minimizer: [10.00000456963858,7.010984469414458, ...]
* Minimum: 2.920265e-08
* Iterations: 1000
* Convergence: false
* |x - x'| ≤ 1.0e-32: false
|x - x'| = 7.66e-06
* |f(x) - f(x')| ≤ 1.0e-32 |f(x)|: false
|f(x) - f(x')| = 2.01e-03 |f(x)|
* |g(x)| ≤ 1.0e-08: false
|g(x)| = 9.14e-06
* Stopped by an increasing objective: false
* Reached Maximum Number of Iterations: true
* Objective Calls: 2510
* Gradient Calls: 2510
julia> optimize(polynomial, polynomial_gradient!, polynomial_hessian!, ap, BFGS())
Results of Optimization Algorithm
* Algorithm: BFGS
* Starting Point: [0.9261639118389935,0.8211620082823579, ...]
* Minimizer: [9.999999997694495,6.999913965528232, ...]
* Minimum: 1.290248e-16
* Iterations: 39
* Convergence: true
* |x - x'| ≤ 1.0e-32: false
|x - x'| = 1.55e-05
* |f(x) - f(x')| ≤ 1.0e-32 |f(x)|: false
|f(x) - f(x')| = 3.07e+00 |f(x)|
* |g(x)| ≤ 1.0e-08: true
|g(x)| = 4.61e-09
* Stopped by an increasing objective: false
* Reached Maximum Number of Iterations: false
* Objective Calls: 134
* Gradient Calls: 134
julia> optimize(polynomial, polynomial_gradient!, polynomial_hessian!, ap, LBFGS())
WARNING: Method definition twoloop!(AbstractArray{T, 1} where T, AbstractArray{T, 1} where T, AbstractArray{T, 1} where T, AbstractArray{T, 2} where T, AbstractArray{T, 2} where T, Integer, Integer, AbstractArray{T, 1} where T, AbstractArray{T, 1} where T, Bool, Any, Any) in module Optim at /home/pkm/.julia/v0.6/Optim/src/multivariate/solvers/first_order/l_bfgs.jl:22 overwritten at /home/pkm/.julia/v0.6/Optim/src/multivariate/solvers/first_order/l_bfgs.jl:22.
Results of Optimization Algorithm
* Algorithm: L-BFGS
* Starting Point: [0.9261639118389935,0.8211620082823579, ...]
* Minimizer: [9.99999999997918,7.000054968318689, ...]
* Minimum: 1.508989e-17
* Iterations: 32
* Convergence: true
* |x - x'| ≤ 1.0e-32: false
|x - x'| = 1.75e-06
* |f(x) - f(x')| ≤ 1.0e-32 |f(x)|: false
|f(x) - f(x')| = 9.89e+01 |f(x)|
* |g(x)| ≤ 1.0e-08: true
|g(x)| = 4.16e-11
* Stopped by an increasing objective: false
* Reached Maximum Number of Iterations: false
* Objective Calls: 112
* Gradient Calls: 112
julia> optimize(polynomial, polynomial_gradient!, polynomial_hessian!, ap, Newton())
Results of Optimization Algorithm
* Algorithm: Newton's Method
* Starting Point: [0.9261639118389935,0.8211620082823579, ...]
* Minimizer: [9.999999999999996,6.9999829381032095, ...]
* Minimum: 7.801052e-15
* Iterations: 2
* Convergence: true
* |x - x'| ≤ 1.0e-32: false
|x - x'| = 1.81e+01
* |f(x) - f(x')| ≤ 1.0e-32 |f(x)|: false
|f(x) - f(x')| = 4.22e+16 |f(x)|
* |g(x)| ≤ 1.0e-08: true
|g(x)| = 1.05e-10
* Stopped by an increasing objective: false
* Reached Maximum Number of Iterations: false
* Objective Calls: 7
* Gradient Calls: 7
* Hessian Calls: 2
julia> optimize(polynomial, polynomial_gradient!, polynomial_hessian!, ap, NewtonTrustRegion())
Results of Optimization Algorithm
* Algorithm: Newton's Method (Trust Region)
* Starting Point: [0.9261639118389935,0.8211620082823579, ...]
* Minimizer: [10.0,6.999891492857457, ...]
* Minimum: 3.277275e-12
* Iterations: 32
* Convergence: true
* |x - x'| ≤ 1.0e-32: false
|x - x'| = 6.73e-04
* |f(x) - f(x')| ≤ 1.0e-32 |f(x)|: false
|f(x) - f(x')| = 4.06e+00 |f(x)|
* |g(x)| ≤ 1.0e-08: true
|g(x)| = 9.74e-09
* Stopped by an increasing objective: false
* Reached Maximum Number of Iterations: false
* Objective Calls: 33
* Gradient Calls: 33
* Hessian Calls: 32
```