# Survey of Non-Linear Optimization Modeling Layers in Julia

**URL:** <https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168>\
**Category:** Optimization (Mathematical)\
**Tags:** nonlinear\
**Created:** [March 20, 2022, 3:39pm UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168 "2022-03-20T15:39:20Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![ccoffrin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ccoffrin/32/400_2.png) [@ccoffrin](https://discourse.julialang.org/u/ccoffrin)\
**Post date:** [March 20, 2022, 3:39pm UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/1 "2022-03-20T15:39:20Z")

</div>

I am doing a survey of the NLP modeling layers in Julia to see which might be applicable to the kinds of problems that I regularly solve. I have the following basic requirements of the modeling layer,

- Support for non-convex functions for the objective
- Support for a system non-convex equality and inequality constraint functions (the equality constraints usually cannot be expressed explicitly as a manifold)
- Support for polynomial and transcendental functions (e.g. `x^2*y^3`,`sin(x)`)
- Some kind of automatic differentiation system (so I don’t have to write derivative oracles by hand)

Here is what I found so far (in alphabetical order),

| Package | NL Objective | NL Constraints | AD |
| --- | --- | --- | --- |
| [ADNLPModels](https://github.com/JuliaSmoothOptimizers/ADNLPModels.jl) | ✅ | ✅ | ✅ |
| [BlackBoxOptim](https://github.com/robertfeldt/BlackBoxOptim.jl) | ✅ | ❌ | NA |
| [Convex](https://github.com/jump-dev/Convex.jl) | ❌ | ❌ | NA |
| [ExaModels](https://github.com/exanauts/ExaModels.jl) | ✅ | ✅ | ✅ |
| [JuMP](https://github.com/jump-dev/JuMP.jl) | ✅ | ✅ | ✅ |
| [Manopt](https://github.com/JuliaManifolds/Manopt.jl) | ✅ | ✅ | 🟡 |
| [Nonconvex](https://github.com/JuliaNonconvex/Nonconvex.jl) | ✅ | ✅ | ✅ |
| [Optim](https://github.com/JuliaNLSolvers/Optim.jl) | ✅ | ✅ | ✅ |
| [Optimization](https://github.com/SciML/Optimization.jl) | ✅ | ✅ | ✅ |
| [ProximalAlgorithms](https://github.com/JuliaFirstOrder/ProximalAlgorithms.jl) | ✅ | ❌ | ✅ |

If I have made an error in the characterization of any listed packages corrections are greatly appreciated.

So far, it seems that ADNLPModels, GalacticOptim, JuMP, Nonconvex and Optim are the packages that currently support all of my requirements. However, if you know of some other Julia package that might be able to solve such problems, I would very much like to hear about it.

Note: original post has been revised based on info from this discussion.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [March 20, 2022, 4:22pm UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/2 "2022-03-20T16:22:57Z")

</div>

You’re missing GalacticOptim.jl which has probably the most coverage via:

 ![Screenshot 2022-03-20 122130](https://global.discourse-cdn.com/julialang/original/3X/b/f/bfaadaeb6c9d2019474b7c9babb0ea230ba9424b.png)

[https://galacticoptim.sciml.ai/dev/](https://galacticoptim.sciml.ai/dev/)

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [March 20, 2022, 4:23pm UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/3 "2022-03-20T16:23:43Z")

</div>

And wide AD support:

 ![Screenshot 2022-03-20 122320](https://global.discourse-cdn.com/julialang/original/3X/4/e/4ef71b57aa2b8e78c35157fb2dfdfa88b9d8c60c.png)

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [March 20, 2022, 4:53pm UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/4 "2022-03-20T16:53:09Z")

</div>

Optim has IPNewton which supports NL constraints [Optim.jl](https://julianlsolvers.github.io/Optim.jl/stable/#examples/generated/ipnewton_basics/)

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [March 20, 2022, 5:51pm UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/5 "2022-03-20T17:51:33Z")

</div>

What does global constrained column mean? And why does MOI not support that?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [March 20, 2022, 6:29pm UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/6 "2022-03-20T18:29:55Z")

</div>

We haven’t wrapped it yet.

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [March 20, 2022, 7:10pm UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/7 "2022-03-20T19:10:18Z")

</div>

I guess I don’t understand the distinction between global and local. It’s up to the solver to figure that out, not MathOptInterface.

---

<div class="post-metadata">

**Author:** ![joaquimg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joaquimg/32/223_2.png) [@joaquimg](https://discourse.julialang.org/u/joaquimg)\
**Post date:** [March 20, 2022, 8:35pm UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/8 "2022-03-20T20:35:33Z")

</div>

Which algorithms Optim rely on for Global constrained and unconstrained?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [March 21, 2022, 12:26am UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/9 "2022-03-21T00:26:27Z")

</div>

> [@odow](#):
>
> I guess I don’t understand the distinction between global and local. It’s up to the solver to figure that out, not MathOptInterface.

Yes, MOI has both algorithms, which is why it has both boxes checked. Flux does not, so it only has one of them checked.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [March 21, 2022, 12:27am UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/10 "2022-03-21T00:27:15Z")

</div>

> [@joaquimg](#):
>
> Which algorithms Optim rely on for Global constrained and unconstrained?

[https://galacticoptim.sciml.ai/dev/optimization\_packages/optim/#Global-Optimizer](https://galacticoptim.sciml.ai/dev/optimization_packages/optim/#Global-Optimizer)

---

<div class="post-metadata">

**Author:** ![joaquimg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joaquimg/32/223_2.png) [@joaquimg](https://discourse.julialang.org/u/joaquimg)\
**Post date:** [March 21, 2022, 1:01am UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/11 "2022-03-21T01:01:19Z")

</div>

> [@ChrisRackauckas](#):
>
> Yes, MOI has both algorithms, which is why it has both boxes checked. Flux does not, so it only has one of them checked.

I think he means the very last column, which is not checked for MOI.

---

<div class="post-metadata">

**Author:** ![joaquimg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joaquimg/32/223_2.png) [@joaquimg](https://discourse.julialang.org/u/joaquimg)\
**Post date:** [March 21, 2022, 1:12am UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/12 "2022-03-21T01:12:52Z")

</div>

Thanks! Now I see the difference in terminology. In MOI/JuMP, the difference between global and local is the capacity of a solver to certify (of course, there might be numerical issues) that the solution is a global optimum or only a local optimum. For GalacticOptim, it seems more about how the feasible space is explored.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [March 21, 2022, 1:50am UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/13 "2022-03-21T01:50:08Z")

</div>

> [@joaquimg](#):
>
> In MOI/JuMP, the difference between global and local is the capacity of a solver to certify (of course, there might be numerical issues) that the solution is a global optimum or only a local optimum. For GalacticOptim, it seems more about how the feasible space is explored.

In GalacticOptim, MOI is a package that is wrapped, and that wrapper does not support the constraints right now so it’s not checked.

---

<div class="post-metadata">

**Author:** ![ccoffrin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ccoffrin/32/400_2.png) [@ccoffrin](https://discourse.julialang.org/u/ccoffrin)\
**Post date:** [March 21, 2022, 3:44am UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/14 "2022-03-21T03:44:34Z")

</div>

@mohamed82008, thanks for the tip about Optim. In this example I did not see how to combine what is shown here with an AD approach for the Jacobian and Hessian. Do you know of an example doing this?

@ChrisRackauckas, I will give GalaticOptim a try going to Ipopt through the MOI backend, unless you suggestion a different one. What AD system do you recommend for sparse large scale problems? `AutoModelingToolkit` sounds like the best choice from the docs you post, correct?

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [March 21, 2022, 4:02am UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/15 "2022-03-21T04:02:15Z")

</div>

> [@ccoffrin](#):
>
> I did not see how to combine what is shown here with an AD approach for the Jacobian and Hessian.

You would probably need to do your own AD when defining gradient/jacobian/hessian functions. @pkofod can correct me if I am wrong. (sorry for the ping)

---

<div class="post-metadata">

**Author:** ![ccoffrin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ccoffrin/32/400_2.png) [@ccoffrin](https://discourse.julialang.org/u/ccoffrin)\
**Post date:** [March 21, 2022, 4:10am UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/16 "2022-03-21T04:10:02Z")

</div>

> [@mohamed82008](#):
>
> You would probably need to do your own AD when defining gradient/jacobian/hessian functions. @pkofod can correct me if I am wrong. (sorry for the ping)

Ok, an AD system is a hard requirement for me. Will wait to hear from @pkofod to confirm the status, but updating the original post to reflect new info.

---

<div class="post-metadata">

**Author:** ![ccoffrin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ccoffrin/32/400_2.png) [@ccoffrin](https://discourse.julialang.org/u/ccoffrin)\
**Post date:** [March 21, 2022, 4:16am UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/17 "2022-03-21T04:16:54Z")

</div>

@ChrisRackauckas, do you have an example of how to use GalaticOptim with constraint functions? I reviewed these docs,

[https://galacticoptim.sciml.ai/stable/tutorials/intro/](https://galacticoptim.sciml.ai/stable/tutorials/intro/)  
[https://galacticoptim.sciml.ai/stable/API/optimization\_problem/](https://galacticoptim.sciml.ai/stable/API/optimization_problem/)  
[https://galacticoptim.sciml.ai/stable/API/optimization\_function/](https://galacticoptim.sciml.ai/stable/API/optimization_function/)

There seems to be a hint of how to specify the constraints through the `cons` argument to OptimizationFunction. But a specification of what this argument should be I could not find.

Also while reviewing these docs,

[https://galacticoptim.sciml.ai/stable/API/optimization\_function/#Defining-Optimization-Functions-Via-AD](https://galacticoptim.sciml.ai/stable/API/optimization_function/#Defining-Optimization-Functions-Via-AD)

I noticed that [AutoForwardDiff](https://galacticoptim.sciml.ai/stable/API/optimization_function/#AutoForwardDiff) is only AD system that says it supports constraints, so it seems like I should use this one instead of [AutoModelingToolkit](https://galacticoptim.sciml.ai/stable/API/optimization_function/#AutoModelingToolkit)?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [March 21, 2022, 4:19am UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/18 "2022-03-21T04:19:03Z")

</div>

> [@ccoffrin](#):
>
> @ChrisRackauckas, I will give GalaticOptim a try going to Ipopt through the MOI backend, unless you suggestion a different one. What AD system do you recommend for sparse large scale problems? `AutoModelingToolkit` sounds like the best choice from the docs you post, correct?

Not necessarily. Each has its own advantages. MTK will scalarize the equations but will generate really fast code. It won’t scale in compile time the best, but for scalar-heavy code that is big and sparse it’s really good, if it compiles in time. Otherwise ReverseDiff with tape compilation is good with similar properties, but it can segfault if the tape gets too long. If the code is heavy in linear algebra, Zygote is a good bet. Tracker is kind of an in-between Zygote-ish thing that can work in some cases where Zygote doesn’t. Forward-mode doesn’t scale as well.

> [@ccoffrin](#):
>
> I noticed that [AutoForwardDiff](https://galacticoptim.sciml.ai/stable/API/optimization_function/#AutoForwardDiff) is only AD system that says it supports constraints, so it seems like I should use this one instead of [AutoModelingToolkit](https://galacticoptim.sciml.ai/stable/API/optimization_function/#AutoModelingToolkit)?

Yes, we should probably add a cons diff overload to MTK. It’s only like 10 lines.

---

<div class="post-metadata">

**Author:** ![ccoffrin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ccoffrin/32/400_2.png) [@ccoffrin](https://discourse.julialang.org/u/ccoffrin)\
**Post date:** [March 21, 2022, 4:25am UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/19 "2022-03-21T04:25:52Z")

</div>

> [@ccoffrin](#):
>
> There seems to be a hint of how to specify the constraints through the `cons` argument to OptimizationFunction. But a specification of what this argument should be I could not find.

@ChrisRackauckas, I found an example in the tests here, [https://github.com/SciML/GalacticOptim.jl/blob/master/test/rosenbrock.jl#L30](https://github.com/SciML/GalacticOptim.jl/blob/master/test/rosenbrock.jl#L30)

---

<div class="post-metadata">

**Author:** ![jd-foster](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jd-foster/32/35824_2.png) [@jd-foster](https://discourse.julialang.org/u/jd-foster)\
**Post date:** [March 21, 2022, 5:54am UTC](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168/20 "2022-03-21T05:54:03Z")

</div>

Not sure if I understand the question, but here is a simple example of AD using the Optim ecosystem:

```julia
using NLsolve
import NLsolve.NLSolversBase: OnceDifferentiable, TwiceDifferentiable

# Beale function:
B(x) = (1.5 - x[1] + x[1].*x[2]).^2 + (2.25 - x[1] + x[1].*x[2].^2).^2 + (2.625 - x[1] + x[1].*x[2].^3).^2
## Himmelblau function:
HM(x) = (x[1].^2 + x[2] - 11).^2 + (x[1]+ x[2].^2 - 7).^2
# Rastrigin function:
RS(x) = 10*2 + x[1].^2 + x[2].^2 - 10*cos(2*pi*x[1]) - 10*cos(2*pi*x[2]);
# Rosenbrock function
R(x) = (1.0 - x[1])^2 + 100.0 * (x[2] - x[1]^2)^2

## Define the test function:
testfun = RS

## Auto-differentiation:
x0 = [10.0; 10.0] # initial point
dfn = TwiceDifferentiable(testfun,x0);

# Find the zeros of the gradient and the iterate states:
base_solver_results = nlsolve(dfn.df, x0,
        method=:newton,
        show_trace=true, store_trace=true, extended_trace=true)

# We can get the vector residual function as the gradient of the test function as:
x_init = zero(x0)
testDiffFun = OnceDifferentiable(dfn.df,x_init,x_init);

```

[Next page](https://discourse.julialang.org/t/survey-of-non-linear-optimization-modeling-layers-in-julia/78168.md?page=2)